Who Was Asked?
A survey only speaks for the people who answered, and they are rarely a fair mix. They tend to have strong opinions or something to gain, and many shade their answers to look good. Before you trust a result, ask who was left out.
by Darrell Huff · 9 ideas
A survey only speaks for the people who answered, and they are rarely a fair mix. They tend to have strong opinions or something to gain, and many shade their answers to look good. Before you trust a result, ask who was left out.
A result from a handful of cases can come from luck alone. Toss a coin ten times and you may well get seven heads, which proves nothing about the coin. Ask how many cases were counted, and trust a finding more when it holds up again in a bigger test.
Every measurement has some error, so a small difference may be nothing but noise. If a test can be off by three points either way, a score of 101 is not really higher than 99. Before you rank or choose by a gap, ask whether it is bigger than the error.
An average can be the total divided by the count, or the middle value. A few very rich people push the first far above what a typical person earns. When a few extremes pull on the data, ask for the middle value.
Percentages hide the numbers they come from. A risk that "doubles" may go from 1 in a million to 2, and a price cut by half and then raised by half ends up below where it started. Ask for the real numbers behind any percentage.
A chart can make a tiny change look huge by starting its vertical scale just below the data instead of at zero. A rise of 2 percent then fills the whole picture like a leap. Look at where the scale starts before you react to the slope.
A number often proves something close to the claim but not the claim itself. A soap that kills germs in a lab dish has not shown that it keeps you healthy. Ask whether the figure measures exactly what you are told it shows.
Two things that move together don't prove that one causes the other. Ice cream sales and drownings rise together only because both go up in hot weather. Before you accept a cause, ask whether a third thing drives both or whether the cause runs the other way.
Before you believe a number, ask who says so, how they could know and what is missing. An exact count of something nobody could count, like the rats in a city, claims more knowledge than anyone has. If a claim fails these questions, set it aside.