How to sort evidence from B.S.
- Spencer Greenberg and Travis M.
- 2 days ago
- 9 min read
Key Takeaways
Common questions aren't reliable tests of evidence. People often make the mistake of thinking that questions like “Is this consistent with my hypothesis?” can tell you whether something counts as evidence. They can't.
Evidence is anything that makes your hypothesis more likely. If your observation is more likely in a world where your hypothesis is true than in one where it's false, it counts as evidence.
Ask 'The Question of Evidence'. Answering the question “How many times more likely am I to see this evidence if my hypothesis is true than if it's false?” will tell you whether something counts as evidence and how strongly.
Even rough estimates can guide your thinking. To answer The Question of Evidence, you'll often have to make educated guesses. There are techniques for honing those estimates and a procedure for combining them with the other evidence you have.
Bayes’ theorem underlies good critical thinking. These principles help you think more clearly about what supports or undermines your beliefs, and they form a powerful foundation for reasoning well.
Have you ever been debating with someone, offering evidence to support your views, only to have them reply to one of your carefully crafted points with the objection “That’s not evidence!”?
We’re going to tell you how you can decide who is right, and what it really means for one thing to be evidence for something else.
These sorts of disagreements about the nature of evidence happen in all sorts of domains. For example, it happens when...
Atheists are told by theists that the existence of suffering is not actually evidence against god's existence.
Theists are told by atheists that our universe appearing to be finely tuned for life is not evidence that there is a god.
Progressives say that racial disparities in outcomes are evidence of systemic racism, and conservatives dispute that.
Conservatives point to student protests against conservative speakers as evidence of declining support for free speech, and progressives dispute that.
And so on. Sometimes it’s a fair objection, and other times it’s not. We’re going to equip you with the tools to sort the fair cases from the unfair. So you’ll be able to respond to “That’s not evidence!” with “Yes it is, and here’s why…”
What is evidence?
When you’re thinking about the meanings of concepts related to things like knowledge, belief, and reasoning, you’re engaging with an area of philosophy called ‘epistemology’ (which translates to ‘theory of knowledge’). Since consensus is extremely rare in philosophy, it’s perhaps surprising that there is widespread agreement about when something counts as evidence.
Something provides non-zero evidence in favor of a claim if and only if you're more likely to see that thing in a world where that claim is true than in a world where it's false.
This means that something counts as evidence for a hypothesis if and only if it makes the hypothesis at least slightly more likely. If the definition above seems unwieldy, don't worry, we'll give you a simpler version of it in a moment that is more useful in practical situations. But before we do, let's see what it looks like to put this definition into action.
Suppose that, when you get colds, those colds have overwhelmingly tended to affect you for at least three days. But one day you try drinking pomegranate juice at the first sign of a cold, because you heard it helps. This time, your cold only lasts one day instead of the usual 3 or more. Would that observation (that the cold lasted only one day, instead of 3) be evidence that the remedy works? Yes, it would be a non-zero amount of evidence! The reason is because you're more likely to see that result (the cold lasting a shorter amount of time) if indeed the remedy works than if it doesn't work. However, for reasons we'll see shortly, it's only a small amount of evidence, so it shouldn't cause you to believe very much at all in pomegranate juice as a remedy for colds. Since it's a small amount of evidence, it should nudge your belief only slightly.
This is the account of evidence used most widely across scientific research, statistics, decision theory, and much of philosophy. There are philosophical disputes about some interesting finer details about the nature of evidence, but those are details we won’t need to focus on here.
How to Determine the Strength of Evidence
Of course, an important fact about evidence is that it comes in different strengths. Seeing someone holding a smoking gun, standing over a dead body, is much stronger evidence that they have committed a murder than just finding their fingerprint at the scene.
Colloquially, when someone says a phrase like “I have evidence that X”, they usually mean that they have quite a lot of evidence, not just some tiny amount of it. But this has more to do with rules of conversation than facts about what technically counts as evidence: They are likely to be telling you about their evidence only if they think it is (or should be) strong enough to sway your beliefs. Technically speaking, though, even something that you’re only a tiny, tiny amount more likely to see if a hypothesis is true counts as evidence for that hypothesis.
So, if you’re at a poker night and someone says “I have evidence that Molly is cheating,” people will probably expect something overt like cards up her sleeve or a conveniently placed mirror that lets her see players’ hands. But even subtle patterns can technically qualify as evidence (albeit, small amounts of it): If Molly keeps having the best hands at the table, that is initially a small amount of evidence of cheating - not nearly enough to conclude she's doing so, but not nothing. However, if she keeps having the best hands for long enough (e.g., 50 in a row), there will come a point when her luck is so statistically unlikely that the hypothesis that she's cheating becomes far more likely than the hypothesis that she's just having a lucky night, even if she's not the sort of person you would expect to cheat.
All of this points to an important fact: In order to talk about evidence usefully, you must be able to evaluate its strength. So, how do you do that?
You just have to use what we call ‘The Question of Evidence’:
The Question of Evidence “How many times more likely am I to see this evidence if my hypothesis is true than if it's false?” |
The bigger the answer, the stronger the evidence! In other words, this simple question tells us not only whether something is or isn't evidence, but also how strong that evidence is!
For example, you might judge that you’re roughly:
Five as likely to see a conveniently placed mirror in Molly's line of site if she is cheating than if she isn’t (i.e., the answer to the question of evidence in this case would be "5")
Ten times as likely to see a card on the floor next to her if she is cheating than if she's not
Equally likely to see Molly wearing a watch whether or not she’s cheating (i.e., here the answer to the question of evidence would be "one")
In this case, the card on the floor is the strongest evidence (about two times stronger than the mirror) while the watch offers no evidence at all, since it’s equally likely either way.
There are a lot of other questions that people ask, which they think can be used to evaluate whether something counts as evidence and how strong it is, but they don’t work. They’re not the right question (it turns out the "question of evidence" is the only correct question - an interesting mathematical fact!). For example:
“Is this consistent with my hypothesis?” Is extremely common (as seen in this great video) but doesn’t work because something can be consistent with your hypothesis without being evidence for it.
“Can I still explain this if my hypothesis is true?” Oftentimes, there will be a way to explain an observation away, even in situations where the observation actually makes the hypothesis less likely (and so is actually evidence against your hypothesis).
“How likely would we be to see this observation if my hypothesis were true?” This fails to consider a crucial fact: how likely you'd be to see this result if the hypothesis were false.
People often think they can learn much more from these questions than they really can. If you want to reliably know whether something counts as evidence or how strongly it supports a hypothesis, those questions won't do the job. You need The Question of Evidence. It prompts you to wonder whether there is some better explanation for the evidence than your hypothesis.
In most everyday contexts, where you don’t have quantitative data to evaluate, you’ll be merely estimating the number (rather than calculating it), but that doesn’t mean you’ll be thinking nonsense. Your numbers don’t have to be precise; just reasonable. Even rough estimates can help you make better decisions and avoid overreacting to weak evidence or underreacting to strong evidence. There are also techniques (such as these) that can help you make more accurate estimates.
Now, imagine your answer to The Question of Evidence is 2. That is to say, you think you’re around twice as likely to encounter a given piece of evidence in a world where your hypothesis is true than in a world where it is false. You could write that as a ratio, like this:
2:1
Or, if your answer is that you’re 1.0004 times more likely (so, only very slightly more likely than not), you could write that as:
1.0004:1
That means the evidence only nudges your belief very, very, very slightly in favor of the hypothesis. It’s still technically evidence (it increases the odds a tiny bit) but it’s so weak that you might just say, for practical or everyday purposes, that there’s no evidence - much the same way that we might colloquially say "the result was 1" if the result was actually "1.0004"
And, if the answer is less than 1, then E counts as evidence against your hypothesis. For example, if your answer to The Question of Evidence is:
0.5:1
That means the evidence is twice as likely to appear if the hypothesis is false than if it's true (you could also write this as ‘1:2’). So it should push you to become less confident in the hypothesis.
But, of course, before you saw any new evidence, you probably already had some beliefs about the hypothesis! If you thought the odds of the hypothesis being true were 2:1 before seeing the new evidence (so, it’s twice as likely to be true than it is to be false), and now you’re seeing evidence which you think you are roughly 3 times more likely to encounter in worlds where the hypothesis is true, you need to update your odds. You take your odds before (2:1) and multiply them by your answer to The Question of Evidence (3:1). This is the same as 2/1 x 3/1. This means that now, after seeing this new evidence, your odds should be 6:1 in favor of the hypothesis! You went from roughly 67% sure (equivalent to 2:1 odds), to 86% (equivalent to 6:1 odds)!
So, now imagine you’re reading the news and you see a story reporting a small study that found a correlation between drinking coffee and cancer. How strong is this as evidence for coffee causing cancer? To find out, apply The Question of Evidence:
“How many times more likely am I to read about a small study finding a correlation between drinking coffee and cancer if coffee really does cause cancer than if it doesn’t?”
Take a moment to think about it. Are you 100 times more likely to encounter this evidence (a small study finding a correlation between drinking coffee and cancer) if coffee causes cancer than if it doesn't? 10 times more likely? 2 times more likely? 1 times more likely (i.e., it's equally likely to happen whether or not coffee causes cancer)?
The answer is that it's quite weak evidence (bigger than 1, but probably smaller than 5). That's because it's very easy for studies to find small correlations between all kinds of things that are not causes. So you're only very slightly more likely to see a study showing a small correlation between coffee drinking and cancer if coffee really does cause cancer than if it doesn't. The Vox graphic below shows this vividly. There have been small, correlational studies showing that all sorts of things cause cancer, but also, similar studies showing that those same things prevent cancer! That's because that kind of study just can't tell you reliably one way or another what causes cancer:
Results like these have led some people to conclude (tongue in cheek) that “everything both causes and cures cancer.”
That doesn't mean that studies are useless: we just need appropriate types of studies to answer this question. The most reliable way to tell if coffee causes cancer would be to randomize thousands of people to either drink coffee or not, somehow make sure that they stick to what they were told to do, track them all for years, and compare the rates of cancer in the group told to drink coffee compared to those told not to drink it. Unfortunately, that would be a very expensive and difficult type of study to run, but if it were competently run it could provide a large amount of evidence about the causal link between cancer and coffee (if there is any).
Conclusion
Now you have the tools to respond to the scenario at the start of this article. When you’re debating with someone and they tell you that something you’ve pointed to isn’t evidence, start by asking yourself The Question of Evidence: “Is the thing I pointed to more likely (even if only slightly) to be encountered if my hypothesis is true than if it’s false?” If the answer is “Yes”, then you can tell your debating partner something along these lines: “I think this is evidence because it’s the sort of thing that’s more likely to happen if my hypothesis is true than if it’s false. That’s the most widely used definition of evidence.”
All of the ideas discussed in this article are straightforward applications of Bayes' theorem. And, remarkably, you can prove a lot of what we've said mathematically (though we'll spare you the details). In our view, these ideas serve as an important foundation for critical thinking. If you want to take your thinking about this concept even further, why not try our free mini-course all about The Question of Evidence?