When Anecdotes Provide Strong Evidence
- Spencer Greenberg and Travis M.
- 1 hour ago
- 8 min read
Short of time? Read the key takeaways
🧪 Most anecdotes about treatments are very weak evidence. Individual stories of improvement are usually explained by misattribution, placebo effects, natural recovery, or random fluctuations. For this reason, anecdotes should generally be treated with skepticism rather than taken at face value.
⚖️ Under rare but identifiable conditions, a single anecdote can provide strong evidence. When an improvement is accurately reported, dramatic, quick, unprecedented, long-lasting, and clearly attributable to the treatment, an anecdote really can provide strong evidence that rationally justifies a substantial update in belief.
📐 Strong anecdotes work by strongly favoring one explanation over the alternatives. When the criteria in this article are met, the anecdotal improvement is far more likely to have been observed if the treatment genuinely works than if it doesn’t. That’s what makes it strong evidence.
🧭 A strong anecdote shows that a treatment worked for one person, but that can be worth attention. A strong individual case does not establish population-level effectiveness, but it can meaningfully indicate that a treatment is worth further attention, especially for people with similar conditions or few remaining options.
🚨 Care is still required. Even strong anecdotes are only valuable insofar as dangerous ideas, ideas with an especially small chance of working, and the hucksters preying on vulnerability, can be avoided.
Was Einstein’s theory of general relativity confirmed by an anecdote?
Anecdotes often play a large role in how people talk about treatments in health and wellness contexts. You might hear someone tell a story about how some treatment worked for them and be tempted to take that as reasonable evidence it will work for you. We’re going to discuss that kind of reasoning in detail in this article. But first, let's explore the common pop science claim about how Einstein’s theory of general relativity was decisively confirmed via what has been called anecdotal evidence - namely, Eddington’s observations of light from the sun bending around the moon during the total eclipse of 1919.
If that was really what happened, then a single night’s observations (which one statistician has, perhaps uncharitably, described as “an anecdote about what happened at a single event,”) were sufficient for confirmation of a scientific theory that dramatically changed our view of the entire universe. Now that’s a powerful anecdote!
But is that what actually happened? Not really. In fact, while the eclipse observations did play a role in getting general relativity taken more seriously, its widespread acceptance took decades and relied on a lot more experimental results. Even today, it continues to be confirmed through discoveries like gravitational waves.
In hindsight, relying on one expedition’s measurements to establish a new paradigm would have been reckless. Especially given doubts that were later raised about Eddington’s data. As many (including Stephen Hawking) have noted: “[T]he errors were as great as the effect they were trying to measure. Their measurement had been sheer luck, or a case of knowing the result they wanted to get.”
This kind of skepticism can be well placed. When your cousin tells you that drinking garlic water cured his ear infection, it’s reasonable to wonder whether luck, placebo effects, or misattribution are more responsible than his botanical beverage. And yet, surprisingly, there are some rare cases when a single anecdote about a treatment can provide strong evidence in its favor. This article is about those rare cases.
Knowing when anecdotes provide good evidence is important for anyone who wants to be able to evaluate health claims they encounter in the wild, and it can also be helpful for people with hard-to-treat chronic problems who are looking for things to try from among the myriad of suggestions that have unfortunately not yet been tested in high-quality studies.
The Problems with Anecdotes
When a friend, acquaintance, or internet stranger reports that a non-proven treatment seemed to help them, the vast majority of the time, that is very weak evidence that the treatment really works. But, fascinatingly, it is sometimes pretty good evidence. When is it pretty good evidence? Well, to understand that, it helps to see the problems with anecdotes. Anecdotes (e.g., of trying a treatment and getting an improvement) usually fail to provide quality evidence that something works because:
The anecdote may not be an accurate or sufficiently complete report of what actually happened (due to lying, forgetting, or not including all the relevant context)
The improvement may simply be a placebo effect (an improvement that occurs due to the belief that an improvement will occur)
The improvement may be only temporary or just a random fluctuation in the condition
The improvement may be real, lasting, and not a placebo effect, but caused by something other than the treatment in question (and simply be misattributed to the treatment)
The improvement may be caused by the treatment and long-lasting, but not relevant to you (because the condition the person who tried the treatment has is a different condition or the treatment happened to help them but doesn't work for most people with that condition).
That's a lot of problems with anecdotes! And for those reasons, anecdotes usually are not a good reason to believe something works. But, interestingly, if things are just right, those problems can be mitigated enough for the anecdote to provide genuinely good quality evidence.
When Anecdotes Provide Strong Evidence
For an anecdotal report of a treatment working to provide good evidence, it’s typically sufficient (but not necessary) that all of the following are true:
You can be really confident the report doesn’t involve a lie, or an inaccuracy in memory or reporting.
The condition had been happening for a long time, and no improvement even close to this degree had ever happened before.
The treatment worked very quickly.
The benefit lasted indefinitely or for a very long time after.
There were no other meaningful changes in that time that the improvement could be attributed to.
The effect size is large enough to be noticed by the person who claims to have experienced the effect.
The person had tried multiple other treatments previously that they concluded didn't work.
The reasons for the first criterion are obvious. When you read about evidence for a treatment in an academic journal, you can be fairly confident that multiple people with expertise have looked at it and judged it robust enough for publication. Of course, that’s no guarantee of quality (for example, see our article on the problems with peer review), but it is at least some level of safe guard - albeit imperfect. When it comes to anecdotes, you don’t even have that. If you’re going to substantially alter your beliefs or behavior, it is therefore important that you’re confident the anecdote isn’t predicated on a lie or inaccuracy in memory or reporting.
Satisfying the criteria from 2-5 serves to reduce the chances that the recovery is due to things like natural variance, regression to the mean, or the placebo effect.
Criterion 6 is there because more robust forms of evidence (such as high-powered randomized controlled trials) have the data and statistical tools to reliably discover very small effect sizes, but if someone claims they have noticed that they sleep an extra ten seconds a night for every grain of sugar they consume in the day, you might reasonably wonder whether they would really be able to sense that level of precision. In general, you need to be confident that the person whose anecdote you’re hearing would be able to reliably spot the effect size they claim to have experienced.
And without satisfying the final criterion (7), the chances are higher that recovery would have followed almost any intervention at all (perhaps via the placebo effect), and that any specific mechanisms posited by the treatment being advocated aren’t responsible for improvement. It’s highly unlikely that a case of the placebo effect could satisfy all the criteria in this list.
Notice that we’re not saying that all genuinely effective treatments will satisfy these criteria. In fact, many won’t! What we’re saying is: When you’re confronted with anecdotal evidence for a treatment, and you have only the tools of good reasoning to help you determine whether that treatment is truly effective or not, applying these criteria can help you. They are designed to be ‘jointly sufficient’ (not ‘necessary’) for evaluating when a single anecdote can provide good evidence, and the criteria for evaluating other types of evidence (such as studies) would be very different than these.
In the very special case when an anecdote meets the criteria above, it can provide quite good evidence that a treatment worked. The reason is because when those criteria are met, the chance of the observed improvement happening if the treatment actually works is much greater than the chance of it happening if the treatment doesn’t work.
Technically speaking, such a report of improvement provides a large ‘Bayes factor’. Or, put another way, a large number in answer to what we have called The Question of Evidence: “How many times more likely are we to see this evidence if the treatment works compared to if it doesn’t?”
Unfortunately, very few individual anecdotal reports live up to these criteria. And, perhaps even more unfortunately, even when individual anecdotes do live up to each of these, all that means is the treatment worked for that one person! It working for them doesn’t mean it would work for a high percentage of people. Their case may be unusual.
However, as these criteria help illustrate, not all anecdotal reports are useless. Especially when aggregated, they can be a way to source potentially very valuable ideas for things to try when nothing else has worked (once you've tried the treatments for the condition that have the most evidence of effectiveness, and you're looking for what else to try that still has a reasonable chance of helping). But anecdotes are only valuable insofar as dangerous ideas, ideas with an especially small chance of working, and the hucksters preying on vulnerability, can be avoided.
Putting This Into Practice
If you are trying to use anecdotes to decide what may help you, here’s a checklist you can use to help decide if the anecdote is valuable:
Does the person with the anecdote have an incentive to lie or exaggerate, and is there reason to think their report is not reliable and not complete? If so, the evidence is not reliable.
Was their condition occurring only for a short time before the treatment? If so, it's going to be hard to tell that this wasn't just their body naturally healing or a random fluctuation in their condition, unrelated to whatever they tried.
Was their improvement when taking this treatment slow or not substantially larger in magnitude than all prior improvements? If so, it's going to be hard to be confident that this treatment is the cause of the change, rather than natural healing, random fluctuations, or other factors.
Was there another meaningful change around that time that could account for the improvement? If so, the improvements may be due to that change instead of this treatment.
Was their improvement only temporary, or has it only been ongoing for a short time? If so, it's hard to tell whether the improvement will be persistent or soon go away.
Is your situation quite different from the situation of the person with the report for it to be relevant? If so, the treatment may not apply to you.
Has this person never tried other treatments before, or tried other treatments that they also concluded cured them (only to change their mind later)? If so, it's going to be hard to be confident that it really was this treatment that had an effect, rather than a placebo effect.
If you want to take your thinking about this topic further, why not try our mini-course about evidence (it’s free!):