How (And When) To Learn From Crowds
- Travis M.
- Apr 16
- 16 min read
Short of time? Read the key takeaways.
👥 Averaging group judgments often beats individual ones. There are mathematical reasons why asking a crowd and averaging their answers tends to produce more accurate results than relying on a single person's judgment, even when that person is an expert.
🎯 Crowds work best for two question types. They excel at (1) factual questions with definitive answers and (2) estimating what a target population will think or feel.
⚠️ Systematic bias is the biggest threat to crowd wisdom. When averaging responses, individual random errors cancel out. What remains is 'signal' plus systematic bias, so crowds fail mainly when members share the same directional blind spots.
🧍 You can be your own crowd. Studies show that, by repeatedly estimating the same answer independently over time and then averaging your responses, you can simulate crowd wisdom and improve your accuracy compared to relying on a single self-assessment.
📋 A simple four-step process maximizes crowd accuracy. Pick an averageable question, recruit a diverse and independent crowd, calculate the mean or median of responses, and calibrate your confidence based on how likely systematic bias is to affect results.
When you’re making important decisions, there are probably all sorts of questions you wish you knew the answers to. For example:
How persuasive is each of these drafts of an important message?
Which of these two email subject lines is most likely to get opened?
How fairly did I act in this situation?
How appealing is each of these drafts of a dating profile?
How much will this project cost in total?
How effective do each of these plans for managing procrastination seem?
In cases like these, you could rely on your own judgment, but there are mathematical reasons why you’ll often get better results if you ask a group of people and take the average of their answers. This is something a lot of people don’t realize they could be doing, but there’s a vast amount of evidence that it can improve judgments.
In this article, we want to teach you how to use the wisdom of crowds to make better decisions in your own life. The crowds we’ll cover are:
Friends and loved ones
Social media
Existing studies
Yourself (yes, you really can be a crowd by yourself)
You’ll learn:
A four-step plan for consulting these crowds
The questions these crowds can (and can’t) help you with,
How to ask these crowds (and how to find them), and
Why consulting crowds is likely to improve many of your judgments (featuring a simple explanation of the surprising math)
If you want to skip a lot of the details, you can jump ahead to section 5 (“A four-step plan for consulting crowds”), which distills the first 4 sections into an easily actionable list of steps.
1. What questions can crowds help you with?
If you’re going to get insights from crowds, then you need two things: a relevant question to ask and a good enough crowd to answer. Let’s start with how you pick a relevant question.
There are two kinds of questions that crowds can help you with:
Factual questions that have (or will have) a definitive answer. Such as the likelihood of an outcome occurring, whether or not a claim is true, or even who will win an election. The fact that crowds tend to be better than individuals at answering questions like these is what phrases like “the wisdom of crowds” and “collective intelligence” typically refer to (more on that below).
Estimates of what a specific ‘target’ population of people will think or feel about something. Such as how engaging visitors will find your website, or how funny dinner guests will find your speech. In these cases, the crowd will help you towards the ‘correct’ answer, which is the average response of the population you want to understand (the ‘target’ population). And, just like with factual questions, the average answer of a crowd will tend to be much more accurate than a single judgment by an individual person.
Note: For the second type of question, the target population can even be just one person (e.g., you want to predict how your boss or your partner will feel about something). However, as target populations get smaller, the correct answer becomes more sensitive to random variation. Thus, crowds will tend to give more reliable insights into larger target populations than smaller ones (all other things being equal).
2. Which crowds should you ask?
A small handful of key features determine a lot about whether a crowd will be insightful enough to answer your questions. Early research on the topic emphasized that crowds must be:
Diverse in judgment: Members of the crowd should have diverse perspectives, and diverse sources of information. This raises the chances that they don’t tend to share the same biases, misconceptions, or training, thereby lowering the likelihood of systematic biases.
Independent: Members of the crowd aren’t influencing each other’s judgments. When people are allowed to influence each other, social biases can quickly make crowds much less insightful.
But more recent research suggests that some amount of structured discussion (e.g., in small subgroups) can actually improve the accuracy of average group judgments, despite slightly reducing independence and diversity. The idea is that discussion improves average group judgments when discussion leads to the less accurate group members updating their judgments more than the more accurate group members do, thereby dragging the average judgment towards the true answer.
This means it’s important that discussions (if they happen) are conducted in ways that don’t introduce unhelpful biases. For example, if one person dominates discussion, then that would unhelpfully bias other group members towards that person’s judgment, whether or not their judgment is accurate. Thus, researchers say that any discussions must be:
Decentralized. Roughly speaking, this means that things like control, authority, and time spent contributing to the discussion should be distributed evenly.
So, where do you find crowds like this?
Lots of places. Let’s discuss five.
1. Friends and loved ones. You know this crowd! You can consult the crowd of your friends and loved ones in a variety of different ways, from informal chats to sending them a survey. Of course, this isn't likely to be a big enough group to formally calculate an average, but it can improve on your solitary judgment.
Something to be careful of is that your friends and loved ones are more likely to have less diverse perspectives than other crowds, as a result of similar backgrounds, sources of information, or outlooks. And, although you might be able to get them to answer independently (without influencing each other), this probably varies a lot from crowd to crowd. The main advantage of consulting this kind of crowd is how quick and easy it can be.
2. Social media contacts. You can consult this kind of crowd by putting out a poll or asking your question as a post/status and looking at answers in comments. You could post your question just to people who follow you (e.g., on Facebook) or to a forum (e.g., Reddit, Quora).
Again, you’ll have to be careful about the diversity in the crowd and the independence of the answers. For example, a poll where people don’t see other votes until they’ve cast their own is highly independent, but a question asked as a status or post where respondents all reply in the comments (where they can see each other’s replies) is much less so.
3. Positly.com. This is a sister project to Clearer Thinking! It’s a platform that allows you to post a survey or study you’ve designed and then pay people to answer that survey. It’s inexpensive, and we use it all the time to gather insights from crowds. If you are interested, please check it out at positly.com. And if you want some help setting it up, feel free to reach out to us at hi@positly.com.
The power of positly.com is that It allows you to recruit large, diverse samples very quickly, and it ensures that responses are independent. It also allows you to target specific demographics, and has protections against spam and bots (which social media lacks).
4. Existing studies. Someone else might have already asked a crowd your question and published the answer. By searching for existing studies or polls on the topic of your question, you can often gain insights from crowds (thereby improving your own judgments) with much less work than it would be to conduct a study of your own. The main downsides of this approach are that (a) lots of studies are behind paywalls, and (b) you have no control over the precise wording and methodology used. Prediction markets are another version of this, and we have a sister project called Personality Map that gives you free access to a vast number of correlations published in existing studies. Maybe the answer to your question is there already.
5. Yourself (the inner crowd). It might seem silly to imagine that you can be a crowd by yourself, but studies show that you can simulate the activity of a crowd, and gain similar wisdom, by repeatedly estimating the answer to a question and then taking the average of your estimates. The best way to do this is to do things that simulate independence (e.g., spacing out your estimates over time, so you forget old ones when making new ones) or simulate diversity (e.g., by taking on different perspectives or frames, deliberately starting from different premises, or trying to ‘consider the opposite’ point of view).
3. What questions can’t crowds help you with?
Here are some heuristics for determining whether the question you want to ask is not one that crowds can help you with.
Don’t ask a crowd your question when members of the crowd are…
…likely to have no clue about the answer.
If you’re asking a question that people don’t have enough knowledge to make even a rough guess about, then there will not be enough ‘signal’ in people’s judgments to be informative (more on this in the “Why crowds are (sometimes) wise” section below). This rules out questions that are about very obscure facts or highly technical.
For example: “How many species of beetle have been found in the Amazon rainforest during a single year?” is probably too obscure for most crowds to help you with, and “How likely is this method of detecting habitable exoplanets to work?” is probably too technical.
However, when it comes to highly technical questions, if you are able to find a crowd of people with the relevant technical expertise, then you will tend to find that averaging their answers is remarkably reliable at giving you a more accurate answer than a single expert. This has been demonstrated in numerous studies, with technical questions as diverse as diagnosing cancer and decision making during a live volcanic crisis.
…likely to be systematically biased.
A group is systematically biased, when its members’ answers tend to be biased in the same direction (i.e., people tend to overestimate or tend to underestimate). This is sometimes the result of how the group has been selected. For example, if you asked only professional body builders what the average person’s strength is, you’d probably get a result that is systematically biased to overestimate.
But some systematic biases will be present in any crowd of people, because they’re extremely common cognitive biases. If your question is likely to be affected by such biases, you should not expect crowds to give accurate answers. For example:
The question “How likely is it that this risky but exciting opportunity will work out?” is affected by survivorship bias. People may be more likely to hear about the successes than the failures, which distorts their perceptions.
The question “How dangerous is this widely reported threat, really?” is affected by the availability heuristic. People may have heard disturbing examples of this (e.g., on TV), making them think it's more common than it is.
However, since all humans (including you) are prone to these biases, there might still be value in consulting crowds on these kinds of questions: when you aren’t confident in your ability to account for these biases in your own judgments and you only want to get an answer that is better than your own single judgment, then crowds can likely still help.
Okay, so you’ve got a question and a crowd. Let’s address one more detail before we put this all together into a handy four-step plan.
4. How should you phrase your question?
The power of consulting a crowd comes from the act of averaging the responses of the people in that crowd. That means, to use this method, you need to ask your question in such a way that the responses can actually be averaged. The answer options can’t be unordered categories like eye color, favourite sports team, or country of birth because there’s no way of finding the mean or median of answers like ‘USA’, ‘Australia’, and ‘Italy’.
Instead, ask your crowd to do one of the following:
Answer as a percentage
For example, “What percentage likelihood does this strategy have of succeeding?”
Use a Likert scale
This means that each possible answer is a phrase, clearly ordered on a spectrum. You can then assign each phrase a number. A common example is a scale from “Strongly agree” = 3 to “Strongly disagree” = -3.
Answer by picking a number on a scale
For example, “On a scale of 1-10, how memorable is this logo?”
There’s one situation in which you don’t have to avoid unordered categories: when there are only two possible answers. In such cases, you can assign one answer the value ‘1’ and the other ‘0’. The mean then tells you the percentage of 1s.
For example, asking a ‘yes or no’ question where Yes = 1 and No = 0.
Then, when you’re finding the average, you can find either the arithmetic mean or the median. Both are fine methods and there is no consensus about which is better - they are different ways of representing the ‘center’ response.
5. A four-step plan for consulting crowds
Putting all of this together, you get a simple four-step plan:
Step 1: Pick an appropriate question
It should either have a single objective answer, or be about what some group or individual will think or feel about something.
It should not be about very obscure facts or be highly technical (unless your crowd is a crowd of relevant experts)
It should not be typically affected by systematic biases
It should be possible to calculate the average of the answers
Step 2: Ask an appropriate crowd
Consider: friends and loved ones, social media, Positly.com, existing studies, or even yourself.
Try to make sure the way you select your crowd doesn’t introduce systematic biases.
Try to ensure diversity of opinion and independence (don’t let people influence each other’s answers).
If you weaken diversity and independence by allowing discussion, try to ensure discussion is decentralized.
Step 3: Take the average
The arithmetic mean or median are fine. There is no consensus about which is better, and both perform well.
Step 4: Use that as your answer
Reflect on how likely your crowd is to have any relevant systematic biases and proportion your confidence in the result to your confidence in there being no such biases. If you’re very unsure, treat the result cautiously; if you’re confident, take the result as a strong signal.
You now have a basic strategy for gaining insights from crowds. We hope you’ll use it to improve your judgments and life outcomes.
Finally, if you’re someone who likes to know why things work (not just that they do), the next section gives a simple explanation of the surprising math behind why crowds consistently give more accurate judgments than individuals (even individual experts!). Whether or not you keep reading, we’d still love for you to rate this article (via the box of emojis, at the end of this article) or check out the links to our other works (videos, podcasts, and positly.com).
6. Optional section: Why are crowds (sometimes) wise?
If it’s not immediately obvious to you why the average of lots of judgments could be better than any one judgment, then the wisdom of crowds can seem like magic. After all, as the researcher who coined the phrase ‘the wisdom of crowds’ (James Surowiecki) pointed out: If you calculate the average time it takes for a group of runners to complete a race, you get a mediocre result - not an excellent one. Averages are usually not the ‘best’ value in any given set. But the wisdom of crowds is not magic. In reality, it has a straightforward statistical explanation.
Historical background
Many of the insights in this article can be traced back to the time Francis Galton (an English polymath with a very dark history) tried and failed to provide experimental evidence that democracy produces poor judgments. Galton ran an experiment in which 787 people tried to estimate how much a particular ox would weigh after it had been slaughtered and ‘dressed’. Galton reasoned that:
“The average competitor was probably as well fitted for making a just estimate of the dressed weight of the ox, as an average voter is of judging the merits of most political issues on which he votes, and the variety among the voters to judge justly was probably much the same in either case.”
Whether or not you think this reasoning stands up to scrutiny, the result of Galton’s experiment has had profound consequences. According to his calculations, the median estimate of the competition participants was 1207 lb, and the correct weight was 1198 lb. This means that, when their results were averaged, the crowd was only off by 0.8%!
What’s even more remarkable is the fact that, much later (in 2014), some researchers revisited Galton’s notes and discovered that, after they corrected a small transcription error, “the [crowd’s] mean estimate has zero error” - it was exactly correct - none of the individual people outperformed the wisdom of the crowd!
Galton concluded his paper with a (presumably begrudging) acknowledgment that he was surprised by his findings, saying: “The result seems more creditable to the trustworthiness of a democratic judgment than might have been expected.”
The phenomenon Galton uncovered is now referred to as ‘the wisdom of crowds’ (sometimes ‘collective intelligence’) and is characterized by situations in which “the average value of multiple estimates tends to be more accurate than any one single estimate.”
Despite how counterintuitive it might seem, the wisdom of crowds has been widely studied since at least 2004 and it really does appear to replicate reliably, in many studies, across a wide range of contexts. For example:
Why it works
Here’s why it works. When any person in a crowd makes a judgment about a factual issue, you can think of that judgment as having three components:
‘Signal’. The reliable part of the judgment, which leads them towards the correct answer (such as the true weight of the ox).
Individual random noise. Each person will have idiosyncratic error in their judgment that is unique to that person (e.g., some people might have a tendency to overestimate weight estimates while others have a tendency to underestimate, and even from day to day the same person may give different estimates due to irrelevant factors).
Systematic bias. Ways that the people as a group tend to overreact, underreact, or take into account information that's irrelevant (e.g., if people systematically thought that oxen weigh more than they really do because of their muscular builds).
Oversimplifying a bit for clarity, you can write each person's estimate using an equation like this:
Estimate = Signal + Individual Random Noise + Systematic Bias
The remarkable result is that if you average together enough people's estimates, then the “Individual Random Noise” part disappears. Since it’s truly random at the level of each individual, it's positive for some people and negative for others, and it cancels out as you average across people. When the number of people gets large enough, that leaves you basically just with:
Average Estimate = Signal + Systematic Bias
What's amazing about this is that as long as Systematic Bias is small (i.e., there aren't significant biases in the human mind or the way the group was selected that cause all or most people in the group to be wrong in the same direction), the wisdom of the crowds will work! That is, the average estimate of the group will be almost entirely signal and therefore be close (or equal) to the true answer. The most fascinating piece of this is that this process still works even if individual people are very inaccurate, since that just contributes to random noise, which cancels out when you have lots of people. The only thing you have to avoid is systematic bias (reasons that people’s answers tend to be wrong in the same direction).
When it comes to the weight of oxen, Galton's data suggest this works beautifully - individual people are often wrong, but there's no systematic bias in how they are wrong, so the wisdom of crowds succeeds.
Since any single expert is likely to have a bit (or maybe quite a lot) of random noise in their judgments, a group of non-experts that lacks systematic biases in one direction often makes even better estimates than any single estimate by an expert (on certain questions)! The apparent magic of the wisdom of crowds is nothing to do with mystical insight; it’s just that the act of averaging reduces random noise more reliably than any one person is usually able to in a single judgment.
As we’ve discussed in earlier sections, this also relies on judgments being independent (or, if not, discussion being decentralized). If the people making judgments were all copying the answer of the person they trusted most, the method wouldn't work. We need the individuals making judgments without reference to each other.
This doesn’t just apply to factual issues
The discussion above describes why crowds can reliably give more accurate answers to factual questions than individuals. Finally, it’s worth noting that the math behind the wisdom of crowds is extremely similar to the math that explains why we use surveys to investigate how populations of people think and feel about things.
Imagine you want to know how much a product will appeal to people in the US. You could rely on one individual judgment (perhaps your own best guess, or the single assessment of a single expert), but you’ll tend to get a more accurate answer if you ask a bunch of people in the US how appealing they find it and average their responses. Here we can apply a very similar equation to what we had before:
Survey Estimate = True Population Average + Individual Random Noise
+ Systematic Bias
What you want to know is the true population average for all US adults. Each individual person's response involves Individual Random Noise (reflecting how their judgments differ from the average and how their responses vary day to day). And in this context ‘Systematic Bias’ refers to the ways that the sample of people you're surveying are non-representative of the whole US adult population. For instance, if your survey only contains 18-year-olds, and 18-year-olds tend to give different answers than other age groups, there will be substantial Systematic Bias when you try to apply these results to the whole US adult population.
Just like the wisdom of crowds example, when you average the responses of a large number of people, Individual Random Noise cancels out, and you're left just with:
Survey Estimate = True Population Average + Systematic Bias
So long as your sample of people is not too systematically biased (with respect to the question you are asking), your survey then answers the question of interest (e.g., what is the average level of interest in this product or service).
7. Optional section: Complex ways to boost accuracy
Some studies find that you can increase the accuracy of crowd judgments by identifying experts within the crowd and averaging only their responses.
Other studies suggest giving different weights to different members of the crowd (based on things like expertise or confidence) in the averaging calculation.
There’s also a fascinating, more recent development that involves asking people your question and asking them to predict how other people in the crowd will answer your question. Then you select as the answer, whichever option is (by the biggest margin) more popular than people predict.
We don’t have the space to explore these techniques here, but they do appear to increase accuracy even further. However, they come with additional complexity. If you have the desire to embrace that complexity in the search for greater accuracy, then we say go for it! If you don’t, well, we’d just remind you that mathematical modelling of the wisdom of crowds provides evidence that simple averages perform surprisingly well and are reliably better than consulting just one person.
***
We hope this article helps you to get insights that improve your judgments and outcomes. Crowds are a source of helpful information that most people neglect. Now you can be an exception.
If you enjoyed this article, please consider checking out our sister project, Positly.com. It’s a participant recruitment platform that helps you quickly find high-quality survey respondents, with built-in protections against bots, spam, and AI-generated data.
