top of page

Decision Advisor: a randomized controlled trial of a decision-making tool

Updated: 7 days ago


I. Introduction


Decision-making is essential for personal and professional development, problem-solving, and navigating the complexities of life. Improving decision-making skills can lead to better life choices and overall well-being. This is why Clearer Thinking developed a digital program aimed at improving decision-making.  


Decision-making is a complex process influenced by a wide range of factors. There are cognitive factors like the quality and quantity of information available, as well as biases and heuristics. More information can lead to better decisions, but it can also lead to information overload. Different people may interpret the same information differently due to individual differences in perception.



II. Decision Advisor: The program overview


At a high level, the decision-making program was designed to aid with decision-making by helping users:


  1. Brainstorm options for what to do for the current decision that they may not have considered

  2. Consider other sources of information they may want to look into later that could help with their decision

  3. Understand and reflect on cognitive biases that are relevant to and might negatively impact their decision

  4. Go through a process of estimating the expected value of different options (relative to each other) to help evaluate which is best


The full description of the program can be found in Appendix A.



III. The theory of maximizing expected value


The theory of maximizing expected value is a fundamental concept in decision theory and probability theory. It provides a framework for making decisions when faced with uncertainty. In this theory, decision-makers choose the option that will yield the highest value outcome on average. Portions of our decision-making program are modelled on the expected value framework.


The theory involves:


1. Identification and clear definition of the decision: A decision typically involves choosing from a set of options, each with associated outcomes. In our program, the decision was selected at the beginning, and the potential options were brainstormed by the user shortly after that.


2. Assessment of probabilities: For each alternative, probabilities are assigned to different outcomes conditioned on each option being selected. These probabilities represent a degree of belief or expectation about the likelihood of each outcome occurring if each option is chosen. Participants were asked to assign such probabilities in the pro/con evaluation section of our program.


3. Calculation of expected values: For each alternative, the expected value is calculated by multiplying each possible outcome by its associated probability and then summing up all of the products. This calculation provides a numerical representation of the average value that can be expected from an outcome. Our tool carried out this calculation automatically using the inputs provided by the user.


4. Comparison of expected values: After calculating the expected values for each alternative, the alternatives can be compared to each other. The alternative with the highest expected value is considered the best choice according to the theory of maximizing expected value. In our program, the expected value of the options was displayed on the screen in a bar chart form.


This decision-making approach is widely used in various fields, including economics, finance, engineering, and risk management. It assumes that individuals make decisions rationally by weighing the potential outcomes and their probabilities, aiming to maximize their expected gains or utility. Clearer Thinking prepared a tool based on this theory, and this report describes how we analyzed the effectiveness of that tool.



IV. The study outlines and research design


Study participants were recruited on Positly (our platform facilitating fast research participant recruitment). They signed a consent form. During the screening process, they were asked whether they had a decision to make and if they would know the decision’s outcome in 6 months or less. Only participants who indicated that they did have the decision to make and that they would know the outcome of the decision within 6 months were invited to participate in the study. Participants were additionally asked about their ethnicity, political inclinations, and their SAT and ACT scores. (These were decoy questions to help ensure participants were not aware that this was a screener for a particular purpose to avoid creating an incentive to lie.) 


After being admitted to the study, participants were randomized to be in either the control group or the intervention group. The intervention group was asked to make a decision using the decision-making program described above, while the control group was only asked (1) to indicate what decision they had to make, (2) to indicate how many options they were considering for this decision, and (3) to say which option they thought was best for them to take in the decision. 


Before and after making a decision, both groups were given surveys to fill in. The surveys recorded various aspects of participants’ psychological profiles. We also recorded participants’ personalities measured by a standard Big5 IPIP test. All variables measured before and after the decisions were made are listed in Appendix B.


After making a decision, people declared the date by which they would know how the decision would have turned out. We followed up with them at that time and asked them to fill in a survey measuring their levels of satisfaction and their level of regret related to their decision. We used a previously validated decision satisfaction scale by Lawama, Greenberg, Moore, and Lieder. It included 3 decision satisfaction items recorded on a Likert scale and 10 regret items recorded on a Likert scale. The responses for these two dimensions (satisfaction and regret) were used to compute a combined decision score that was used as a primary dependent variable. We did it this way so we could later see what was deciding about how participants felt about their decisions. 


The total decision score was calculated as:


total_decision_score = decision_satisfaction - decision_regret



V. Study participants


We screened 994 people on Positly. Of those, 917 people were eligible to participate: 423 men, 490 women, and 2 nonbinary. Participants were only eligible to be included in the study if they (1) had a decision to make and (2) would know how the decision would turn out within 6 months.  Six people reported technical problems which did not affect the study coherence. 
    


In total, 381 completed the study protocol up to the point of making a decision: 205 in the control group and 176 in the intervention. Out of these 381 people, 194 provided us with the follow-up information in 6 months: 95 people from the control group and 99 from the intervention group. 





VI. The primary outcome: total decision score



After collecting feedback from the study participants, we compared how satisfied and regretful they were about their decisions. 


Much to our surprise, the total decision score (i.e., the combined satisfaction and regret scores: decision_satisfaction - decision_regret) turned out to be higher in the control group than in the intervention. The control mean was 2.38, and the intervention mean was 2.0!





The difference in total decision scores between the intervention and the control was statistically significant (T(193)=2.01, p-value<0.05). While this result is statistically significant, p-value < 0.05, suggests a possibility that the resulting difference may be due to chance. However, given the evidence we have, we think the most likely conclusion is that the decision program caused people to be less happy with their decisions and that the result is not just a statistical fluke. 


The difference between the groups is displayed on the graph below: 





Why was the control group happier with their decision?


To find out what was responsible for such unexpected results, we examined our scales from a few different angles. We checked how groups differed on both of the sub-scales (for decision satisfaction and decision regret). We also examined every item on the scale independently to look for deeper insights into why the control group appeared to be happier with their decision than the intervention group.
 


We discovered that the main factor behind the difference was decision regret. The difference between the decision satisfaction score between the control and intervention groups was not statistically significant (T(193)=1.51, p-value=0.13). The mean satisfaction for the control was 3.22, and the mean satisfaction for the intervention was 3.04. 




The difference between the decision regret in the control and the intervention groups, however, was statistically significant (T(193)=-2.29, p-value=0.02). The mean decision regret in the control was 0.84, and the mean decision regret in the intervention was 1.04.




Since we recorded many variables that could affect the decision score at the start, one of them was regret tendency. We checked to see if people in the intervention group had a stronger prior tendency to experience regret than the control group (before they completed the program). It appeared that the mean of the regret tendency in the intervention was higher at the start, but the difference was not statistically significant (T(193)=-0.6, p-value=0.55, Control = 0.78, Intervention = 0.83).

To understand better which specific items on the total decision score survey indicated differences between groups, we also analyzed each item separately. The differences between the two groups’ responses to most of the survey’s 13 items (when examined individually) were not statistically significant except for the following:


“I postponed this decision for too long.”

(T(193)=-2.55, p-value=0.01, Control = 1.1, Intervention= 1.53)


As indicated by the responses to the above question, the control group procrastinated less about making their decisions according to the self-assessment of study participants. This suggests that the decision-making program somehow caused intervention group participants to delay their decisions for longer than the control group and that this may have had detrimental effects.



VII. Other factors affecting the total decision score


Exploring other potential reasons why people in the intervention group were less happy with their decisions than the control, we examined other variables in the program. Most of them didn’t point us in any specific direction, although we did discover some interesting facts presented in the chapters below.   


In the intervention group, 22% (39 people) selected an option they didn’t consider before doing the program and the remaining 78% of the intervention group (137 people) selected previously considered options. We have not measured this for the control. This suggests that the program may have made participants consider options they might not otherwise have considered (or, at least, hadn't previously thought about). We are not sure yet whether it is a good thing or not. 



CHANGING THEIR MIND DURING THE PROGRAM


In the intervention group, people indicated which option they were "leaning toward" just before working through the portion of the tool that asked them to consider the pros and cons of each option. 30 out of 95 people (32%) changed their minds during the pros-and-cons portion and decided (by the time they reached the end of the tool) to go with a different option than they had picked right before the pros-and-cons portion.


We measured whether or not people who changed their minds during the pro/con part of the program had lower total decision scores than people who stuck to their first choice, but there was no statistical difference between these two groups: T(99)= 0.14, p-value=0.89. (The mean total decision score for people who changed their minds was 2.03 and for people who didn’t change their minds was 1.99.)


Whether or not they changed their minds also has a very low coefficient in the regression predicting total decision scores (0.006), which provides another reason to believe that people changing their minds during the pro/con part of the program didn’t cause them to be less satisfied with their decisions. 


DECISION CONFIDENCE


We ran a regression to test whether or not any of the variables recorded inside the program were predictive of participants’ total decision score, but the regression's R² was very low (ridge R² < 0.006; lasso R² < 0.04). Only one variable was found to be meaningfully predictive of the total decision score, which was the level of confidence participants felt about their final decision at the end of using the tool (i.e., participants’ responses to "On a scale of 0% to 100%, how confident do you feel now that you've chosen the best available option for this decision?"), with a standardized coefficient of 0.8. This means that people who felt confident about their decisions were happier about them later. One possible explanation for this could be that people might be more confident in decisions where it is easier to figure out the best option.  The list of variables included in this regression are listed in Appendix D. 


In the intervention group, the reported confidence in the decision increased from mean=70.05 (before using the program) to mean=77.28 (t(94)=-2.82, p<0.005) after the intervention was completed. It seems that the program made people feel more confident about their decisions, even though long-term they were less satisfied with them. We later discovered that it is the people who did not increase their confidence were responsible for lower decision scores in the intervention group. 


FINDING DECISIONS EMOTIONALLY UPSETTING, WITH HIGH INVESTMENT, AND MANY CONSEQUENCES


We also tested if people with different paths selected in the program differed in the total decision score, but they did not. For example, we checked if people who said that their decisions were emotionally upsetting had different levels of decision scores than the ones who said their decisions were not emotionally upsetting, but the scores were not different. (T(98)=1.53, p=0.13, the mean for the upset group = 1.8, the mean for the not upset group = 2.2.) Similarly, people who declared that they invested a lot of resources didn’t differ in their decision scores from those who said they did not invest much.  (T(98)=0.55, p=0.59, the mean for the invested group = 1.93, the mean for the not invested group = 2.08.) Participants' decision scores also didn’t differ based on the number of consequences the decisions had. (T(98)=-0.61, p=0.54, the mean for a single consequence = 2.11, the mean for many consequences = 1.94.)



VIII. Correlations of specific items with the total decision score


MISSED OPPORTUNITIES


Besides our main dependent variable — the total decision score — we measured three extra variables. We wanted to know if the decision made caused any opportunities to be missed. So we added an item: “I missed out on opportunities because of my decision.” The Pearson correlation coefficient between this missed opportunities item and the total decision score was -0.5; so it was (perhaps unsurprisingly) a factor strongly associated with a negative total decision score (p<0.00001). 


HINDSIGHT


We also checked if participants felt that with hindsight it was obvious what the right decision was. It was measured by the item: “In retrospect, it should have been obvious what the best choice to have made was for this decision, even if it didn't feel obvious at the time.” The Pearson correlation coefficient for this item and total decision scores was -0.02; so the association of these two variables was negligible.


REVERSIBILITY


The last item that we added measured if the decision was reversible: “At this point, it would be very hard to reverse my decision.” The Pearson correlation coefficient for the reversibility of the decision and the total decision score was -0.003; so again, the association of this variable was negligible.


The Pearson correlation coefficient for the group assignment and total decision score was -0.14, which illustrates the fact that the intervention group was less happy with their decisions. 



IX. Factors predictive of the total decision score


As previously mentioned, we asked our participants to fill out a number of questions both before and after they made their decisions. We wanted to know if there were any psychological factors that could be predictive of who was happiest with their decisions and if any aspects of the decisions themselves were predictive of the total decision score.


We first ran a regression to check if any of the 36 factors describing our study participants and their decisions predicted the total decision score. The variables were first normalized by transforming them into z-scores (i.e., subtracting the mean from each column, and dividing by its standard deviation). A few missing values (not more than 7 so 3.6% of the total study sample) in relevant columns were filled with the mean value of the column. We used this method because it is simple and with such a low number of missing values, the method of filling in missing values has little impact. We tested the performance of both Ridge and Lasso regressions. Ridge scored R²=0.15 in predicting the total decision score (i.e., it accounted for 15% of the variance), and Lasso scored R² = 0.12. The variables included in these models had the following coefficients for predicting total decision score (with higher numbers meaning a positive association with people rating their decision as going better):


Variable

Lasso

Regression Coefficient

r2=0.12

Ridge

Regression Coefficient  r2=0.15

Description (how it has been measured)

Correlation with total decision score

personalFreedom

0.18

0.11

“I have total personal freedom: I am free to do whatever I choose to do.”

0.32

identityDecision

0.18

0.13

“The choice I picked for this decision reflects the kind of person I am better than the other choices.”

0.26

agreeableness

0.11

0.08

Agreeableness measured by a standard Big 5 test

0.28

optimism

0.05

0.06

“How much are you the sort of person who is typically optimistic about the future?”

0.27

pragmatism

0.05

0.06

“Where would you rate yourself on the scale from being an idealistic person to being a pragmatic person?”

0.09

selfEfficacy

0.02

0.05

“Do you believe you can do anything you set your mind to?

0.3

conscientiousness

0.01

0.03

Conscientiousness measured by a standard Big5 test

0.24

stability

0.01

0.04

Emotional stability measured by a standard Big5 test

0.25

statusquo

0.003

0.05

“Did the choice that you ended up picking for this decision involve sticking with the status quo or default option, or did it instead involve implementing a change?”

0.25

educationScore

0

0.01

Education level

0.03

numberOfOptions

0

-0.04

“For this decision, how many options were you choosing between?”

-0.05

wantShould

0

-0.03

“This decision involves a conflict between what I actually want to do and what I feel like I should do.”

-0.15

reversibility

0

-0.01

“How reversible was the decision you made?”

-0.00001

extraversion

0

-0.03

Extraversion measured by a standard Big5 test

0.12

openness

0

0.01

Openness to experience measured by a standard Big5 test

0.13

similarityOfOptions

0

-0.03

“How similar to each other were all of the options you came up with for this decision?”

-0.04

daysMeditating

0

0.001

“How many days a week do you meditate for 5 minutes or more?”

0.06

age

0

0.02

Age

0.11

stressorLifeEventsSize

0

-0.02

“In the last 12 months have you experienced a change that significantly affected your well-being?”

-0.15

intuitionReason

0

-0.06

“Are you more the sort of person who trusts your intuition or that follows your reasoning?” (A higher score means they rely more on reason and less on intuition)

-0.11

femaleAs1MaleAs0

0

0.02

Female gender

0.05

householdIncomeScore

0

-0.03

Income level

0.02

conformism

0

-0.02

“When you make decisions, how influenced are you by other people's opinions?; How much do you tend to not act like yourself when you are surrounded by other people?”

-0.12

selfConfidence

0

0.02

“How self-confident are you?”

0.22

regretTendency

0

0.01

“When you make decisions, how much do you tend to regret your choice?”

-0.15

emotionalityRationality

0

0.01

“Where would you rate yourself on a scale from an emotional person to a logical person?”

0.06

reflectiveness

0

0.03

“Are you the sort of person who tends to act quickly on decisions or deeply reflect on decisions?”

0.01

minutesPerDaySmartphone

0

0.02

“How many minutes a day do you use your smartphone?”

-0.001

minutesPerDayGames

0

0.02

“How many minutes a day do you spend playing video games or mobile phone games?

-0.009

minutesReading

0

-0.04

“How many minutes a day do you spend reading?”

-0.12

negativeStressors

-0.02

-0.04

Negative items from the stressors’ list

-0.17

identifiesAsDepressed

-0.04

-0.05

“Do you believe that you are ‘depressed’?”

-0.29

GroupAssignement

-0.03

-0.05

They were assigned to be in the Intervention group (1) or the Control group (0)

-0.14

meYouHappy

-0.04

-0.05

“This decision involves a conflict between what makes me happy and what someone else wants me to do.”

-0.17

minutesListening

-0.07

-0.06

“How many minutes a day do you spend listening to podcasts, audiobooks, radio, or other audio content?”

-0.16

totalDecisionImportance

-0.08

-0.06

The number of important decisions they have to make in their life right now and their intensity measured by a test

-0.22


We'll focus here on the variables that had non-negligible coefficients in both the ridge regression and lasso regression. 


It seems that people tended to be happier with their decision ultimately if the option they chose was aligned with their identity (as reflected by greater agreement to the statement “The choice I picked for this decision reflects the kind of person I am better than the other choices.") This is consistent with research on self-identify’s role in the decision-making process, but it is also one of the most interesting findings of our study. 


People were also happier with their decisions if they had a higher degree of personal freedom (as reflected by higher levels of agreement to the statement “I have total personal freedom: I am free to do whatever I choose to do.”).


Other factors predictive of decision satisfaction included some positive personality traits like agreeableness, optimism, pragmatism, emotional stability, conscientiousness, and self-efficacy (faith in one’s abilities). 


Choosing the status quo instead of a change was also slightly predictive of making satisfying decisions, but the coefficient was small (0.003). 


Belonging to the intervention group turned out to be a negative predictor of the total decision score (as expected based on results discussed earlier). This further supports the findings reported in the sections above. 


Other negative predictors of total decision score included depression, number of negative stressors, and making decisions out of a sense of obligation to someone else rather than out of a participant’s genuine desire.


We measured how many decisions in different aspects of life people had to make at the time of our study and how important these decisions were. We measured these by asking if they had a decision to make in each major aspect of life (health, family, relationships, finances, job, living arrangements); and then for each aspect, we asked participants to assess how important these decisions were on a 4-level Likert scale. A higher number of decisions to make was predictive of worse decision scores, supporting the idea that people need a certain amount of cognitive resources (energy and attention) to make satisfying decisions and that when they have too many decisions to make, they may decide less well. 


The last negative predictor of low decision scores (which was not, however, as strong as the strongest positive predictors) was the number of minutes of listening to audio content per week. It is hard to explain why this might be the case. This result is puzzling and needs to be considered in future research. 


In summary, we can say that people with positive personality traits, a high degree of personal freedom, and less cognitive overload tended to be happier with their decisions. Completion of the decision-making program did not enhance their total decision score and appeared to have the opposite effect. 



X. Qualitative analysis 


In addition to the quantitative questions described above, we also asked participants qualitative (open-ended) questions. They were collecting information on what types of decisions people were making. The responses to these questions were coded and interpreted by one researcher; and from that work, 15 thematic categories emerged. (Each answer could belong to one or more categories depending on its content.) The number of open-ended responses that fell into each of the 15 categories are shown in this table:


Theme

Total count

Count for the intervention

Count for the control

work choices

48

25

23

relocation

17

11

6

money / investments

33

11

22

medical choices

16

10

6

buying a house or renting

21

9

12

family decisions

24

9

15

home or car upgrade

16

7

9

romantic relationships

11

5

6

change of habits

8

5

3

business decision

8

4

4

lack of any decision

7

4

3

education

4

2

2

holidays

6

2

4

personal values

1

1

0

politics

1

0

1


Among the themes we discovered in our qualitative material, work choices emerged as the most prevalent, with 48 instances where individuals considered different options of career paths and professional trajectories. Financial considerations were common, as 33 decisions revolved around money and investments. The prospect of relocation occurred in 17 instances and buying a house or renting in 21. 16 people made decisions about their medical treatment. Family considerations came into play 24 times, while romantic relationships only appeared 11 times. So it seems that our tool was used less often for more subjective decisions.


Our study participants also tended to use the tool more often to make decisions that had more objective rather than subjective outcomes. This may be because of the original phrasing when they were asked to write the decision they were considering, which was: "What's a decision you are considering where you'll know within 6 months how well it turned out?" Perhaps it’s more difficult in a limited amount of time to tell how “well” decisions with more subjective outcomes turned out. 


The number of cases in each category for the intervention and the control differs somewhat, but they follow a similar pattern. Because participants chose what decision to focus on before any differences occurred between the control and intervention groups, we would not expect any systematic differences between the two. 


We also qualitatively categorized responses each participant gave in response to the question "Please describe how this decision worked out for you" into "positive", "neutral" and "negative" based on how well it appeared that the decision turned out for the participant.



Count for the intervention

% for intervention

count for the control

% for control

Z test

p-value

Positive outcomes according to the researcher

50

50%

59

62%

-1.687

p=0.09

Neutral outcomes according to the researcher

37

37%

29

31%

0.884

p=0.39

Negative outcomes according to the researcher

13

13%

7

7%

1.391

p=0.17

Total

100

100%

95

100%




Qualitatively, it appeared that participants' evaluations of how well their decisions turned out often depended on factors they couldn't control. But if these factors went in their favor, they were happy with the outcome; and if they went against them, they were unhappy. Some decisions involved the behavior of others, and that created additional factors beyond the control of participants. A potentially useful feature that the intervention could have included (but did not) is a submodule teaching participants the difference between things they can and cannot control, and the psychological benefits of focusing on what you can control rather than what you can't.


One challenge with evaluating outcomes is that it is hard to separate how well things turned out objectively (e.g., were the concrete outcomes what a person wanted) compared to how someone felt about how things turned out (e.g., were they looking for a silver lining in an otherwise bad situation). Some people tend to look at the worst in a good situation, while others look at the best in a bad situation; and these discrepancies between the objective and subjective outcomes made evaluating whether outcomes turned out positively, neutrally, or negatively especially difficult. 


The results of the decisions, beyond just being positive, neutral, or negative, also differed in the intensity of outcomes, with some outcomes being very good and some very bad.  


The decisions being made by participants also differed a great deal in how difficult those decisions were. Example decisions included: buying a house, upgrading an electrical panel at home, eating healthier, and getting a divorce.



XI. Exploration of the intervention program


We investigated all elements of the intervention program, trying to find out, what might have caused the intervention group to be less satisfied with their decisions compared with the control. 


We ran a regression predicting the total decision score based on various elements of the program, and we obtained the following results:  



Variable

Ridge

Regression Coefficient

(r^2=0.02)

Lasso

Regression Coefficient (r^2=0.04)

Description

Correlation with total decision score

finalconfidence

0.075

0.09

Confidence in the decision at the end of the intervention program

0.36

confidenceIn

Decision

0.06

0.04

Increase in confidence measured by a question asking participants at the end of the program, if they confidence changed where 1 meant increased, -1 meant decreased and 0 meant stayed the same

0.29

change_in_confidence

0.06

0.02

The difference between the confidence prior to pros and cons and the final confidence

0.23

hadNotPreviously

ConsideredTaking

ThisOption

0.04

0

If the option selected was thought of during the use of the program or was it the option considered earlier

0.1

persontodiscusswith

0.004

0

Whether participants planned to discuss the decision with someone

0.03

changedmind

0.003

0

At the start of the program participants declared which option they were most likely to pick and this variable of a record of whether they ended up with this option as their final choice

0.01

extrainfo

0.001

0

Whether participants planned to seek extra information

0.01

invested_ornot

-0.0005

0

Whether participants were highly invested in the decision

-0.06

a

-0.005

0

The number of possible good qualities of a decision option

-0.03

b

-0.005

0

The number of possible bad qualities of a decision option

0

choseAsFinalOptionOneFromNarrowFraming

Exercise

-0.01

0

Picking an option from the narrow framing exercise

0.02

picked_recommended

-0.02

0

If a participant picked the option recommended by the program

0.03

manyconsequences_ornot

-0.02

0

If the decision had many consequences


upsetting_ornot

-0.05

0

If the decision was upsetting

-0.15

numberOfOptions

-0.06

-0.05

The number of options they chose from

0.13



We obtained the optimal result for Lasso with r2=0.04. It showed that two factors were very slightly predictive of total decision satisfaction: final decision confidence and the number of options. It seems that the more options participants listed, the less likely they were to be satisfied with their decisions. It may be linked to final decision confidence — the degree to which participants believed that they made the right choice — because having a larger number of options from which to choose may dilute a person’s overall confidence that they’ve chosen the right option, which may be a sort of post-decision form of decision paralysis. Minding that r2=0.04 for this regression, we conclude that the predictive power of these factors is rather limited. If we remove most of the tested factors from this regression and leave only final decision confidence and change in confidence, the r2  for both Ridge and Lasso = 0.08. (For Ridge: finalconfidence = 0.19, change_in_confidence = 0.12; for Lasso finalconfidence = 0.24, change_in_confidence = 0.13.)


We also investigated if people chose the option the program recommended to them. It seems that in the majority of cases (71 out of 99, or 72%), participants decided to do what the program told them to do. This might have led them to ignore their natural instincts and/or to feel a smaller sense of ownership or agency in the decision, which may in turn have resulted in lower decision satisfaction. 


We believe that the final part of the program, which included comparing the pros and cons of different options, was not responsible for program users being less happy with their decisions than people in the control group. We came to this conclusion because final decision confidence was the main positive predictor of total decision satisfaction, and decision confidence went up during the pros and cons part of the program. (In the intervention group, the reported confidence in the decision increased from mean=70.05 prior to using the pro/con section of the program to mean=77.28 (t(94)=-2.82, p<0.005) after that section was completed.)


We additionally tested the hypothesis that the pro/con section could have caused people to be less happy with their decision by subtracting the prior decision confidence from the final decision confidence and correlating this change in confidence with the final decision score. We found that it was positively correlated (r2=0.23, p-value<0.02). So, the pro/con section of the tool increased people's confidence in their decision on average, and an increase in confidence in the decision was associated with ultimately being happier with their decision at the follow-up point. It is possible that people who became less confident in their decision after using the pro/con section of the tool were caused by the pro/con section to have lower total decision scores. However, even if this was the case, it represents only a very small portion of study participants and therefore is very unlikely to explain the effect of those in the intervention group having lower total decision scores (only 24 out of 99 participants in the intervention group ended up with a lower confidence after using the pro/con portion than before using it).

 

In total, 75% of participants recorded an increase in their decision confidence. The mean change was 7.23 with std=14.28.





We wanted to check if there was a significant difference between the total decision scores of people whose confidence increased and people whose confidence didn’t increase. To do that we divided the intervention group into 2 subgroups: a group where people’s confidence didn’t increase more than 5 points and a group where people’s confidence did increase more than 5 points. (It would have been more intuitive to split the group across 0; but if we had done that, we would not have had enough observations in the no-change-of-confidence group to calculate a comparison.) We had 51 observations in the first group and 61 observations in the second. The mean total decision score in the first group was 1.67, and the mean total decision score in the second group was 2.26. The difference between these two groups was statistically significant with T(98)=2.58 and p-value=0.011. Hence, participants who increased their confidence in their decision by more than 5 points during the pro/con portion ended up happier with their decision at the follow-up point than those who didn't have at least a 5-point increase in confidence.


We also compared both of these groups to the control. There was no difference between the group where confidence increased and the control: T(155)=-0.54, p-value=0.6. There was, however, a big difference between the group with no confidence increase and the control: T(145)= -3.39, p-value<0.001. It seems that people whose confidence in their decision didn’t increase were probably responsible for lower total decision scores in the intervention group than in the control group!


We also measured if people who changed their minds during the pro/con part of the program had lower total decision scores than people who stuck to their first choice; but there was no statistical difference between these two groups: T(99)= 0.14, p-value=0.89. (The mean total decision score for people who changed their minds was 2.03 and for people who didn’t change their minds was 1.99.)


Whether or not they changed their minds also had a very low coefficient in the regression predicting total decision scores (0.006), which suggests that changing people’s minds during the pro/con part of the program didn’t cause them to be less satisfied with their decisions. This suggests that the pro/con portion of the intervention was not causing participants to have lower total decision scores; because if it was, we would expect those who changed their mind between the beginning of the pro/con section and the end to be the ones who ended up with lower total decision scores.



XII. Conclusions


A. The reasons the program didn’t work


The Decision Advisor program included these four components:


  • brainstorming new options for the current decision

  • considering other sources of information that could help with the current decision

  • understanding and reflecting on cognitive biases that are relevant to the current decision

  • estimating the expected value of different options with the pro/con tool


In this study, we identified several factors related to decision satisfaction and decision regret. However, the decision-making program that we investigated did not improve them. 


It seems that the four components included in the program made people somewhat more confident with their decisions at the time of making them, but did not make people happy with their decisions in the long term, and may have led to more missed opportunities in the future. 


It is hard to be sure why the program didn’t work to improve people's happiness with their decisions and may have even made them less happy with their decisions. Here are a few speculative hypotheses about these results that could be investigated in future research:  


  1. Cognitive overload: The program was very extensive, and perhaps that created a cognitive overload in the study participants, which caused them to make worse decisions. (It took an average control participant 11.8 minutes to complete the program, and it took an average intervention participant 25.8 minutes to complete the program. This means that the intervention itself was about 15 minutes long.)

  2. Cognitive bias education backfiring: The intervention educated participants about cognitive biases that might be relevant to their situation by asking them detailed questions about the context surrounding their decision and then briefly discussing relevant cognitive biases based on their answers (e.g., if they said it was a decision where they had already invested into one option substantially, they were told about the sunk cost fallacy). Perhaps this cognitive bias education caused participants to make worse decisions, either because it nudged them away from good options (i.e., options that may have simply sounded like they related to cognitive biases) because it caused them to second-guess valid intuitions, or perhaps for some other reason. 

  3. A statistical fluke: While we found that the intervention performed worse than the control, the p-value was <0.05 so on the threshold of statistical significance. This means it's technically possible that this negative effect was random and that the decision program has no effects rather than negative effects. A p-value of 0.05 means that there is a 5% chance it was a random result and a 95% chance it was a meaningful result. 


Since p = 0.05 is the threshold for statistical significance and this p-value sits right on that border,  we find it difficult to rule out the possibility that the difference in overall decision scores between the intervention and control groups was the result of a fluke. That being said, based on the data we have collected and our desire to err on the side of caution, we believe it is more likely that the decision program caused people to be less happy with their decisions than that our result is due to a statistical fluke.


We have investigated all aspects of the program to find out what might have caused people to be less happy with their decisions after using the program. After thorough analysis, we ruled out the possibility that it was the pro/con section of the program. In any new versions of the program, we will include the pro/con section but remove the parts of the program that are not clearly efficacious. 



  1. Considering too many options: The intervention included a section on "narrow framing", where participants were asked to think about what would make a decision option good and bad, and then they were asked to generate at least one more option in addition to those they already listed. It is possible that this caused study participants to consider too many options, which reduced their ability to choose the best option or caused them to deliberate on their decision (rather than picking an option) for too long due to analysis paralysis or a "paradox of choice". This is potentially supported by the fact that those who received the intervention delayed their decision and were thus more likely to agree with the statement "I postponed this decision for too long" than the control group.We were able to obtain further evidence for this hypothesis by checking whether participants who ended up choosing as their final option (at the end of the tool) one of th