A randomized controlled trial of the Clearer Thinking ‘Daily Ritual' module.
INTRODUCTION
A habit is a type of behavior that has become automatic or routine. Our everyday life is full of habits, and much of what we do is driven by them. Habits can be desirable, like brushing our teeth twice a day or looking both ways before crossing the road, but they can also be undesirable, like smoking or compulsive spending. What if there were a tool that could help you more reliably create new, beneficial habits? Clearer Thinking set out to create such a tool (Daily Ritual: a habit creation system), and in this report, we discuss a randomized controlled trial that we ran to test the effectiveness of this tool.
Here is a quick summary of the results - below you can find a lot more
The Daily Ritual program increased the number of times people performed their desired habits weekly (compared to the control group)
The benefits of the program lasted at least 6 weeks
After 6 weeks it may be ideal to repeat the tool to continue getting benefits
Some factors measured in the study slightly increase the chances of a new habit formation but the effect is rather small
If you'd like to try the Daily Ritual program for yourself to see if it helps you form new habits, you can do so here.
RESEARCH DESIGN AND STUDY PARTICIPANTS
‘Daily Ritual' is a free, interactive web-based program that asks a user what habit they want to form and then walks the user through easy-to-implement psychological techniques aiming to improve a user’s performance in the formation of the new habit. Clearer Thinking has previously done pilot research on which techniques yield the best results, Daily Ritual implements all five of the habit formation techniques that were found to be most promising in that prior research:
listing habit benefits: a user is asked to list anticipated benefits that forming a habit will have for them
home reminders: a user is asked to put up a note somewhere in their home, reminding them about their new habit formation plans
support of a friend: a user is asked to write a short letter to a friend who could support their habit formation efforts
mini habits: a user is asked to perform a mini version of the chosen habit when the full version of the habit is not possible (e.g., if they don't have time to do the full version of the habit)
habit reflection: a user is asked to think about their previous experiences with forming a habit and reflect on how they can apply strategies to this habit formation goal that have proven useful to them in the past
We screened 939 people on Positly (our platform facilitating fast research participant recruitment). Of those, 596 people were eligible to participate: 307 men and 289 women. The inclusion criterion was a desire to form a new habit. This was checked by asking whether prospective participants wanted to form a new habit in the screening survey. We obtained informed consent from 404 of the eligible candidates and randomly allocated them to two even groups ( each had 202 people in them): one of the groups completed the Daily Habit Formation Program and the other group completed a short program that looked like a habit formation program but didn’t include any of the active components (i.e., no habit formation techniques). The intervention received the Daily Ritual tool (which walked them step by step through all 5 of the exercises mentioned above), and people in the control group were only asked to concretely describe the habit they wanted to form. Both groups received a variety of questions including ones about how they would measure success for the habit, and how motivated they were to form this habit.
Each week for eight weeks following that, we asked all participants how many days per week they performed the habit. We tracked the number of days a week when people performed their habits as planned (called ‘practice days’), the number of days a week when people did not perform their habits at all (called ‘failure days’), and the number of days a week when people performed a part (but not all) of their planned habits (called ‘incomplete days’). Each week, we also tracked "success rate" - which was determined by a question asking whether they felt that they succeeded with their habit formation plans. (The question asked: ‘In your opinion, how well did you accomplish your goal of forming a new daily habit?’ with an answer scale form: ‘Extremely well’, ‘ Very well’, ‘Somewhat well’, ‘Slightly well’, ‘Not at all well’.) Within eight weeks, some participants missed their feedback or dropped out, so the number of participants in the intervention group varies from week to week in the intervention group from 129 to 150 and in the control group from 136 to 154.
We also collected data on a variety of other exploratory factors to study whether they are predictive of successful habit formation. The list of variables included:
education
gender
number of children and children’s age
relationship status
place of living
depression
medical conditions
alcohol intake
body weight
the perceived amount of stress
self-predicted probability of success with creating the habit
the reported level of willpower
free time
habits previously formed
reading
smartphone use
need for stimulation
the expectation of fast results
fixed mindset as described in the theory by Carol Dweck
whether not starting a habit will have negative consequences or starting will have immediate benefits
whether the habit is aligned with their understanding of themselves and if they see the habit as a way to improve themselves
whether the habit is pleasant or an obligation
whether the habit is a priority
whether the habit is wanted intellectually or emotionally
RESULTS
We calculated the mean difference in the listed variables between the intervention group and the control group, and here is what we found:
The intervention group reported a larger number of days per week practicing their habit (mean = 4.15 days) than the control group did (mean = 3.54 days), and this difference (an extra 0.61 days of doing their habit week) was statistically significant (Student's t-test t(322) = 2.51, p < 0.013). Consistent with this, the intervention group also reported a smaller number of failure days (mean = 2.32 days) than the control group did (mean = 2.84 days), and this difference was statistically significant (Student's t-test t(322) = 2.31, p < 0.022). The intervention group has also reported a higher level of success (mean = 2.09 on a scale from 4 to 0) than the control (mean = 1.8) and the difference was statistically significant (Student's t-test t(322) = 1.05, p < 0.03).
There was no difference between groups in the number of days they partially (i.e., incompletely) practiced their habits (p<0.3).
The variable of days partially completing the habit had a low correlation with other independent variables (suggesting it may have confused respondents) and was later dropped from the analysis. The correlation coefficient between practice days and incomplete practice days was -0.21, while the correlation coefficient between incomplete practice days and failure days was 0.01 (in theory they should be highly correlated because the only options are to fully complete a habit, partially complete it, or not complete it at all, hence why we think study participants likely misunderstood what this question was asking).
If we look at the illustrations below, we can see that the number of days people practiced their habits is higher in the intervention group compared to the control group up to at least week 6 and possibly (though less convincingly) up to 8 weeks (and similarly, the number of days people failed at practicing their chosen habit is lower in the intervention group compared to the control group). After that, the difference begins to fade. At the same time, the intervention group seemed to be consistently happier with their results compared to the control group.
Here is what the difference between groups looked like a week to week:
Difference between practice days (t-test statistic=2.51, p-value=0.013)
Week number | Mean practice days per week (intervention) | Mean practice days per week (control) |
1 | 4.61 | 3.26 |
2 | 4.18 | 3.63 |
3 | 4.02 | 3.43 |
4 | 3.98 | 3.69 |
5 | 4 | 3.56 |
6 | 3.96 | 3.61 |
7 | 3.95 | 3.66 |
8 | 3.78 | 3.67 |
Total | 4.15 | 3.54 |
Difference between failure days (t-test statistic=2.31, p-value=0.022)
Week number | Mean failure days per week (intervention) | Mean failure days per week (control) |
1 | 2 | 3.27 |
2 | 2.38 | 3 |
3 | 2.39 | 2.88 |
4 | 2.22 | 2.61 |
5 | 2.41 | 2.86 |
6 | 2.58 | 2.64 |
7 | 2.45 | 2.54 |
8 | 2.54 | 2.63 |
Total | 2.32 | 2.84 |
The difference between groups in how successful they felt about their progress: T-test statistic=-2.12, p-value=0.035. (The question asked: ‘In your opinion, how well did you accomplish your goal of forming a new daily habit?’)
Week number | Estimated success (intervention) | Estimated success (control) |
1 | 2.13 | 1.45 |
2 | 1.95 | 1.74 |
3 | 1.93 | 1.7 |
4 | 2.09 | 1.91 |
5 | 2 | 1.76 |
6 | 2.1 | 1.97 |
7 | 2.1 | 1.96 |
8 | 2.02 | 1.94 |
Total | 2.09 | 1.8 |
The majority of people in the intervention group (163, or 85% of people responding) would want to recommend the program to a friend (versus 28, or 15% of people responding, who would not want that).
SENSITIVITY ANALYSIS
Out of 323 participants' responses, 63 contained missing values. (This means they were incomplete because some of the participants missed some feedback surveys). This constitutes 20% of the analyzed data. We carried out a sensitivity analysis filling missing values with 0 (instead of treating them as missing) in order to see if this would affect our results (i.e., treating any non-reported day as a failure to do the habit). There are differences between p-values between analyses performed on the dataset with missing values excluded and analyses performed on the dataset with missing values treated as 0, but it doesn’t change the overall conclusions: p-values for both ways of analyzing the data indicate statistically significant differences between practice days, failure days, and success rate, but not for incomplete days.
Testing the differences between the intervention and the control
| T-test statistic after setting missing values to 0 | P value after setting missing values to 0 | Means after setting missing values to 0 | P value | T-test statistic | Means |
Difference between practice days | -2.75 | 0.006 | difference: 0.6 control: 3.55 intervention: 4.15 | 0.013 | 2.51 | difference: 0.28 control: 3.27 intervention: 3.55 |
Difference between failure days | 2.87 | 0.004 | difference: -0.58 control: 2.9 intervention: 2.32 | 0.022 | 2.31 | difference: -0.36 control: 2.33 intervention: 1.97 |
Difference between success rate | -2.57 | 0.011 | difference: 0.32 control: 1.77 intervention: 2.09 | 0.035 | -2.12 | difference: 0.18 control: 1.6 intervention: 1.78 |
FACTORS THAT INFLUENCED HABIT FORMATION
In order to try to predict which variables are predictors of successful habit formation, we also ran a lasso regression that showed us a few factors that (very modestly) influenced the number of days that people practiced their habits, although the R² (i.e., variance in the outcome accounted for by the model) is low. Lasso regression allows us to investigate (simultaneously) whether the many factors we collected that we thought may predict habit formation success were actually predictive (lasso automatically sets to zero coefficients of variables that the model finds to not be sufficiently predictive). Furthermore, by using cross-validation to set the lasso's complexity parameter we are able to reduce the effects of overfitting that normally occur when testing many factors at once. The columns of the data were normalized before running the lasso regression (by subtracting the mean from each and dividing each by its standard deviation) to make the coefficients comparable to each other.
The lasso regression predicting the number of days when a habit was practiced each week had an R² value of 0.09 on the out-of-sample cross-validation folds - a measure designed to be less biased than the result on the training data (with an R² value of 0.18 on the training data), and the regression predicting how successful participants felt they were at forming their habit had an R² value of 0.1 on the out-of-sample cross-validation folds (R² value of 0.23 on the training data). Hence, this model had quite a limited ability to successfully predict how many days people would complete their habit or how successful they would feel about it. Nonetheless, there were some coefficients that were used by the regression that we may be able to learn from.
The variables we collected that were related to other habits (like how often the person works out or engages in meditation) were also predictive of successful new habit formation. This is probably not surprising (that being successful at a prior habit makes you more likely to succeed at a new one). It also is not clear what the underlying mechanism for this is. This is why we decided to run the analysis on the reduced number of variables where the variables related to habits were dropped out. The results of the analysis conducted on the full list of variables are attached beneath, in appendix 1.
The factors most predictive (in the lasso model) of the number of days habits were practiced were related to personality traits and overall mental health. These were:
level of energy and motivation (both general level of motivation and specific motivation to start a new habit)
emotional stability
openness to experience
emotional regulation skills (like reappraisal and suppression)
agreeableness
conscientiousness
Previously formed habits also correlated positively with how many days people practiced their habits, while the amount of personal freedom correlated negatively. Belonging to the intervention group was (as expected) also predictive but its coefficient was fairly low (0.05). Reporting a higher level of energy had a coefficient of 0.19 (meaning that a 1 standard deviation increase in reported energy levels produced an average of 0.19 more days of habit per week).
The factors most predictive of how people felt about their success at habit formation were also related to personality traits and overall mental health. The factors with the largest coefficients were:
motivation
openness to experience
emotional regulation skills (like reappraisal and suppression)
agreeableness
conscientiousness
support network
Previously formed habits and employment also correlated positively with how people felt about their success at habit formation, while the amount of free time, amount of personal freedom, and being in a relationship correlated negatively. Belonging to the intervention group was also predictive but its coefficient was relatively low compared to the others (0.3). We had one variable in the analysis that was hard to interpret: fixed mindset (the opposite of growth mindset) correlated positively with how people felt about their success at habit formation (the opposite direction one might expect) but this result might have occurred due to low R² and the fact that the fixed mindset was measured by a short survey involving only 3 questions.
The factors that were associated with new habit formation included:
Variable | Coefficient predicting estimated success (self-reported answers to the question ‘In your opinion, how well did you accomplish your goal of forming a new daily habit?’) | Coefficient predicting "practice days" (the number of days completing the habit each week) |
Motivation | 0.09 | 0.29 |
Openness | 0.09 | 0.02 |
Fixed mindset | 0.04 | 0 |
Reappraisal | 0.04 | 0.02 |
Suppression | 0.04 | 0.12 |
Agreeableness | 0.04 | 0.04 |
Conscientiousness | 0.04 | 0.04 |
Intervention | 0.03 | 0.05 |
Habits previously formed | 0.02 | 0.02 |
Employment | 0.02 | 0 |
Support network | 0.01 | 0 |
Free time | -0.01 | 0 |
Motivation to start a new habit | 0 | 0.2 |
Energy level | 0 | 0.19 |
Emotional stability | 0 | 0.02 |
We have collected additional information about the intervention group (specifically variables measuring their attitude toward the planned habit formation) and run a regression on them but the only factors that showed predictive were a self-predicted success (0.47 for success rate and 0.75 for the number of days habit was practiced) and expected habit enjoyment (0.04 for success rate and 0.22 for the number of days a habit was practiced). R² = 0.19 for the success rate and R² = 0.15 for the number of days a habit was practiced.
DISCUSSION
This randomized controlled trial of Clearer Thinking's Daily Ritual habit formation tool has shown that people who were assigned to use the tool performed their chosen habit more often than the control group. It is not certain, though, if the obtained result would persist long-term. The results were statistically significantly different from the control up until 6 weeks, but declined and no longer statistically significant after that point. This suggests that initial improvement tends to attenuate over time, and it may be advisable to recommend that people should repeat the program after six weeks or receive some other type of "booster" intervention to help the effects extend further. Interestingly, despite the effect of the program fading away after six weeks, people in the intervention group consistently felt that they were more successful in habit formation than participants from the control group, and the effect remained until the 8th week (the end) of the study. The duration effect needs further investigation.
It is also worth noting that compared to what people do normally with regard to habit formation this study may actually underestimate the effects of the Daily Ritual intervention. In particular, on any given day a person is very unlikely to even think to start a new habit, let alone to set an intention to start right away, to take the time to choose what precisely that habit is (e.g., by writing down their precise habit), and reporting every week on their progress. So, being in a control group and being asked to do all of these things, may have caused users to succeed at habits far above the rates that they normally would have, making this a potentially quite stringent test of Daily Ritual.
Regarding factors that could potentially affect habit formation, despite our efforts to test as many as possible, only a few showed some impact on habit formation, and the impact was rather small. The entire prediction model achieved an R² of only about 0.1. This means that all factors in the model can predict an increase in the number of days a habit was practiced weekly by 0.1 days. Most of the factors relevant to habit formation were unsurprising, like motivation, energy level, support network, employment, personality traits, emotional regulation, and the history of forming other habits. There were a few puzzling factors like fixed mindset and relationship status, but we cannot conclude much from this regression considering the low R² value.
It would be very interesting to see factors shown to be predictive in our model investigated by future studies.
References
Appendix 1
Appendix 2
Study programs
Part 1 (The screening)
Pre-screener code:
The main screener code:
Part 2 (The main study)
Part 3 (Feedback surveys)
Appendix 3
Difference between incomplete days (T-test statistic=1.05, p-value=0.295)
Week number | Mean incomplete days per week (intervention) | Mean incomplete days per week (control) |
1 | 1.21 | 1.26 |
2 | 1.03 | 1.11 |
3 | 0.99 | 1.52 |
4 | 0.99 | 1.05 |
5 | 1 | 1.1 |
6 | 0.92 | 1.18 |
7 | 0.76 | 1.08 |
8 | 0.77 | 1.14 |