James Hodgson and Daniel Nielson, Political Science

Important policy decisions, that affect lives all around the world, are often made without consulting the best available evidence. Many governments have called for an increase in evidence-based policy making in an effort to make policies more effective and based on research. From the baseline survey we collected we found that while 92% of government officials said they thought using evidenced based reports would improve the quality of their work, only 8% reported that they use evidence based reports in all of their policy decisions. Academics and program evaluators have amassed a large knowledge base on the effectiveness of public programs but government officials rarely consult this evidence. This research looks at how the delivery of information can affect policymakers’ use of academic findings. Specifically, we look at the behavior of 368 government officials from Peru, Tanzania, and India as they interact with different versions of a website database containing summaries of impact evidence of various development related policies.This project looks at how adding game elements such as a quiz mode, user points, and scoreboards, to the website can motivate policy makers in developing countries to engage more deeply with this information.

We utilize a randomized control trial that occurred in India, Tanzania, and Peru to make a causal argument about officials’ engagement with evidence-based reports. Officials were given unique login identification allowing us to anonymously track their website usage over time; this usage serves as our behavioral measure of their engagement with the database. Our primary outcome is an index that measures a user’s engagement with the database by summing various actions they took such as how many studies they viewed, the time they spent on the website, the sectors they browsed etc. Understanding the impact of information delivery on this population is a needed step in bridging the evidence-based policy making gap.

We find that adding a scoreboard in which government officials compare points they earn to others significantly increased a user’s engagement with the database of evidence-based reports. The scoreboard treatment increases an official’s engagement compared to the group that did not have game elements. This finding is supported by various analysis methods including linear models, randomization inference, and quantile regression, under a variety of methods this main treatment effect holds. We do not find differences between the various countries in overall usage, nor significant usage over time as of six months from time of treatment.

This finding indicates that government officials are interested in evidence-based reports and reducing the barriers for them to use these reports is a welcome measure, as in our survey government officials reported that they do think using these resources are a good idea. Additionally, the framing of evidence influences how officials interact and engage with that evidence. Incremental gamification elements do not seem to have strong implications for policy maker engagement with evidence based reports, however larger changes like implementing a scoreboard do significantly change user engagement.

Figure 1 – Shows the results of the regression of the engagement index that measures a user’s behavior on the website on each website version of the website. Each coefficient is in comparison to the control version of the website. The outcome measure is standard deviations of the website engagement index. The diamonds show the average treatment effect for each treatment group, while the bars show the confidence intervals for each estimation at 90% and 95%, sample size of 368 government officials.

Figure 2 – Shows the mean difference in the usage of the website database between those officials that were randomly given the control version and the officials that were randomly assigned a version of the website that included scoreboards based on the number of questions they answered correctly. It also breaks down treatment effects by country.

Regardless of where one lives in the world, we all rely on public policy makers to help improve our lives. Yet these important policy decision, that affect lives all around the world, are often made without consulting the best available evidence. This research offers evidence of potential solutions to bridging the information gap between policy makers and the best available data.