Nick Moffit and Darren Hawkins, Political Science
Many academics and government officials promote the use of scientifically rigorous policy evaluations, yet studies show that most practitioners seldom employ such evaluations, despite a growing body of policy-relevant academic literature. This represents a fundamental flaw of delegation, as citizens prefer the most efficient provision of public goods, but government officials fail to use available information to improve policy outcomes. This study examines how screening and monitoring of government officials are associated with practitioner engagement in academic research.
To evaluate what institutional characteristics are associated with greater use of academic research by policy practitioners, U.S. and local university students administered a survey to around 300 government officials in Peru and India in summer 2017. The survey collected information on officials’ experience with and interest in academic research as well as workplace factors and personal characteristics that might be associated with those outcomes. The students administered the survey in the context of appointments with hundreds of government officials to train them on how to use a website the team developed that contains accessible information on more than 400 academic studies based on randomized controlled trials, experiments in which different groups of subjects are randomly assigned to receive different policy interventions. The website uses easy-to-read graphs and accessible language to report the results of rigorous impact evaluations of government programs in developing countries. Survey sampling included those who had either responded positively to an invitation for training on the use of our website or joined a training group at the invitation of a supervisor or co-worker.
An initial summary of the survey data revealed that 17% of officials had never used academic research in their policy making decisions, yet 92% of them said they believed that using academic research would improve the quality of their work. We used reported interest in and use of academic research as our dependent variables, and to more fully measure that, the survey asked several questions about officials’ engagement with academic research; we later combined the responses to these questions into three indices. The master index consists of nine questions asking about policymakers’ attitudes toward academic research which can be characterized as engagement with that research. The interest index is made up of a subset of three questions from the master index about the respondents’ interest in and perceived helpfulness of using academic research. The use index comprises another set of three questions from the master index about when and how often the official uses academic research.
Results and Discussion
In the master index, individuals who took a civil service exam prior to hiring score about 6% higher than other government officials (figure 1). These results suggest the importance of implementing screening practices in hiring. Governments should use screening processes to more effectively avoid hiring unknowledgeable job candidates and promote more effective policy making by hiring more people who will use evidence-based research more often.
The importance that supervisors place on using academic research also makes a significant difference in their employees’ engagement in academic research, this time only regarding interest, not use. If supervisors place more emphasis on such reports, then their practitioners report greater interest in them, think they are useful, and will help them work better. In fact, employees who say their boss rates studies as the first of seven important factors in decisionmaking rank about 10% higher on the interest index than employees who say their boss rates academic research as last in importance (figure 2). However, a supervisor’s emphasis on academic research does not increase reported use of evidence-based research, either in the use index or in the individual questions that comprise that index. These results suggest two important considerations for implementing future monitoring policy. First, supervisors exert considerable power over what methods their employees identify as most effective in helping them to do their jobs. However, these findings also highlight the limitations of monitoring.
Survey evidence in this article suggests that a significant number of government officials do not use academic research in their policy implementations. However, the survey also finds that screening and monitoring are indeed associated with engagement in academic studies and can therefore be used as tools for fixing the delegation problem of bureaucrats that are not effectively using academic policy research. High-level government managers can act as gatekeepers to better screen applicants through civil service exams for analytical skills important to using rigorous studies. Managers can also monitor the use of rigorous research by requiring subordinates to regularly report on their use of academic studies. .5