James Hodgson and Dr. Daniel Nielson, Political Science Department

Rigorous evaluation has become the gold standard in international development. However, evaluations are limited to NGO’s that are willing to potentially be proven wrong and advised to adjust their strategy. Do characteristics of a proposed evaluation make it more or less appealing to an organization? This research focuses on the willingness of NGO’s to participate in different types of evaluations and specifically isolates the effect of proposing participatory evaluations in comparison to more traditional approaches to evaluation proposals.

We utilize experimental methods to create a strong causal argument by randomizing emailed evaluation proposals between a participatory treatment group and a placebo control group. We randomized a list of NGO’s into treatment and control groups. We sent emails that open up a dialogue about potential partnerships with the NGO in a participatory evaluation or in the control a more standard bid. The treatment email includes a paragraph about the participatory approach and mentions training members of their team to help with the evaluation. The control group of NGO’s receive a generic email that offers an evaluation relationship and briefly mentions that collecting data is important. Within the treatment and control emails are links to a survey to be filled out by the interested organizations. We also sent out reminder emails.

The primary outcome of interest in this experiment is the difference in response rate between the participatory treatment and the control group. We compare the number of responses that we received to the actual number of emails that were sent out to each group. The following table reports our primary results and shows the difference in response rate.

Table 1 indicates that we have received 120 responses from the treatment group and 205 from the control group, accounting for the total emails sent to each group this gives us response rates of approximately 1.61% and 1.22% respectively. Given the large number of organizations that contact was attempted with, the results still show a large difference in the framing effects of the language. The response rate difference is .003864 or about .4%; given the already low response rate, the treatment is roughly 30% better than the control. This is a substantively large treatment effect and indicates that researchers and intergovernmental organizations would do well to increase the participatory nature of their evaluation proposals.

Our findings suggest that researchers could increase the number and quality of evaluations as they frame their bids to NGO’s in ways that are most appealing to the organization. We suspect that many evaluators already advocate a participatory approach to evaluation, but these results demonstrate that explicitly outlining this approach in proposals with organizations could increase the number of actual accepted evaluations. These results also imply that evaluation practitioners will be more successful in establishing evidence based development as they utilize a participatory approach and explicitly involve organizations in the design and implementation of evaluations.