Impact Evaluation - Mixed Methods

Although there is an extensive literature on quantitative versus qualitative methods in impact evaluation, there is also a growing acceptance of the need for integrating the two approaches. Impact evaluations using quantitative data from statistically representative samples are better suited to assessing causality by using econometric methods or reaching generalizable conclusions. However, qualitative methods allow the indepth study of selected issues, cases, or events and can provide critical insights into beneficiaries’ perspectives, the dynamics of a particular reform, or the reasons behind certain results observed in a quantitative analysis. There are significant tradeoffs in selecting one technique over another (Baker).

Triangulation is a key concept that embodies much of the rationale behind doing mixed method research  and represents a set of principles to fortify the design, analysis and interpretation of findings in Impact Evaluation. Triangulation is about looking at things from multiple points of view, a method “to overcome the problems that stem from studies relying upon a single theory, a single method, a single set of data […] and from a single investigator” (Mikkelsen). 

There are different types of triangulation:

Data triangulation—To study a problem using different types of data, different points in time, or different units of analysis

• Investigator triangulation—Multiple researchers looking at the same problem
• Discipline triangulation—Researchers trained in different disciplines looking at the same problem
• Theory triangulation—Using multiple competing theories to explain and analyze a problem
• Methodological triangulation—Using different methods, or the same method over time, to study a problem.

Advantages of mixed-methods approaches to impact evaluation are the following:

• A mix of methods can be used to assess important outcomes or impacts of the intervention being studied. If the results from different methods converge, then inferences about the nature and magnitude of these impacts will be stronger.
• A mix of methods can be used to assess different facets of complex outcomes or impacts,yielding a broader, richer portrait than one method alone can. Quantitative impact evaluation techniques work well for a limited set of pre-established variables (preferably determined and measured ex ante) but less well for capturing unintended, less expected (indirect) effects of interventions. Qualitative methods or descriptive (secondary) data analysis can be helpful in better understanding the latter.
• One set of methods could be used to assess outcomes or impacts and another set to assessthe quality and character of program implementation, including program integrity andthe experiences during the implementation phase.
• Multiple methods can help ensure that the sampling frame and the sample selectionstrategies cover the whole of the target intervention and comparison populations.
  

Sources: 

Baker, J. L. (2000): Evaluating the Impact of Development Projects on Poverty. A Handbook for Practioners. The World Bank, Washington, D.C.

Leeuw, Frans & Vaessen, Jos (2009): Impact Evaluations and Development. Nonie Guidance on Impact Evaluation. Draft Version for Discussion at the Cairo conference March-April, 2009. Nonie – Network on Impact Evaluation, p.48- 50. 

Impact Evaluations and Development: NONIE Guidance on Impact Evaluation 2009: URL: http://www.worldbank.org/ieg/nonie/guidance.html 02/11/2009.

Mikkelsen, B. (2005) Methods for development work and research, Sage Publications, Thousand Oaks, p. 96.