Difference between revisions of "Impact Evaluation - Mixed Methods"

From energypedia
***** (***** | *****)
m
***** (***** | *****)
m
Line 1: Line 1:
 +
see"[[Methoden|Methods]]"
 +
 
===== Methodological Triangulation  =====
 
===== Methodological Triangulation  =====
  
Line 5: Line 7:
 
There are different types of triangulation:  
 
There are different types of triangulation:  
  
*<span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial">Data triangulation—To study a problem using different types of data, different points in time, or different units of analysis</font></span>  
+
*<span style="font-size: 10pt;"><font face="Arial">Data triangulation—To study a problem using different types of data, different points in time, or different units of analysis</font></span>  
*<span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial">Investigator triangulation—Multiple researchers looking at the same problem</font></span>  
+
*<span style="font-size: 10pt;"><font face="Arial">Investigator triangulation—Multiple researchers looking at the same problem</font></span>  
*<span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial">Discipline triangulation—Researchers trained in different disciplines looking at the same problem</font></span>  
+
*<span style="font-size: 10pt;"><font face="Arial">Discipline triangulation—Researchers trained in different disciplines looking at the same problem</font></span>  
*<span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial">Theory triangulation—Using multiple competing theories to explain and analyze a problem</font></span>  
+
*<span style="font-size: 10pt;"><font face="Arial">Theory triangulation—Using multiple competing theories to explain and analyze a problem</font></span>  
*'''<span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial">Methodological triangulation</font></span>'''<span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial">—Using different methods, or the same method over time, to</font></span> <span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial">study a problem.</font></span>
+
*'''<span style="font-size: 10pt;"><font face="Arial">Methodological triangulation</font></span>'''<span style="font-size: 10pt;"><font face="Arial">—Using different methods, or the same method over time, to</font></span> <span style="font-size: 10pt;"><font face="Arial">study a problem.</font></span>
  
 
===== Mixed Methods  =====
 
===== Mixed Methods  =====
Line 18: Line 20:
 
*A mix of methods can be used to assess different facets of complex outcomes or impacts, yielding a broader, richer portrait than one method alone can. Quantitative impact evaluation techniques work well for a limited set of pre-established variables (preferably determined and measured ex ante) but less well for capturing unintended, less expected (indirect) effects of interventions. Qualitative methods or descriptive (secondary) data analysis can be helpful in better understanding the latter.  
 
*A mix of methods can be used to assess different facets of complex outcomes or impacts, yielding a broader, richer portrait than one method alone can. Quantitative impact evaluation techniques work well for a limited set of pre-established variables (preferably determined and measured ex ante) but less well for capturing unintended, less expected (indirect) effects of interventions. Qualitative methods or descriptive (secondary) data analysis can be helpful in better understanding the latter.  
 
*One set of methods could be used to assess outcomes or impacts and another set to assess the quality and character of program implementation, including program integrity and the experiences during the implementation phase.  
 
*One set of methods could be used to assess outcomes or impacts and another set to assess the quality and character of program implementation, including program integrity and the experiences during the implementation phase.  
*Multiple methods can help ensure that the sampling frame and the sample selectionstrategies cover the whole of the target intervention and comparison populations.<span style="font-size: 10pt; mso-bidi-font-family: arial"><font face="Arial"></font></span>
+
*Multiple methods can help ensure that the sampling frame and the sample selectionstrategies cover the whole of the target intervention and comparison populations.<span style="font-size: 10pt;"><font face="Arial"></font></span>
  
 
===== '''Quantitative impact evaluation'''  =====
 
===== '''Quantitative impact evaluation'''  =====
  
Quantitative impact evaluation have a comparative advantage in addressing the issue of attribution: <span style="font-size: 10pt"><font face="Arial">Evaluations can either be <font color="#002bb8">[[Monitoring_Glossary|experimental]]</font>&nbsp;(or randomized control designed)&nbsp;as when the evaluator purposely collects data and designs evaluations in advance or <font color="#002bb8">[[Monitoring_Glossary|quasi-experimental]]</font> as when data are collected to mimic an experimental situation. <font color="#002bb8">[[Monitoring_Glossary|Multiple regression analysis]]</font> is the all-purpose technique that can be used in virtually all settings; when the experiment is organized in such a way that no controls are needed, a simple comparison of means can be used instead of a regression since it will give the same answer.</font></span><span style="font-size: 10pt"><font face="Arial"><span style="font-size: 10pt; font-family: 'arial','sans-serif'">Three related problems that quantitative impact evaluation techniques attempt to address are the following:&nbsp;</span></font></span>  
+
Quantitative impact evaluation have a comparative advantage in addressing the issue of attribution: <span style="font-size: 10pt;"><font face="Arial">Evaluations can either be <font color="#002bb8">[[Monitoring Glossary|experimental]]</font>&nbsp;(or randomized control designed)&nbsp;as when the evaluator purposely collects data and designs evaluations in advance or <font color="#002bb8">[[Monitoring Glossary|quasi-experimental]]</font> as when data are collected to mimic an experimental situation. <font color="#002bb8">[[Monitoring Glossary|Multiple regression analysis]]</font> is the all-purpose technique that can be used in virtually all settings; when the experiment is organized in such a way that no controls are needed, a simple comparison of means can be used instead of a regression since it will give the same answer.</font></span><span style="font-size: 10pt;"><font face="Arial"><span style="font-size: 10pt; font-family: 'arial','sans-serif';">Three related problems that quantitative impact evaluation techniques attempt to address are the following:&nbsp;</span></font></span>  
  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">the establishment of a ''counterfactual'': What would have happened in the absence of the intervention(s);&nbsp;</span>  
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">the establishment of a ''counterfactual'': What would have happened in the absence of the intervention(s);&nbsp;</span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">the elimination of <font color="#002bb8">''[[Monitoring_Glossary|selection bias/effects]]'',</font> leading to differences between intervention group (or treatment group) and control group;&nbsp;</span>  
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">the elimination of <font color="#002bb8">''[[Monitoring Glossary|selection bias/effects]]'',</font> leading to differences between intervention group (or treatment group) and control group;&nbsp;</span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">a solution for the problem of ''unobservables'': the omission of one or more unobserved variables, leading to biased estimates.&nbsp; </span>
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">a solution for the problem of ''unobservables'': the omission of one or more unobserved variables, leading to biased estimates.&nbsp; </span>
  
 
===== '''Qualitative&nbsp;impact evaluation'''&nbsp;  =====
 
===== '''Qualitative&nbsp;impact evaluation'''&nbsp;  =====
  
Survey data collection and [[Monitoring Glossary|semi-structured interviews]], and [[Monitoring Glossary|focus-group interviews are]] but a few of the specific methods that are found throughout the landscape of methodological approaches to impact evaluation.Qualitative techniques cannot quantify the changes attributable to interventions but should be used to evaluate important issues for which quantification is not feasible or practical, and to develop complementary and in-depth perspectives on processes of change induced by interventions.
+
Survey data collection and [[Monitoring Glossary|semi-structured interviews]], and [[Monitoring Glossary|focus-group interviews are]] but a few of the specific methods that are found throughout the landscape of methodological approaches to impact evaluation.Qualitative techniques cannot quantify the changes attributable to interventions but should be used to evaluate important issues for which quantification is not feasible or practical, and to develop complementary and in-depth perspectives on processes of change induced by interventions.  
  
 
===== '''Other approaches'''  =====
 
===== '''Other approaches'''  =====
  
<span style="font-size: 10pt; font-family: 'arial','sans-serif'">Nowadays, participatory methods have become ‘mainstream’ tools in development in almost every area of policy intervention. Participatory evaluation approaches are built on the principle that stakeholders should be involved in some or all stages of the evaluation. In the case of impact evaluation this includes aspects such as the determination of objectives, indicators to be taken into account, as well as stakeholder participation in data collection and analysis. </span>
+
<span style="font-size: 10pt; font-family: 'arial','sans-serif';">Nowadays, participatory methods have become ‘mainstream’ tools in development in almost every area of policy intervention. Participatory evaluation approaches are built on the principle that stakeholders should be involved in some or all stages of the evaluation. In the case of impact evaluation this includes aspects such as the determination of objectives, indicators to be taken into account, as well as stakeholder participation in data collection and analysis. </span>  
  
<span style="font-size: 10pt; font-family: 'arial','sans-serif'">Methodologies commonly included under this umbrella include: </span>
+
<span style="font-size: 10pt; font-family: 'arial','sans-serif';">Methodologies commonly included under this umbrella include: </span>  
  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">[http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20509352~menuPK:1278203~pagePK:148956~piPK:216618~theSitePK:410306,00.html Participatory Impact Monitoring] </span>
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">[http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20509352~menuPK:1278203~pagePK:148956~piPK:216618~theSitePK:410306,00.html Participatory Impact Monitoring] </span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">the Participatory Learning and Action (PLA) family</span>  
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">the Participatory Learning and Action (PLA) family</span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">including Rapid Rural Appraisal (RRA), </span>
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">including Rapid Rural Appraisal (RRA), </span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">[http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20507691~pagePK:148956~piPK:216618~theSitePK:410306,00.html Participatory Rural Appraisal (PRA), ]and </span>
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">[http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20507691~pagePK:148956~piPK:216618~theSitePK:410306,00.html Participatory Rural Appraisal (PRA), ]and </span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">Participatory Poverty Assessment (PPA), </span>
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">Participatory Poverty Assessment (PPA), </span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">Policy and Social Impact Analysis (PSIA), </span>
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">Policy and Social Impact Analysis (PSIA), </span>  
*<span style="font-size: 10pt; font-family: 'arial','sans-serif'">Social Assessment (SA).</span>&nbsp;
+
*<span style="font-size: 10pt; font-family: 'arial','sans-serif';">Social Assessment (SA).</span>&nbsp;
  
 
&nbsp;  
 
&nbsp;  
Line 50: Line 52:
 
''Sources:&nbsp;''  
 
''Sources:&nbsp;''  
  
'''''<span style="font-weight: normal; font-size: 10pt; font-family: 'arial','sans-serif'; mso-bidi-font-weight: bold">Impact Evaluations and Development: NONIE Guidance on Impact Evaluation 2009: URL: </span>'''<span style="font-weight: normal; font-size: 10pt; font-family: 'arial','sans-serif'; mso-bidi-font-weight: bold">[http://www.worldbank.org/ieg/nonie/guidance.html <font color="#002bb8">http://www.worldbank.org/ieg/nonie/guidance.html</font>] 02/11/2009.</span>''  
+
'''''<span style="font-weight: normal; font-size: 10pt; font-family: 'arial','sans-serif';">Impact Evaluations and Development: NONIE Guidance on Impact Evaluation 2009: URL: </span>'''<span style="font-weight: normal; font-size: 10pt; font-family: 'arial','sans-serif';">[http://www.worldbank.org/ieg/nonie/guidance.html <font color="#002bb8">http://www.worldbank.org/ieg/nonie/guidance.html</font>] 02/11/2009.</span>''  
  
 
''Leeuw, Frans &amp; Vaessen, Jos (2009): Impact Evaluations and Development. Nonie Guidance on Impact Evaluation. Draft Version for Discussion at the Cairo conference March-April, 2009. Nonie – Network on Impact Evaluation, p.48- 50.&nbsp;''  
 
''Leeuw, Frans &amp; Vaessen, Jos (2009): Impact Evaluations and Development. Nonie Guidance on Impact Evaluation. Draft Version for Discussion at the Cairo conference March-April, 2009. Nonie – Network on Impact Evaluation, p.48- 50.&nbsp;''  

Revision as of 11:07, 8 June 2010

see"Methods"

Methodological Triangulation

Triangulation is a key concept that embodies much of the rationale behind doing mixed method research  and represents a set of principles to fortify the design, analysis and interpretation of findings in Impact Evaluation. Triangulation is about looking at things from multiple points of view, a method “to overcome the problems that stem from studies relying upon a single theory, a single method, a single set of data […] and from a single investigator” (Mikkelsen). 

There are different types of triangulation:

  • Data triangulation—To study a problem using different types of data, different points in time, or different units of analysis
  • Investigator triangulation—Multiple researchers looking at the same problem
  • Discipline triangulation—Researchers trained in different disciplines looking at the same problem
  • Theory triangulation—Using multiple competing theories to explain and analyze a problem
  • Methodological triangulation—Using different methods, or the same method over time, to study a problem.
Mixed Methods

Advantages of mixed-methods approaches to impact evaluation are the following:

  • A mix of methods can be used to assess important outcomes or impacts of the intervention being studied. If the results from different methods converge, then inferences about the nature and magnitude of these impacts will be stronger.
  • A mix of methods can be used to assess different facets of complex outcomes or impacts, yielding a broader, richer portrait than one method alone can. Quantitative impact evaluation techniques work well for a limited set of pre-established variables (preferably determined and measured ex ante) but less well for capturing unintended, less expected (indirect) effects of interventions. Qualitative methods or descriptive (secondary) data analysis can be helpful in better understanding the latter.
  • One set of methods could be used to assess outcomes or impacts and another set to assess the quality and character of program implementation, including program integrity and the experiences during the implementation phase.
  • Multiple methods can help ensure that the sampling frame and the sample selectionstrategies cover the whole of the target intervention and comparison populations.
Quantitative impact evaluation

Quantitative impact evaluation have a comparative advantage in addressing the issue of attribution: Evaluations can either be experimental (or randomized control designed) as when the evaluator purposely collects data and designs evaluations in advance or quasi-experimental as when data are collected to mimic an experimental situation. Multiple regression analysis is the all-purpose technique that can be used in virtually all settings; when the experiment is organized in such a way that no controls are needed, a simple comparison of means can be used instead of a regression since it will give the same answer.Three related problems that quantitative impact evaluation techniques attempt to address are the following: 

  • the establishment of a counterfactual: What would have happened in the absence of the intervention(s); 
  • the elimination of selection bias/effects, leading to differences between intervention group (or treatment group) and control group; 
  • a solution for the problem of unobservables: the omission of one or more unobserved variables, leading to biased estimates. 
Qualitative impact evaluation 

Survey data collection and semi-structured interviews, and focus-group interviews are but a few of the specific methods that are found throughout the landscape of methodological approaches to impact evaluation.Qualitative techniques cannot quantify the changes attributable to interventions but should be used to evaluate important issues for which quantification is not feasible or practical, and to develop complementary and in-depth perspectives on processes of change induced by interventions.

Other approaches

Nowadays, participatory methods have become ‘mainstream’ tools in development in almost every area of policy intervention. Participatory evaluation approaches are built on the principle that stakeholders should be involved in some or all stages of the evaluation. In the case of impact evaluation this includes aspects such as the determination of objectives, indicators to be taken into account, as well as stakeholder participation in data collection and analysis.

Methodologies commonly included under this umbrella include:

 

Sources: 

Impact Evaluations and Development: NONIE Guidance on Impact Evaluation 2009: URL: http://www.worldbank.org/ieg/nonie/guidance.html 02/11/2009.

Leeuw, Frans & Vaessen, Jos (2009): Impact Evaluations and Development. Nonie Guidance on Impact Evaluation. Draft Version for Discussion at the Cairo conference March-April, 2009. Nonie – Network on Impact Evaluation, p.48- 50. 

Mikkelsen, B. (2005) Methods for development work and research, Sage Publications, Thousand Oaks, p. 96.