Difference between revisions of "Result Based Monitoring of Cookstove Projects"

From energypedia
***** (***** | *****)
***** (***** | *****)
m
 
(109 intermediate revisions by 15 users not shown)
Line 1: Line 1:
[[Cooking Energy Compendium|--> Back to Overview Compendium]]
 
  
&nbsp;<br>
+
[[File:GIZ HERA Cooking Energy Compendium small.png|left|831px|GIZ HERA Cooking Energy Compendium|alt=GIZ HERA Cooking Energy Compendium small.png|link=GIZ HERA Cooking Energy Compendium]]<br/><br/><!--
  
<span style="color: #ff0000">NEU</span>
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | Cooking Energy System |'''[[GIZ HERA Cooking Energy Compendium#Cooking Energy Technologies and Practices|Cooking Energy System]]''' {{!}} | [[GIZ HERA Cooking Energy Compendium#Cooking Energy Technologies and Practices|Cooking Energy System]] {{!}} | }} <!--
  
Back to overview GIZ&nbsp;HERA&nbsp;Compendium
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | Basics |'''[[GIZ HERA Cooking Energy Compendium#Basics about Cooking Energy|Basics]]''' {{!}} | [[GIZ HERA Cooking Energy Compendium#Basics about Cooking Energy|Basics]] {{!}} | }} <!--
  
[http://energypedia.info/index.php/GIZ_HERA_Cooking_Energy_Compendium http://energypedia.info/index.php/GIZ_HERA_Cooking_Energy_Compendium]  
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | Policy Advice |'''[[GIZ HERA Cooking Energy Compendium#Policy Advice on Cooking Energy|Policy Advice]]''' {{!}} | [[GIZ HERA Cooking Energy Compendium#Policy Advice on Cooking Energy|Policy Advice]] {{!}} | }} <!--
  
= Basics about Results Based Monitoring&nbsp;  =
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | Planning |'''[[GIZ HERA Cooking Energy Compendium#Planning Cooking Energy Interventions|Planning]]''' {{!}} | [[GIZ HERA Cooking Energy Compendium#Planning Cooking Energy Interventions|Planning]] {{!}} | }} <!--
  
== Why monitoring? ==
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | ICS Supply |'''[[GIZ HERA Cooking Energy Compendium#Designing and Implementing Improved Cookstoves .28ICS.29 Supply Interventions|Designing and Implementing ICS Supply]]''' {{!}} | [[GIZ HERA Cooking Energy Compendium#Designing and Implementing Improved Cookstoves .28ICS.29 Supply Interventions|Designing and Implementing ICS Supply]] {{!}} | }} <!--
  
Projects introduce development changes to make a difference to the lives of target groups. They strive to achieve positive change, and it is essential for them to monitor and evaluate the results&nbsp;of their intervention.
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | Woodfuel Supply |'''[[GIZ HERA Cooking Energy Compendium#Designing and Implementing Woodfuel Supply Interventions|Designing and Implementing Woodfuel Supply]]''' {{!}} | [[GIZ HERA Cooking Energy Compendium#Designing and Implementing Woodfuel Supply Interventions|Designing and Implementing Woodfuel Supply]] {{!}} | }} <!--
  
Monitoring&nbsp;serves different purposes:  
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | Climate Change |'''[[GIZ HERA Cooking Energy Compendium#Climate Change Related Issues|Climate Change]]''' | [[GIZ HERA Cooking Energy Compendium#Climate Change Related Issues|Climate Change]] {{!}} | }} <!--
  
*To check that set targets have been met  
+
-->{{#ifeq: {{#show: {{PAGENAME}} |?Hera category}} | Extra |'''[[GIZ HERA Cooking Energy Compendium#Climate Change Related Issues|Extra]]''' | [[GIZ HERA Cooking Energy Compendium#Climate Change Related Issues|Extra]] }}
*To provide data and information for reviewing the strategy  
+
 
*To steer and make changes, where necessary, to an intervention  
+
<br/>
*To create ownership among various project actors  
+
 
 +
= Basics about Results Based Monitoring<br/> =
 +
 
 +
== Why Monitoring<br/> ==
 +
 
 +
Projects introduce development changes to make a difference in the lives of target groups. They strive to achieve positive change, and it is essential for them to monitor and evaluate the results of their intervention.
 +
 
 +
<u>Monitoring serves different purposes:</u>
 +
 
 +
*To check that set targets have been met
 +
*To provide data and information for reviewing the strategy
 +
*To steer and make changes, where necessary, to an intervention
 +
*To create ownership among various project actors
 
*To provide evidence on progress/ changes/ achievements for national partners (e.g. ministries).
 
*To provide evidence on progress/ changes/ achievements for national partners (e.g. ministries).
  
Monitoring requires time, personnel and funds, and thus it needs to be included into the activities and budget planning&nbsp;from the very beginning of the intervention to ensure that it is an integral part of the approach. Often, however, it is only considered at a later stage, when resteering, reviewing the strategy and using it as management tool is not possible any more. <br>It is crucial for capacity development interventions to involve project partners and counterparts into all steps of&nbsp;monitoring (planning, implementation, analysis&nbsp;and interpretation). They themselves learn to use tools and instruments, review their own interventions and receive more ownership. Finally results and recommendations should be&nbsp;be discussed with&nbsp;stakeholders and counterparts to jointly draw conclusions.&nbsp;  
+
Monitoring requires time, personnel and funds, and thus it needs to be included into the activities and budget planning from the very beginning of the intervention to ensure that it is an integral part of the approach. Often, however, it is only considered at a later stage, when countersteering, reviewing the strategy and using it as a management tool is not possible any more.<br/>It is crucial for capacity development interventions to involve project partners and counterparts into all steps of monitoring (planning, implementation, analysis and interpretation). They themselves learn to use tools and instruments, review their own interventions and receive more ownership. Finally results and recommendations should be be discussed with stakeholders and counterparts to jointly draw conclusions.
 +
 
 +
<br/>
 +
 
 +
== Introduction to Results Based Monitoring (RBM)<br/> ==
 +
 
 +
'''[[:File:Gtz results-based monitoring.pdf|Results Based Monitoring (RBM)]]''' is an international monitoring standard developed and agreed by the '''OECD DAC (Development Assistance Committee <font size="2">of the Organisation for Economic Co-operation and Development)</font>''' to monitor development results. Results are defined as development changes that follow directly from an intervention; they can be outputs, outcomes or impacts (intended or unintended, positive and/or negative) resulting from a development intervention. According to the OECD-DAC definition, results occur as causal sequences from a development intervention towards the desired objectives. RBM is a method to examine the result hypotheses in a empirical and systematic way, which should be regularly applied by GIZ programmes.
 +
 
 +
<br/>
  
<br>
+
<br/>
  
== Introduction to Results Based Monitoring  ==
+
== Results Chains<br/> ==
  
Results Based Monitoring (RBM) is an international monitoring standard developed and agreed by the OECD DAC (Development Assistance Committee&nbsp;<font size="2">of the Organisation for Economic Co-operation and Development)</font> to monitor development results. Results are defined as development changes that follow directly from an intervention; they can be outputs, outcomes or impacts (intended or unintended, positive and/or negative) resulting from a development intervention. According to the OECD-DAC definition, results occur as causal sequences from a development intervention towards the desired objectives. RBM is a method to examine the result hypotheses in a empirical and systematic way, which should be&nbsp;regularly applied by GIZ programmes.<br>
+
The basis of RBM are the results chains, which describe how a development interventions, through a step-by-step process, contributes to development results. The intervention starts with the inputs used to perform activities, which lead to the outputs of the project. These outputs, which are used by target groups or intermediaries, lead to outcomes and impacts.
  
== Results chains  ==
+
In most cases, it is relatively easy to attribute changes, up to the level that identifies the uses of the output. Beyond this, climbing up to the levels of ‘outcome’ and ‘impact’, external factors influence whether the intended results can be achieved. These external factors can only be controlled or influenced to a certain extent (if at all) by the project or programme. Whether the objectives are met no longer depends solely on the performance of the project, but depends on all the actors and external factors involved. Therefore, achievements are only attributable to the intervention up to a certain level – called the ‘outcome’. Beyond the outcome, effects can no longer be directly linked to this one development intervention. The attribution gap widens at the stage where changes, although observed in the target area, cannot be solely related to project outputs - as the following GIZ figure describes:
  
The basis of RBM are&nbsp;the results chains, which describe how a development interventions, through a step-by-step process, contributes to development results. The intervention starts with the inputs used to perform activities, and these lead to the outputs of the project. These outputs, which are used by target groups or intermediaries, lead to outcomes and impacts.
+
<br/>
  
In most cases, it is relatively easy to attribute changes, up to the level that identifies the uses of the output. Beyond this, climbing up to the levels of ‘outcome’ and ‘impact’, external factors influence whether the intended results can be achieved. These external factors can only be controlled or influenced to a certain extent (if at all) by the project or programme.&nbsp;Whether the objectives are met no longer depends solely on the performance of the project, but depends on all the actors and external factors involved. Therefore, achievements are only attributable to the intervention up to a certain level – called the ‘outcome’.&nbsp;Beyond the outcome, effects can no longer be directly linked to this one development intervention. The attribution gap widens at the stage where changes, although observed in the target area, cannot be solely related to project outputs - as the following GIZ figure describes:
+
[[File:GTZ Results Model.jpg|center|601px|GTZ Results Model.jpg|alt=GTZ Results Model.jpg]]
  
<br>
+
For interpretation and analysis of observed results it is important to have a baseline. Baseline data generated at the beginning of the intervention, providing information about the status quo (people with access to improved cookstoves, woodfuel demand, ...), are still neglected or forgotten by programmes. But the advatage is obvious: the programme is able to relate the observed results with the baseline data and draw respective conclusions about implemented interventions.
  
&nbsp; [[Image:Schaubild.jpg]]
+
<br/>
  
For interpretation and analysis of observed results it is important to have a baseline. Baseline data generated at the beginning of the intervention, providing information about the status quo (people with access to improved cookstoves,&nbsp;woodfuel demand, ...), are still neglected or forgotten by programmes. But the advatage is obvious:&nbsp;the programme is able to relate the observed results with the baseline data and draw respective conclusions about&nbsp;implemented interventions.&nbsp;&nbsp;
+
<br/>
  
<br>
+
<br/>
  
'''<font size="2">Additional Information&nbsp;</font>'''
+
<br/>
  
{| style="width: 758px; height: 597px" cellspacing="1" cellpadding="1" width="758" border="1"
+
== Further Information<br/> ==
 +
 
 +
{| style="width: 758px; height: 597px" cellspacing="1" cellpadding="1" border="1"
 
|-
 
|-
 
|  
 
|  
 
GTZ (2008): Wirkungsorientiertes Monitoring Leitfaden für die Technische Zusammenarbeit (deutsch)
 
GTZ (2008): Wirkungsorientiertes Monitoring Leitfaden für die Technische Zusammenarbeit (deutsch)
  
| [[Image:Wirkungsorientiertes-monitoring-leitfaden-de.pdf]]
+
Englische Version:Results-based Monitoring Guidelines for Technical Cooperation
 +
 
 +
| [[File:Wirkungsorientiertes-monitoring-leitfaden-de.pdf|180px|Wirkungsorientiertes-monitoring-leitfaden-de.pdf|alt=Wirkungsorientiertes-monitoring-leitfaden-de.pdf]]
 +
|-
 +
| [http://www.oecd.org/development/peer-reviews/2754804.pdf RBM terminology 30 May.doc OECD/DAC (2002) – Glossary of Key Terms in Evaluation and Results Based Management]
 +
| <br/>
 
|-
 
|-
| [http://www.undp.org/execbrd/word/Final%20RBM%20terminology%2030%20May.doc OECD/DAC (2002)&nbsp;– <span>Glossary of Key Terms in Evaluation and Results Based Management</span>]
 
 
|  
 
|  
 +
GVEP M&E Group (2006) - [http://www.gvepinternational.org/sites/default/files/monitoring_and_evaluation_guide.pdf A Guide to Monitoring and Evaluation for Energy Projects]<br/>For setting up Results Based Monitoring for Energy projects compare
 +
 +
This Guide proposes a step by step approach to building project-specific M&E procedures. The guide is intended for energy access projects, which don’t have already donor or stakeholder determined M&E methods. The guide was developed by the International Working Group on Monitoring and Evaluation in Energy for Development (M&EED)
 +
 +
| [[File:Meed guide final version english-1-.pdf|180px|Meed guide final version english-1-.pdf|alt=Meed guide final version english-1-.pdf]]
 
|-
 
|-
 
|  
 
|  
GVEP M&amp;EED Group (2006) - [http://www.gvepinternational.org/en/community/resources/meed-guide-final-version-english A Guide&nbsp;to Monitoring and Evaluation for Energy Projects]<br>For setting up Results Based Monitoring for Energy projects compare&nbsp;
+
Boiling Point No 55 "Monitoring and&nbsp;Evaluation"&nbsp;(June 2008) on the web: [http://www.hedon.info/View+issue&itemId=8824 http://www.hedon.info/]
  
This Guide proposes a step by step approach to building project-specific M&amp;E procedures. The guide is intended for energy access projects, which don’t have already donor or stakeholder determined M&amp;E methods. The guide was developed by the International Working Group on Monitoring and Evaluation in Energy for Development (M&amp;EED)
+
| <br/>
 +
|-
 +
|
 +
[http://www.hedon.info/tiki-download_item_attachment.php?attId=65 Toolkit - Six steps to Results Based Monitoring]<br/>The following six steps should help you to set up your Results Based Monitoring (RBM)<br/>system. From identifying all stakeholders in the system, formulating the assumptions<br/>on which your strategy is based, analysing risks and side effects, choosing observation<br/>fields, specifying indicators for measurement up to their operationalisation to data<br/>collection – and there you go!<br/>Author: Melanie Djedje<br/>In: Boiling Point 55 (2008)
 +
 
 +
|
 +
[http://www.hedon.info/tiki-download_item_attachment.php?attId=65 www.hedon.info/tiki-download_item_attachment.php?attId=65]
 +
 
 +
(paste web address in your browser to open)
  
| [[Image:Meed guide final version english-1-.pdf]]
 
 
|-
 
|-
 +
| [http://www2.gtz.de/monitoring/ GIZ-Community on Results-Based Monitoring and Evaluation]
 +
| <br/>
 +
|-
 +
| More information about Impact Monitoring on [[Portal:Impacts|Energypedia - Impacts]]
 +
|-
 +
| [[:File:How to Monitor and Evaluate Household Energy Projects?.pdf|GTZ (1996) – Measuring Successes and Setbacks. How to Monitor and Evaluate Household Energy Projects]]
 
|  
 
|  
Boiling Point No 55 "Monitoring and&nbsp;Evaluation"&nbsp;(June 2008) on the web: [http://www.hedon.info/BoilingPoint55-June2008 http://www.hedon.info/]
+
|-
 +
| [[:file:Suivi des impacts Djedje -Kompatibilitätsmodus-.pdf|Suivi des impacts]]
 +
| <br/>
 +
|}
  
| <span style="color: #ff0000">Boiling Point No 55 (June 2008) as </span>[http://www.hedon.info/BoilingPoint55-June2008 <span style="color: #ff0000">pdf&nbsp;&nbsp;</span>]
+
<br/>
|}
 
  
<br>
+
<br/>
  
<br>
+
<br/>
  
= <span lang="EN-GB"><font face="Arial">Results chains for cooking energy interventions</font></span> =
+
= <span lang="EN-GB"><font face="Arial">Results Chains for Cooking Energy Interventions</font></span><br/> =
  
The following example of the Energy Advisory Project&nbsp;(EAP) in Uganda&nbsp;shows two main chains for ‘Scaling up of improved biomass stoves’:  
+
<u>The following example of the '''Energy Advisory Project (EAP)''' in Uganda shows two main chains for ‘Scaling up of improved biomass stoves’:</u>
  
*for stove supply; targeted at producers and traders&nbsp;
+
*for stove supply; targeted at producers and traders
*for stove demand; targeting users and the public sector.
+
*for stove demand; targeting users and the public sector
<div><div id="ftn1">
+
<div><div id="ftn1"></div></div>
&nbsp;
+
[[File:Results chain.JPG|center|603px|Results chain.JPG|alt=Results chain.JPG]]
</div></div>
 
[[Image:Results chain.JPG]]  
 
  
<br>
+
<br/>
  
Description of figure:  
+
<u>'''Description of Figure:'''</u>
  
'''Input '''is not mentioned here, but comprises all the resources provided by all partners: donor organisations, implementers, government partners (money, personnel and material).  
+
'''Input '''is not mentioned here, but comprises all the resources provided by all partners: donor organisations, implementers, government partners (money, personnel and material).
  
'''Activities '''that particularly target stove supply include: technology development, training of trainers and producers, marketing, and quality control. Activities focused on stove demand include: mainstreaming cooking energy in the public sector, information and awareness campaigns. They could include establishment of a credit scheme, although this activity is not shown on the table as it was not relevant in this project.  
+
'''Activities '''that particularly target stove supply include: technology development, training of trainers and producers, marketing, and quality control. Activities focused on stove demand include: mainstreaming cooking energy in the public sector, information and awareness campaigns. They could include establishment of a credit scheme, although this activity is not shown on the table as it was not relevant in this project.
  
The production '''outputs''' of the interventions comprise: improved stove models available, producers trained and qualified, marketing campaigns implemented and quality control system established. This leads to a well-established production and promotion stove initiative, and more stoves on the market.  
+
The production '''outputs''' of the interventions comprise: improved stove models available, producers trained and qualified, marketing campaigns implemented and quality control system established. This leads to a well-established production and promotion of stove initiatives, and to more stoves on the market.
  
On the consumption side: the '''outputs''' include: increased access to information and knowledge, and household energy mainstreamed into the public sector. This results in more awareness and higher probability of purchase.  
+
On the consumption side: the '''outputs''' include: increased access to information and knowledge, and household energy mainstreamed into the public sector. This results in more awareness and a higher probability of purchase.
  
Between the output and the use of output lies the '''system''' '''boundary of the project'''. Up to this point, the project is directly influencing the results; thereafter the success of the intervention is influenced by the targeted groups and their behaviour. But the project is still responsible for achieving results, and therefore the importance of participation and ownership of the target groups becomes very crucial.  
+
Between the output and the use of output lies the '''system''' '''boundary of the project'''. Up to this point, the project is directly influencing the results; thereafter the success of the intervention is influenced by the targeted groups and their behaviour. But the project is still responsible for achieving results, and therefore the importance of participation and ownership of the target groups becomes very crucial.
  
The increased '''use of the output''', in this case improved biomass stoves, depends on the future users, the project environment and the levels of promotion provided by producers and sales personnel. Support to manufacturers and supplies should be established, or strengthened, by the project.  
+
The increased '''use of the output''', in this case improved biomass stoves, depends on the future users, the project environment and the levels of promotion provided by producers and sales personnel. Support to manufacturers and supplies should be established, or strengthened by the project.
  
The '''outcome''' of the whole exercise is an increase in access to modern cooking energy technologies. This is the target of the EAP. It enables stove producers to sell their technologies, to generate income and to facilitate the increased involvement of women. Provided that the users cook effectively and efficiently with their new stoves, there will be reduced biomass consumption, a positive impact on time and money, and a reduction in GHG emissions, indoor air pollution, and accidents.  
+
The '''outcome''' of the whole exercise is an increase in access to modern cooking energy technologies. This is the target of the EAP. It enables stove producers to sell their technologies, to generate income and to facilitate the increased involvement of women. Provided that the users cook effectively and efficiently with their new stoves, there will be reduced biomass consumption, a positive impact on time and money, and a reduction in GHG emissions, indoor air pollution, and accidents.
  
Further results beyond the outcome are not isolated and cannot be attributed to only one project. This is where the '''attribution gap''' between outcomes and impacts is located. '''Impacts''' are the positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Positive impacts are for example less deforestation, less disease, improved working conditions and more jobs and small/medium enterprises (SMEs) created. A negative impact might be less time for social interaction of women during firewood collection. Where the impacts of scaling up improved biomass stoves finally contribute to the MDGs, they are described as '''highly aggregated impacts'''.  
+
Further results beyond the outcome are not isolated and cannot be attributed to only one project. This is where the '''attribution gap''' between outcomes and impacts is located. '''Impacts''' are the positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Positive impacts are for example less deforestation, less diseases, improved working conditions, more jobs and the creation of small/medium enterprises (SMEs). A negative impact might be less time for social interaction of women during firewood collection. Where the impacts of scaling up improved biomass stoves finally contribute to the MDGs, they are described as '''highly aggregated impacts'''.
  
Note: This is not a cast-in-stone system.&nbsp;The allocation of the different result levels may vary according to the project approach. The boundaries are not fixed, but fluent.  
+
Note: This is not a cast-in-stone system. The allocation of the different result levels may vary according to the project approach. The boundaries are not fixed, but fluent.
  
<br>
+
<br/>
  
RBM&nbsp;is intended to measure the progress and success of the project towards achieving its objectives. To measure this progress, '''indicators '''<span>have to be developed. They are quantitative or qualitative values that describe the real situation, and indicate the degree of change. Ideally, these indicators will be measured at the beginning of the project (baseline), during the project, at the end of the project, and perhaps several years later. </span>
+
RBM is intended to measure the progress and success of the project towards achieving its objectives. To measure this progress, '''indicators '''<span>have to be developed. They are quantitative or qualitative values that describe the real situation, and indicate the degree of change. Ideally, these indicators will be measured at the beginning of the project (baseline), during the project, at the end of the project, and perhaps several years later.</span>
  
The '''baseline''' describes the situation prior to the development intervention. It provides a basis against which to monitor whether the intervention has achieved the desired results. Through '''indicators''', project managers are able to trace back the achievement of set targets, to identify unexpected changes, and to identify unintended impacts. It is possible to determine whether outputs achieved have really proved useful, and whether their use has really lead to a worthwhile outcome in development terms. If necessary, the project strategy can be adjusted, additional activities can be included, or further key stakeholders can be involved.  
+
The '''baseline''' describes the situation prior to the development intervention. It provides a basis against which to monitor whether the intervention has achieved the desired results. Through '''indicators''', project managers are able to trace back the achievement of set targets, to identify unexpected changes and unintended impacts. It is possible to determine whether the outputs achieved have really proved useful, and whether their use has really lead to a worthwhile outcome in terms of development . If necessary, the project strategy can be adjusted, additional activities can be included, or further key stakeholders can be involved.
  
&nbsp;&nbsp;
+
<br/>
  
'''Additional information<br>'''
+
== Further Information<br/> ==
  
{| style="width: 765px; height: 324px" cellspacing="1" cellpadding="1" width="765" border="1"
+
{| style="width: 765px; width: 765px;  height: 324px" cellspacing="1" cellpadding="1" border="1"
 
|-
 
|-
| Results chain Ethiopia  
+
| Results chain Ethiopia
| [[Image:Results chains ethiopia.pdf]]
+
|  
 +
[[File:Results Chains for Ethiopia.pdf|180px|File:Results Chains for Ethiopia.pdf|alt=File:Results Chains for Ethiopia.pdf]]
 +
 
 +
(Click twice for readable version)
 +
 
 
|-
 
|-
| [http://energypedia.info/images/d/d0/Example_ics_results_chains_kenya.doc Results chain Kenya]
 
 
|  
 
|  
|-
+
[[:File:Example of Improved Cookstoves Result Chains in Kenya.doc|Results chain Kenya]]
| GTZ (1996) – Measuring Successes and Setbacks. How to Monitor and Evaluate Household Energy Projects
+
 
| [[Image:Howmonitorandevaluatebec.pdf|105x149px|Howmonitorandevaluatebec.pdf]]
+
| <br/>See hyperlink to the left
 
|}
 
|}
  
<br>
+
<br/>
 +
 
 +
<br/>
 +
 
 +
= <span lang="EN-GB"><font face="Arial">Monitoring</font></span> of Access to Cooking Energy<br/> =
 +
 
 +
To describe the outcomes (depending on the objective of the intervention), all information about access to cooking energy services should be monitored. In this instance, people with access to modern and clean cooking energy would be monitored by sales or construction figures, as well as the correct usage by households, social institutions and SMEs.<br/>The two key results are usually considered to be stoves sales and correct stoves use, since all further outcomes and impacts depend on stoves on the market and their correct use. The stove’s effects very much depend on the capacity of the user to achieve maximum reduction of fuel use and emissions.
 +
 
 +
<u>Therefore, the following is monitored on a regular basis by GIZ interventions:</u>
 +
 
 +
*production and sales figures (incl. costs and prices)
 +
*purchase and use data
 +
 
 +
<br/>
 +
 
 +
<br/>
 +
 
 +
== Further Information<br/> ==
 +
 
 +
The GIZ programme Energising Development has developed an elaborate monitoring tool for energy access programmes: <span style="color: rgb(255, 0, 0)">[[:file:Technische Unterstützung für wirkungsorientiertes Monitoring.pdf|Technische Unterstützung für wirkungsorientiertes Monitoring: Arbeitshilfe für den Einsatz von Microsoft Excel]]</span>
 +
 
 +
<br/>
 +
 
 +
<br/>
 +
 
 +
<br/>
 +
 
 +
= Monitoring of Impacts - Methodology =
 +
 
 +
Cooking energy projects that enable access to improved energy, contribute to the achievement of the MDGs. A plausible hypothesis on the project’s contribution to the MDGs should be provided, even though the impacts cannot be monitored, and thus cannot be attributed directly to the intervention. However, after a certain time of implementation it is worthwhile to do an impact assessment to determine whether the assumptions/ hypotheses about intended impacts, on which the project was based, have been achieved.<br/>With these impact assessments, very specific results can be monitored and compared with the baseline. These include: income generation from stove production, women engaged in new activities, time and expenses saved, perceived reductions in the levels of indoor air pollution, and reduction in the number of accidents. Conclusions regarding the contribution to the achievement of higher aggregated impacts, such as increased business development, women’s empowerment, decreased deforestation, decreased respiratory and eye diseases, and finally the MDGs can be plausibly demonstrated with the results of the impact assessment.
 +
 
 +
To collect empirical data on the impacts of energy projects, a wide range of methods and approaches is available. These may differ significantly in terms of time, money and expertise needed for implementation.
 +
 
 +
<br/>
  
<br>
+
== Qualitative Methods ==
  
= <span lang="EN-GB"><font face="Arial">Monitoring</font></span> of Access to Cooking Energy  =
+
Qualitative methods emphasise personal experience and interpretation. They are well suited to reveal background information and detailed opinions regarding particular themes. The information gained through qualitative methods is not necessarily valid for the total target population. This type of interview is generally conducted for a small number of sampling units. It is advisable to make use of semi-structured questionnaires containing several open questions.
  
To describe the outcomes (depending on the objective of the intervention), all information about access to cooking energy services should be monitored. In this instance, people with access to modern and clean cooking energy would be monitored by sales or construction figures, as well as monitoring correct usage by households, social institutions and SMEs.<br>The two key results are usually considered to be stoves sales and correct stoves use, since all further outcomes and impacts depend on stoves on the market and correct use.&nbsp;The stove’s effects very much depend on the capacity of the user to achieve maximum reduction of fuel use and emissions.
+
<br/>
  
What is monitored therefore on a regular basis by GIZ interventions is:<br>- production and sales figures (incl. costs and prices)<br>- purchase and use data (the correct use can be veryfied e.g. with <span style="color: #ff0000">KPTs</span>)
+
== Focus Group Discussions ==
  
&nbsp;
+
Focus Group Discussions are one type of qualitative methods, allowing interviewers to study people in a more natural setting than in a one-to-one interview. Focus Group Discussions are low in cost, one can get results relatively quickly, and they provide several opinions by talking to various people at once. Care must be taken during these discussions to ensure that those who make the most noise do not dominate the groups with their views.
  
'''Additional information'''
+
<br/>
  
The GIZ programme&nbsp;Energising Development has developed an elaborate monitoring tool for energy access programmes. For any questions please contact [mailto:robert.heine@giz.de <span style="color: #ff0000">robert.heine@giz.de</span>]<span style="color: #ff0000">&nbsp;</span>&nbsp;
+
=== Observations ===
  
&nbsp;
+
Observation of the target groups, villages and project interventions through local field staff is a cheap and simple way of gaining a first impression of energy impacts. Even though not as accurate and representative as structured surveys, observations can be especially useful for preparing further scientific research (e.g. questionnaires) by providing site-specific knowledge.
  
= Assessing Impacts - Methodology  =
+
<br/>
  
Cooking energy projects that enable access to improved energy, contribute to achieve the MDGs. A plausible hypothesis on the project’s contribution to the MDGs should be provided, even though the impacts cannot be monitored, and thus cannot be attributed directly to the intervention.&nbsp;However, after a certain time of implementation it is worthwhile to do an impact assessment to determine whether the assumptions/ hypotheses about intended impacts, on which the project was based, have been achieved. <br>With these impact assessments very specific results can be monitored and compared with the baseline. These include: income generation from stove production, women engaged in new activities, time and expenses saved, perceived reductions in the levels of indoor air pollution, and reduction in the number of accidents. Conclusions regarding the contribution to the achievement of higher aggregated impacts, such as increased business development, women’s empowerment, decreased deforestation, decreased respiratory and eye diseases, and finally the MDGs can be plausibly demonstrated with the results of the impact assessment.
+
== Quantitative Methods ==
  
To collect empirical data on the impacts of energy projects, a wide range of methods and approaches is available. These may differ significantly in terms of time, money and expertise needed for implementation.  
+
Quantitative approaches are concerned to quantify social phenomena by collecting and analysing numerical data in a large number. Sampling and questionnaires have to fulfil certain criteria for allowing statistically reliable and valid data to be gained.
  
The main challenge in determining impacts results from the fact that it is impossible to observe how people undergoing an intervention would have acted if this intervention had not occurred. Surveying also control or comparison groups that have not been part of the project intervention helps to avoid wrong conclusions. <br>The control group is hence included in order to simulate how the target group would have developed if the intervention would not have taken place. In order to effectively fulfil this task, this control group should be composed of individuals that have not undergone the intervention, but live under similar general conditions and exhibit – on average – similar socio-economic characteristics (e.g. educational background). <br>Please note that households without ICS in the target region should not serve as control group since the decision to buy a stove might have been influenced by variables such as education, income, motivation and other household characteristics. <br><span style="color: #008000">'''More information on control groups and rigorous monitoring approaches: link? '''</span>
+
<br/>
  
&nbsp;
+
=== Experimental / Quasi-experimental Designs ===
  
== Qualitative Methods  ==
+
The main challenge in determining impacts results from the fact that it is impossible to observe how people undergoing an intervention would have acted if this intervention had not occurred. Additionally, surveying control or comparison groups that have not been part of the project intervention helps to avoid wrong conclusions.<br/>The control group is hence included in order to simulate how the target group would have developed if the intervention had not taken place. In order to effectively fulfil this task, this control group should be composed of individuals that have not undergone any intervention, but live under similar general conditions and exhibit – on average – similar socio-economic characteristics (e.g. educational background).<br/>Please note that households without ICS in the target region should not serve as a control group since the decision to buy a stove might have been influenced by variables such as education, income, motivation and other household characteristics.
  
Qualitative methods emphasise personal experience and interpretation. They are well suited to reveal background information and detailed opinions regarding particular themes. The information gained through qualitative methods is not necessarily valid for the total target population. This type of interview is generally conducted for a small number of sampling units. It is advisable to make use of semi-structured questionnaires containing several open questions.
+
<br/>
  
=== <br>Focus Group Discussions  ===
+
=== Interview Types and Questionnaires ===
  
Focus Group Discussions are one type of qualitative method, allowing interviewers to study people in a more natural setting than in a one-to-one interview. Focus Group Discussions are low in cost, one can get results relatively quickly, and they provide several opinions by talking to several people at once. Care must be taken during these discussions to ensure that those who make the most noise do not dominate the groups with their views.  
+
Standardized interviews (also known as structured interviews) are used in quantitative surveys to obtain comparable information representative for the total target group. Structured interviews are the most common form used in impact assessments. The wording of questions and response items as well as their sequence is identical for all respondents. They use close-ended questions.<br/>In semi-standardized questionnaires, additional qualitative information can be gathered by the use of open questions.
  
=== <br>Observations  ===
+
<br/>
  
Observation of the target groups, villages and project interventions through local field staff is a cheap and simple way of gaining a first impression of energy impacts. Even though not as accurate and representative as structured surveys, observations can be especially useful for preparing further scientific research (e.g. questionnaires) by providing site-specific knowledge.
+
<br/>
  
&nbsp;
+
== Sample ==
  
== Quantitative Methods  ==
+
It is usually not advisable to survey the whole population in the project region. This would not only require a large amount of time and efforts, but with larger populations it is just not manageable. Conducting interviews with all people present in the study area is only advisable for small, manageable numbers of sampling units. In most cases it is sufficient to draw smaller samples, since beyond a certain number, results are repetitive and the significance is not really affected by increasing sample size.
  
Quantitative approaches are concerned to quantify social phenomena by collecting and analysing numerical data in a large number. Sampling and questionnaires have to fulfil certain criteria for allowing statistically reliable and valid data to be gained.  
+
The sample is fundamental for quantitative surveys. To be able to gather information from the interviewees valid for the total target population, the size of the sample as well as its representativeness is key. It has to be statistically significant. As a rule of thumb for household surveys, it is advisable to draw a sample of at least 40 to 50 households for each stratum (see below: stratified sample). In the case of very small numbers of sampling units, e.g. a very limited number of hospitals in the project area, it may be advisable to opt for qualitative approaches.<br/>There are different ways of sampling.
  
&nbsp;
+
<u>Depending on the survey context, one of the following sampling methods may be chosen:</u>
  
=== Interview Types and Questionnaires  ===
+
<br/>
  
Standardized interviews (also known as structured interviews) are used in quantitative surveys to obtain comparable information representative for the total target group. Structured interviews are the most common form in impact assessments. The wording of questions and response items as well as their sequence is identical for all respondents. They use close-ended questions. <br>In semi-standardized questionnaires additional qualitative information can be gathered by the use of open questions.<br>
+
=== Random Sample / Systematic Sample<br/> ===
  
== <br>Sample  ==
+
In a random sample, every individual unit in the population has an equal chance of being selected for the sample; selection occurs by chance, so knowledge of the total population group is needed to ensure that everyone is represented. A totally random selection of households is difficult in practice, and the systematic sample is a common technique to draw near-random samples. For a systematic sample, every xth unit is selected (e.g. interviews are conducted in every third household on the interviewers’ way, beginning from a randomly selected starting point).<br/>Random / systematic samples provide information about distributions within the whole population. For example, to gain information about the percentage of efficient stoves existing in a village, a random sample should be drawn.
  
It is usually not advisable to survey the whole population in the project region. This would not only require a large amount of time and efforts, with larger populations it is just not manageable. Conducting interviews with all people present in the study area is only advisable for small, manageable numbers of sampling units. In most instances it is sufficient to draw smaller samples, since beyond a certain number, results are repetitive and the significance is not really affected by increasing same size.
+
<br/>
  
The sample is fundamental for quantitative surveys. To be able to gather information from the interviewees valid for the total target population the size of the sample as well as its representativeness is key. It has to be statistically significant. As a rule of thumb for household surveys, it is advisable to draw a sample of at least 40 to 50 households for each stratum (see below: stratified sample). In the case of very small numbers of sampling units, e.g. a very limited number of hospitals in the project area, it may be advisable to opt for qualitative approaches. <br>There are different ways of sampling. Depending on the survey context, one of the following sampling methods may be chosen:
+
=== Stratified Sample<br/> ===
  
=== <br>Random sample / Systematic sample ===
+
Stratified samples are used to ensure that appropriate numbers of small subgroups are included in the sample. This could apply to surveys focusing on technologies that have not yet been widely disseminated, such as Solar Cookers. In this case ownership/non-ownership of a solar cooker might be a typical variable. Others might include socio-economic groups, districts, professional groups, etc. The percentage drawn from each stratum might be identical or variable, according to the needs of the survey. In the case of different percentages drawn for each stratum, data has to be weighted for analysis to represent the population correctly. For example, in a sample stratified by two household income groups, with 10% of all high-income households and 30% of all low-income households being interviewed, the data for high-income households has to be weighted by a factor of three. To develop a stratified sample, and to structure the sample accurately requires a lot of reliable information about the total population group. This is a challenge in many cases, since available information is often limited.
  
In a random sample, every individual unit in the population has an equal chance of being selected for the sample; selection occurs by chance, so a knowledge of the total population group is needed to ensure that everyone is represented. A totally random selection of households is difficult in practice, and the systematic sample is a common technique to draw near-random samples. For a systematic sample, every xth unit is selected (e.g. interviews are conducted in every third household on the interviewers’ way, beginning from a randomly selected starting point).<br>Random / systematic samples provide information about distributions within the whole population. For example, to gain information about the percentage of efficient stoves existing in a village, a random sample should be drawn.
+
<br/>
  
=== <br>Stratified sample  ===
+
=== Interview Partner<br/> ===
  
Stratified samples are used to ensure that appropriate numbers of small subgroups are included in the sample. This could apply to surveys focusing on technologies that have not yet been widely disseminated, such as Solar Cookers. In this case ownership/non-ownership of a Solar Cooker might be a typical variable. Others might include socio-economic groups, districts, professional groups, etc. The percentage drawn from each stratum might be equal or variable, according to the needs of the survey. In the case of different percentages drawn for each stratum, data has to be weighted for analysis to represent the population correctly. For example, in a sample stratified by two household income groups, with 10% of all high-income households and 30% of all low-income households being interviewed, the data for high-income households has to be weighted by a factor of three. To develop a stratified sample, and to structure the sample accurately, requires a lot of reliable information about the total population group. This is a challenge in many cases, since available information is sometimes limited.  
+
After drawing the sample on level of households, SMEs or social institutions, the sampling method for selecting the right interview partner should also be thought about . A pure random sample would e.g. be drawn by asking for the person in the household whose birthday was the most recent. In most cases it is the cook who is interviewed in cooking energy surveys. However interview partner can also be the head of the household or the manager of a SME or a social institution. Please consider that those persons may not be able to answer questions regarding the daily use of the stove.
  
&nbsp;
+
<br/>
  
=== Interview partner  ===
+
<br/>
  
After drawing the sample on level of households, SMEs or social institutions it should also be thought about the sampling method for selecting the right interview partner. A pure random sample would e.g. be drawn by asking for the person in the household whose birthday was the most recent. In most cases it is the cook who is interviewed in cooking energy surveys. However interview partcer can also be the head of the household, the manager of a SME or a social institution. Please consider that those persons may not be able to answer questions regarding the daily use of the stove.
+
== Review of Documents and Statistics ==
  
<br>
+
Besides information that needs to be gathered from interviews or observations, data on particular indicators might be readily available from local databases, statistics and registers.
  
== Review of Documents and Statistics  ==
+
<br/>
  
Besides information that needs to be gathered from interviews or observations, data on particular indicators might be readily available from local databases, statistics and registers.
+
== Procedure for Developing a Questionnaire and Interview Guidelines<br/> ==
  
&nbsp;
+
*Analyse the project strategy
 +
*Consider the projects results chains – do they cover all the relevant results hypotheses to be evaluated in the impact assessment? Identify relevant assessment fields for the impact assessment along results chains
 +
*Formulate results indicators and convert these into questions and answers, making sure both questions and categories from which an answer is to be chosen, are easy to understand.
 +
*To achieve the best response rates, questions should rank from the least sensitive to the most sensitive one. Questions regarding socio-economic data are thus placed at the end of the questionnaire.
 +
*Avoid suggestive questions- they force the respondents into a corner and reducing their freedom to answer the questions. Here is an example: "Many scientists believe that stove emissions can negatively impact your health. Do you think this view is correct?"
 +
*It may be necessary to translate the questionnaire into a local language.
 +
*Pilot test your questionnaire before rolling it out to all the involved households.
  
<br>
+
<br/>
  
'''Procedure for developing a questionnaire and interview guidelines: <br>'''Analyse the project strategy <br>Consider the projects results chains – do they cover all the relevant results hypotheses to be assessed in the impact assessment? <br>Identify relevant assessment fields for the impact assessment along results chains <br>Formulate results indicators and convert these into questions and answers, making sure it is easy to understand both the questions and the categories from which an answer is to be chosen. <br>To achieve the best response rates, questions should flow from the least sensitive to the most sensitive. Questions regarding socio-economic data are thus placed at the end of the questionnaire.<br>Avoid suggestive questions.<br>It may be necessary to translate the questionnaire into a local language. <br>Pilot test your questionnaire before rolling it out to all the involved households. <br>
+
<u>'''Please consider:'''</u><br/>There can be a lot of undesired effects on the respondents and his/her answers, both from the questionnaire and the interviewer. These effects can lead to distorted data.
  
&nbsp;
+
*Sequence of items: Typical effects from questionnaires can arouse from the sequence of answer items, e.g. primacy-recency effects. Respondents remember first or last answer items best and forget about the items in the middle of a long list of answer possibilities.
 +
*Social-desirability: people tend to answer not according to their own opinion but according to social norms or what the respondent thinks would be the desired answer for the interviewer. In development projects where the interviewer is associated with money or other advantages this might be a serious problem.
  
'''Please consider:<br>'''There can be a lot of undesired effects on the respondents and his/her answers, both from the questionnaire and the interviewer. These effects can lead to a biased data.
+
<br/>
  
• Sequence of items: Typical effects from questionnaires can arouse from the sequence of answer items, e.g. primacy-recency effects. Respondents remember first or last answer items best and forget about the items in the middle of a long list of answer possibilities.
+
== Further Information on Methodology<br/> ==
  
• Social-desirability: people tend to answer not according to their own opinion but according to social norms or what the respondent thinks would be the desired answer for the interviewer. In development projects where the interviewer is associated with money or other advantages this might be a serious problem.  
+
*[http://www.socialresearchmethods.net/kb/index.php Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page] (version current as of October 20, 2006).
 +
*Baker, J. L. (2000): '''[http://books.google.de/books?id=AMgvY6vpFN8C&lpg=PR6&ots=g_TivF0nbL&dq=Baker, J. L. (2000): Evaluating the Impact of Development Projects on Poverty. A Handbook for Practioners.&lr&pg=PP1#v=onepage&q&f=false Evaluating the Impact of Development Projects on Poverty. A Handbook for Practioners.]''' The World Bank, Washington, D.C.
 +
*SCHUMANN, Siegfried (2000): '''[http://books.google.de/books?id=VJNZyIEWjZAC&printsec=frontcover&dq=schumann+statistik&source=bl&ots=3dk40unUki&sig=aZdrocVquFZDMMW7X8Vswfcp0Eo&hl=de&ei=I7amTZiuLM-Rswa-wfGPCA&sa=X&oi=book_result&ct=result&resnum=7&ved=0CDQQ6AEwBg#v=onepage&q=schumann statistik&f=false Repräsentative Umfrage. Praxisorientierte Einführung in empirische Methoden und statistische Analyseverfahren.]''' 3., überarb. Auflage. München, Wien: Oldenbourg.
 +
*Eurostat (2006): '''[http://epp.eurostat.ec.europa.eu/portal/page/portal/research_methodology/documents/Handbook_questionnaire_development_2006.pdf Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System]''' This handbook gives a good overview on general questionnaire design, including conceptual frame and writing of questions (types of questions, possible distorting effects of questions,e tc.)
  
&nbsp;
+
<br/>
  
== Additional information  ==
+
<br/>
  
SCHUMANN, Siegfried (2000): Repräsentative Umfrage. Praxisorientierte Einführung in empirische<br>Methoden und statistische Analyseverfahren. 3., überarb. Auflage. München, Wien: Oldenbourg.
+
<br/>
  
Research methods knowledge base: [http://www.socialresearchmethods.net/kb/index.php http://www.socialresearchmethods.net/kb/index.php]&nbsp;
+
== Example Kenya ==
  
== Example Kenya ==
+
The Kenyan impact assessment was conducted together with national and regional counterpart staff with the intention to create capacity and increase ownership. Involvement of project staff, counterparts, local NGOs and students in various parts of the assessment (like preparation and data interpretation) was helpful for the exchange among all stakeholders. In Kenya, agricultural officers reported the enthusiasm of the targeted beneficiaries. One student who conducted the interviews in Kenya started his own stove business. Such examples show the secondary effect of these monitoring instruments, which might not have been anticipated.
  
The Kenyan impact assessment was conducted together with national and regional counterpart staff with the intention to create capacity and increase ownership. Involvement of project staff, counterparts, local NGOs and students in various parts of the assessment (like preparation and data interpretation) is helpful for the exchange among all stakeholders. In Kenya, agricultural officers reported the enthusiasm of the targeted beneficiaries. One student who conducted the interviews in Kenya started his own stove business. Such examples show the secondary effect of these monitoring instruments, which might not have been anticipated.  
+
In addition, a national consultant was involved in the whole process of impact assessment, from advising on the crucial indicators to questionnaire development, to taking part in the data entry, analysis, and interpretation. Enumerators (local students) were trained to conduct the interviews.
  
In addition a national consultant was involved in the whole process of impact assessment, from advising on the crucial indicators through questionnaire development, and taking part in the data entry, analysis, and interpretation. Enumerators (local students) were trained to conduct th einterviews.&nbsp;
+
The target group representatives in districts and villages were selected using a set of selection criteria. Qualitative interviews using [[:File:Local Chiefs Interview Guidelines.pdf|interview guidelines]] were conducted with selected local authorities. Selected women’s groups engaged in focus group discussions at village level using a set of tools developed for Participatory Rural Appraisal (<font color="#002bb8">[[:file:Gtz Pra-tools women group Kenya.pdf|PRA tools]]</font>). These included elements such as trend analyses, activity lists, and influence matrices. Trained PRA experts were included in this exercise.
  
The&nbsp;target group representatives in districts and villages were selected&nbsp;using a set of selection criteria. Qualitative interviews using [http://www.energypedia.info/index.php/File:Interview_guide_local_chief.pdf <font color="#002bb8">interview guidelines</font>]&nbsp;were conducted with selected local authorities. Selected women’s groups engaged in focus group discussions at village level, using a set of tools developed for Participatory Rural Appraisal (<font color="#002bb8">[http://www.energypedia.info/index.php/File:Gtz_Pra-tools_women_group_Kenya.pdf PRA tools]</font>). These included elements such as trend analyses, activity lists, and influence matrices. Trained PRA experts were included in this exercise.  
+
Finally structured interviews were conducted to produce a quantitative sample survey, administering questionnaires with the selected users (<font color="#002bb8">[[:File:Questionnaires for Households with and without Improved Cookstoves in Kenya.pdf|households]]</font>, [[:File:Questionnaire for Social Institutions in Kenya.pdf|schools and]] <font color="#002bb8">[[:File:Questionnaire for Productive Use of Improved Cookstoves in Restaurants in Kenya.pdf|restaurants]]</font>) and producers. These interviews were carried out by carefully chosen and trained enumerators, in most cases local students.
  
Finally structured interviews were conducted to produce a quantitative sample survey, administering questionnaires with the selected users (<font color="#002bb8">[http://www.energypedia.info/index.php/File:Gtz_Example_ics_hh_questionnaire_stoves_kenya.pdf households]</font>, [http://www.energypedia.info/index.php/File:Gtz_Questionnaire_for_si_Kenya.pdf <font color="#002bb8">schools</font>&nbsp;]and <font color="#002bb8">[http://www.energypedia.info/index.php/File:Gtz_Example_ics_restaurant_questionnaire_kenya.pdf restaurants]</font>) and producers. These interviews were carried out by and carefully chosen and trained enumerators, in most cases local students.
+
<br/>
  
{| cellspacing="1" cellpadding="1" width="100%" border="1"
+
{| style="width: 100%;  width: 100%" cellspacing="1" cellpadding="1" border="1"
 +
|-
 +
| '''Interview guidelines'''<br/>
 +
| '''PRA tools'''<br/>
 
|-
 
|-
| Interview guidelines
+
| [[File:Local Chiefs Interview Guidelines.pdf|180px|File:Local Chiefs Interview Guidelines.pdf|alt=File:Local Chiefs Interview Guidelines.pdf]]<br/>
| PRA tools  
+
| [[File:Gtz Pra-tools women group Kenya.pdf|none|180px|Gtz Pra-tools women group Kenya.pdf|alt=Gtz Pra-tools women group Kenya.pdf]]<br/>
| Housholds
 
| Schools
 
| Restaurants
 
 
|-
 
|-
| [[Image:Interview guide local chief.pdf]]
+
| '''Housholds'''
| [[Image:Gtz Pra-tools women group Kenya.pdf]]  
+
| '''Schools'''
| [[Image:Gtz Example ics hh questionnaire stoves kenya.pdf]]
+
| '''Restaurants'''
| [[Image:Gtz Questionnaire for si Kenya.pdf]]
+
|-
| [[Image:Gtz Example ics restaurant questionnaire kenya.pdf]]
+
|
|}
+
[[File:Questionnaires for Households with and without Improved Cookstoves in Kenya.pdf|180px|File:Questionnaires for Households with and without Improved Cookstoves in Kenya.pdf|alt=File:Questionnaires for Households with and without Improved Cookstoves in Kenya.pdf]]<br/>
 +
 
 +
|  
 +
[[File:Questionnaire for Social Institutions in Kenya.pdf|180px|File:Questionnaire for Social Institutions in Kenya.pdf|alt=File:Questionnaire for Social Institutions in Kenya.pdf]]<br/>
  
<br>&nbsp;<br>'''Additional information'''
+
|
 +
[[File:Questionnaire for Productive Use of Improved Cookstoves in Restaurants in Kenya.pdf|180px|File:Questionnaire for Productive Use of Improved Cookstoves in Restaurants in Kenya.pdf|alt=File:Questionnaire for Productive Use of Improved Cookstoves in Restaurants in Kenya.pdf]]<br/>
  
<br>
+
|}
  
{| cellspacing="1" cellpadding="1" width="100%" border="1"
+
{| style="width: 100%" cellspacing="1" cellpadding="1" border="1"
 
|-
 
|-
| GTZ EnDev Guide Impact Assessment for Energising Development Projects, GTZ 2007
+
| <div>
| [[Image:07.09.11 impact monitoring guide energising development.he.pdf]]
+
The [[:File:Gtz-kenya-resultsassessment-final-nov-2009.pdf|final report "Result assessment"]] in Kenya was published in 2009.
 +
</div>
 +
| <br/>
 
|-
 
|-
| [http://energypedia.info/index.php?title=File:Gtz-kenya-resultsassessment-final-nov-2009.pdf&page=1 The final report on Impact Assessment Kenya was published in 2009.]
+
| Model List of Impacts, Observation Fields and Indicators for IA Kenya, GTZ HERA 2007<br/>
 
|  
 
|  
|-
 
| Model List of Impacts, Observation Fields and Indicators for IA Kenya, GTZ HERA 2007
 
| [[Image:Preparation kenya - impacts, observation fields and indicators.pdf]]
 
 
|-
 
|-
 
|  
 
|  
[http://energypedia.info/index.php?title=File:Steps_and_procedures.pdf&page=1 Steps and Procedures - documented&nbsp;according to IA in Kenya. GTZ HERA 2007]  
+
[[:File:Steps and Procedures for Impact Assessment in Kenya.pdf|Steps and Procedures - documented according to IA in Kenya. GTZ HERA 2007]]
  
These describe the procedure of IA&nbsp;in Kenya as well as recommendations for&nbsp;training for enumerators and lessons learnt during the IA.
+
These describe the procedure of IA in Kenya as well as recommendations for training for enumerators and lessons learnt during the IA.
  
 
|  
 
|  
 +
<br/>
 +
 
|-
 
|-
 
|  
 
|  
Selection Criteria for IA in Kenya, GTZ HERA 2007
+
[[:file:Selection-criteria.pdf|Selection Criteria for IA in Kenya, GTZ HERA 2007]]
  
| [[Image:Selection-criteria.pdf]]<br>
+
|  
 
|-
 
|-
 
|  
 
|  
A set of <font color="#002bb8">[http://www.energypedia.info/index.php/File:Gtz_Example_hh_guiding_questions_for_data_analysis_v2_Kenya.doc guiding questions]</font>&nbsp;for data analysis and interpretation was developed to support for the consultant in Kenya.
+
A set of <font color="#002bb8">[[:file:Gtz Example hh guiding questions for data analysis v2 Kenya.doc|guiding questions]]</font> for data analysis and interpretation was developed to support for the consultant in Kenya.
 +
 
 +
| <br/>
 +
|}
 +
 
 +
<br/>
 +
 
 +
<br/>
 +
 
 +
<br/>
 +
 
 +
<br/>
 +
 
 +
== Further Information<br/> ==
 +
 
 +
{| style="width: 100%" cellspacing="1" cellpadding="1" border="1"
 +
|-
 +
| colspan="2" |
 +
[[:file:Impact Assessment for EnDev Projects.pdf|GTZ EnDev Guide Impact Assessment for Energising Development Projects, GTZ 2007]]
 +
 
 +
<br/>
 +
 
 +
GIZ EnDev has developed Impact Monitoring Guideline - available on Energypedia
 +
 
 +
[[Impact Monitoring Guidelines|Impact_Monitoring_Guidelines]]
 +
 
 +
<br/>
 +
 
 +
Webinar on Mobile Monitoring for Data Tracking and Management, available on [http://www.pciaonline.org/webinars http://www.pciaonline.org/webinars]<br/>
 +
 
 +
|-
 +
| colspan="2" |
 +
[[EnDev Report on Impacts|GTZ EnDev Report on Impacts]]
 +
 
 +
|-
 +
| colspan="2" |
 +
A [[Indicator Rural Electrification and Stoves#Indicators and Observation Fields for Improved Cooking Stoves .28ICS.29|List of impact indicators]] was developed by GIZ HERA and is available under Energypedia
  
| [[Image:Gtz Example hh guiding questions for data analysis v2 Kenya.doc]]
 
 
|-
 
|-
| Impact Assessment of Mirt stoves, GTZ SUN-E, Ethiopia 2008
+
|  
| [[Image:Mirt impact assessment rpt final.pdf]]
+
<br/>[[File:Mirt impact assessment rpt final.pdf|180px|Mirt impact assessment rpt final.pdf|alt=Mirt impact assessment rpt final.pdf]]
 +
 
 +
| [[File:Impact assessment clay stoves malawi 2008.pdf|180px|Impact assessment clay stoves malawi 2008.pdf|alt=Impact assessment clay stoves malawi 2008.pdf]]
 
|-
 
|-
| Impact Assessment of Chitetezo Mbaula, GTZ ProBEC Malawi, 2008  
+
|  
| [[Image:Impact assessment clay stoves malawi 2008.pdf]]
+
Impact Assessment of Chitetezo Mbaula, GTZ ProBEC Malawi, 2008
 +
 
 +
| Impact Assessment of Mirt stoves, GTZ SUN-E, Ethiopia 2008
 
|}
 
|}
  
<br>
+
<br/>
 +
 
 +
<br/>
 +
 
 +
= Participatory Impact Assessment – An experiment?<br/> =
 +
 
 +
In selected SADC countries, where the Programme for Biomass Energy Conservation, ProBEC, is implemented, impact assessment interviews were conducted by local stove artisans. In 2004, the stove promoters and producers from Malawi, Mozambique, Zimbabwe, Zambia, Tanzania and Kenya were invited to Malawi for a ProBEC Workshop about experience exchange on low-cost clay and ceramics.
 +
 
 +
Very interesting feedback was given by producers and promoters during the workshop concerning the results of impact assessment. For many of them, it was the first time they had exchanged ideas with their customers, and thus they learnt firsthand about how the stoves were used, the way people had understood stove instructions, difficulties, problems and demands.
 +
 
 +
The stove producers, builders, and promoters considered this to be so important that they passed a resolution asking for training in monitoring and impact assessment as part of their regular training programme. They realized that by asking the questions themselves, it would increase their awareness of the quality of their stoves, improve their marketing skills and thus their access to customers.
 +
 
 +
This approach to impact assessment requires high involvement by the producers, and creates business awareness. It should be considered as complementary or additional to an external, unbiased impact assessment by an external person. The results of an assessment by the manufacturers should not be considered neutral.
 +
 
 +
<br/>
 +
 
 +
== Further Information<br/> ==
 +
 
 +
*Brinkmann & Klingshirn: Participatory Impact Assessment, 2005:
 +
 
 +
[[File:Questionnaire assessment vb 052005.pdf|180px|Questionnaire assessment vb 052005.pdf|alt=Questionnaire assessment vb 052005.pdf]]
 +
 
 +
<br/>
 +
 
 +
= Cost-Benefit-Analysis (CBA)<br/> =
 +
 
 +
The data from impact assessment can provide the starting point for an economic evaluation of the project, including a '''Cost-Benefit-Analysis (CBA)''' and a '''Cost-Effectiveness Analysis (CEA)'''. Analysing both the economic efficiency of the investments, and the benefits deriving from energy efficient stoves on a macro and micro level can be helpful for further lobbying, public relations, and keeping control of the project.
  
= Participatory Impact Assessment – an experiment?  =
+
A [[:file:Economic Evaluation of the Improved Household Cooking Stove Dissemination Programme in Uganda v02 2007 .pdf|Cost Benefit Analysis of the Energy Advisory Project]] in Uganda demonstrated the economic value of the use of Lorena Rocket Stoves for individual households as well as for the public sector. The same was shown in a [[:file:Costs-benefits-institutional-stoves malawi-probec-2008.pdf|Cost Benefit Analysis in ProBEC Malawi for]] social institutions such as schools, hospitals and prisons.
  
In selected SADC countries, where the Programme for Biomass Energy Conservation, ProBEC, is implemented, impact assessment interviews were conducted by local stove artisans. In 2004 the stove promoters and producers from Malawi, Mozambique, Zimbabwe, Zambia, Tanzania and Kenya were invited to Malawi for a ProBEC Workshop about experience exchange on low-cost clay and ceramics.
+
<br/>
  
Very interesting feedback was given by producers and promoters during the workshop about the results of the impact assessment. For many of them, it was the first time they had&nbsp;exchanged ideas with their customers, and thus they learnt firsthand about how the stoves were used, the way people had understood stove instructions, difficulties, problems and demands.
+
== GTZ Cost-Benefit Analysis – Tool<br/> ==
  
The stove producers, builders and promoters considered this so important that they passed a resolution asking for training in monitoring and impact assessment as part of their regular training programme. They realized that by asking the questions themselves, it would increase their awareness of the quality of their stoves, improve their marketing skills and thus their access to customers.  
+
This tool is provided as a [[:file:En-tool-matrix-for-cost-benefit-analysis-2007.xls|set of prepared excel sheets]], where all data and information required for the conduction of an economic CBA needs to be entered into reserved cells. In addition [[:file:Introduction and Explanation to the Benefit Matrix (2007) .pdf|GTZ HERA provides explanation on how to use the CBA tool]].
  
This approach to impact assessment requires high involvement by the producers, and creates business awareness. It should be considered as complementary or additional to an external, unbiased impact assessment by an external person. The results of an assessment by the manufacturers should not be considered neutral.
+
<br/>
  
Additional information
+
== Further Information<br/> ==
  
*Brinkmann &amp; Klingshirn: Participatory Impact Assessment, 2005:
+
An evaluation of the costs and benefits of household energy and health interventions at global and regional levels as well as guidelines for conducting such cost-benefit analyses can be found at the [http://www.who.int/indoorair/interventions/cost_effectiveness/en/index.html WHO website]
  
[[Image:Questionnaire assessment vb 052005.pdf]]<br>
+
WHO used the '''Cost-effectiveness analysis (CEA)''' as a tool for assessing the cost-effectiveness ratio of different interventions regarding indoor air pollution. This was applied to identify the intervention that provides the highest ‘value for money’. CEA results can thus help policy makers to choose the interventions and programmes that maximize health benefits for the available resources.
  
= Cost-Benefit-Analysis  =
+
Three scenarios have been assessed: providing the population with access to cleaner fuels (kerosene, liquid petroleum gas); providing the population with access to improved stoves; providing part of the population with access to cleaner fuels and part of the population with improved stoves.
  
The data from impact assessment can provide the starting point for an economic evaluation of the project, including a Cost-Benefit-Analysis (CBA) and a Cost-Effectiveness Analysis (CEA). Analysing both the economic efficiency of the investments, and the benefits deriving from energy efficient stoves on a macro and micro level can be helpful for further lobbying, public relations and keeping control of the project.
+
<br/>
  
A [http://energypedia.info/index.php/File:Uganda_cba_may_v02_b.pdf Cost Benefit Analysis of the Energy Advisory Project] in Uganda demonstrated the economic value, for individual households as well as for the public sector, of using the Lorena Rocket stove. The same was shown in a [http://energypedia.info/index.php?title=File:Costs-benefits-institutional-stoves_malawi-probec-2008.pdf&page=1 Cost Benefit Analysis in ProBEC Malawi ]for social institutions like schools, hospitals and prisons.
+
<br/>
  
== GTZ Cost-Benefit Analysis –Tool  ==
+
= References<br/> =
  
This tool is provided as a [http://energypedia.info/images/7/76/En-tool-matrix-for-cost-benefit-analysis-2007.xls set of prepared excel sheets], where all data and information required for conduction an economic CBA needs to be entered into reserved cells. In addition [http://energypedia.info/index.php/File:En-cbatool-explanation.pdf GTZ HERA provides explanation on how to use the CBA tool].  
+
This article was originally published by [http://www.giz.de/fachexpertise/html/2769.html GIZ HERA]. It is basically based on experiences, lessons learned and information gathered by GIZ cook stove projects. You can find more information about the authors and experts of the original “Cooking Energy Compendium” in the [[Imprint - GIZ HERA Cooking Energy Compendium|Imprint]].
  
&nbsp;
+
<references />
  
'''Additional information'''
+
<br/>
  
An evaluation of the costs and benefits of household energy and health interventions at global and regional levels as well as guidelines for conducting such cost-benefit analyses can be found at the WHO website:<br>[http://www.who.int/indoorair/interventions/cost_effectiveness/en/index.html <font color="#800080">http://www.who.int/indoorair/interventions/cost_effectiveness/en/index.html</font>]&nbsp;
+
[[Result Based Monitoring of Cookstove Projects#Basics about Results Based Monitoring|Top of the page]]
  
WHO used the Cost-effectiveness analysis (CEA) as a tool for assessing the cost-effectiveness ratio of different interventions regarding indoor air pollution. This was applied to identify the intervention that provides the highest ‘value for money’. CEA results can thus help policy makers to choose the interventions and programmes that maximize health benefits for the available resources.
+
[[GIZ HERA Cooking Energy Compendium|--> Back to Overview GIZ HERA Cooking Energy Compendium]]
  
Three scenarios have been assessed: providing the population with access to cleaner fuels (kerosene, liquid petroleum gas); providing the population with access to improved stoves; providing part of the population with access to cleaner fuels and part of the population with improved stoves.
+
{{#set: Hera category=ICS Supply}}
  
<br>
+
[[Category:Cooking_Energy_Compendium_(GIZ_HERA)]]
 +
[[Category:Cooking_Energy]]
 +
[[Category:Monitoring]]
 +
[[Category:Impacts]]
 +
[[Category:Cooking]]
 +
[[Category:Tools]]
 +
[[Category:Cookstoves]]

Latest revision as of 13:17, 4 September 2018

GIZ HERA Cooking Energy Compendium small.png



Cooking Energy System | Basics | Policy Advice | Planning | Designing and Implementing ICS Supply | Designing and Implementing Woodfuel Supply | Climate Change | Extra


Basics about Results Based Monitoring

Why Monitoring

Projects introduce development changes to make a difference in the lives of target groups. They strive to achieve positive change, and it is essential for them to monitor and evaluate the results of their intervention.

Monitoring serves different purposes:

  • To check that set targets have been met
  • To provide data and information for reviewing the strategy
  • To steer and make changes, where necessary, to an intervention
  • To create ownership among various project actors
  • To provide evidence on progress/ changes/ achievements for national partners (e.g. ministries).

Monitoring requires time, personnel and funds, and thus it needs to be included into the activities and budget planning from the very beginning of the intervention to ensure that it is an integral part of the approach. Often, however, it is only considered at a later stage, when countersteering, reviewing the strategy and using it as a management tool is not possible any more.
It is crucial for capacity development interventions to involve project partners and counterparts into all steps of monitoring (planning, implementation, analysis and interpretation). They themselves learn to use tools and instruments, review their own interventions and receive more ownership. Finally results and recommendations should be be discussed with stakeholders and counterparts to jointly draw conclusions.


Introduction to Results Based Monitoring (RBM)

Results Based Monitoring (RBM) is an international monitoring standard developed and agreed by the OECD DAC (Development Assistance Committee of the Organisation for Economic Co-operation and Development) to monitor development results. Results are defined as development changes that follow directly from an intervention; they can be outputs, outcomes or impacts (intended or unintended, positive and/or negative) resulting from a development intervention. According to the OECD-DAC definition, results occur as causal sequences from a development intervention towards the desired objectives. RBM is a method to examine the result hypotheses in a empirical and systematic way, which should be regularly applied by GIZ programmes.



Results Chains

The basis of RBM are the results chains, which describe how a development interventions, through a step-by-step process, contributes to development results. The intervention starts with the inputs used to perform activities, which lead to the outputs of the project. These outputs, which are used by target groups or intermediaries, lead to outcomes and impacts.

In most cases, it is relatively easy to attribute changes, up to the level that identifies the uses of the output. Beyond this, climbing up to the levels of ‘outcome’ and ‘impact’, external factors influence whether the intended results can be achieved. These external factors can only be controlled or influenced to a certain extent (if at all) by the project or programme. Whether the objectives are met no longer depends solely on the performance of the project, but depends on all the actors and external factors involved. Therefore, achievements are only attributable to the intervention up to a certain level – called the ‘outcome’. Beyond the outcome, effects can no longer be directly linked to this one development intervention. The attribution gap widens at the stage where changes, although observed in the target area, cannot be solely related to project outputs - as the following GIZ figure describes:


GTZ Results Model.jpg

For interpretation and analysis of observed results it is important to have a baseline. Baseline data generated at the beginning of the intervention, providing information about the status quo (people with access to improved cookstoves, woodfuel demand, ...), are still neglected or forgotten by programmes. But the advatage is obvious: the programme is able to relate the observed results with the baseline data and draw respective conclusions about implemented interventions.





Further Information

GTZ (2008): Wirkungsorientiertes Monitoring Leitfaden für die Technische Zusammenarbeit (deutsch)

Englische Version:Results-based Monitoring Guidelines for Technical Cooperation

Wirkungsorientiertes-monitoring-leitfaden-de.pdf
RBM terminology 30 May.doc OECD/DAC (2002) – Glossary of Key Terms in Evaluation and Results Based Management

GVEP M&E Group (2006) - A Guide to Monitoring and Evaluation for Energy Projects
For setting up Results Based Monitoring for Energy projects compare

This Guide proposes a step by step approach to building project-specific M&E procedures. The guide is intended for energy access projects, which don’t have already donor or stakeholder determined M&E methods. The guide was developed by the International Working Group on Monitoring and Evaluation in Energy for Development (M&EED)

Meed guide final version english-1-.pdf

Boiling Point No 55 "Monitoring and Evaluation" (June 2008) on the web: http://www.hedon.info/


Toolkit - Six steps to Results Based Monitoring
The following six steps should help you to set up your Results Based Monitoring (RBM)
system. From identifying all stakeholders in the system, formulating the assumptions
on which your strategy is based, analysing risks and side effects, choosing observation
fields, specifying indicators for measurement up to their operationalisation to data
collection – and there you go!
Author: Melanie Djedje
In: Boiling Point 55 (2008)

www.hedon.info/tiki-download_item_attachment.php?attId=65

(paste web address in your browser to open)

GIZ-Community on Results-Based Monitoring and Evaluation
More information about Impact Monitoring on Energypedia - Impacts
GTZ (1996) – Measuring Successes and Setbacks. How to Monitor and Evaluate Household Energy Projects
Suivi des impacts




Results Chains for Cooking Energy Interventions

The following example of the Energy Advisory Project (EAP) in Uganda shows two main chains for ‘Scaling up of improved biomass stoves’:

  • for stove supply; targeted at producers and traders
  • for stove demand; targeting users and the public sector
Results chain.JPG


Description of Figure:

Input is not mentioned here, but comprises all the resources provided by all partners: donor organisations, implementers, government partners (money, personnel and material).

Activities that particularly target stove supply include: technology development, training of trainers and producers, marketing, and quality control. Activities focused on stove demand include: mainstreaming cooking energy in the public sector, information and awareness campaigns. They could include establishment of a credit scheme, although this activity is not shown on the table as it was not relevant in this project.

The production outputs of the interventions comprise: improved stove models available, producers trained and qualified, marketing campaigns implemented and quality control system established. This leads to a well-established production and promotion of stove initiatives, and to more stoves on the market.

On the consumption side: the outputs include: increased access to information and knowledge, and household energy mainstreamed into the public sector. This results in more awareness and a higher probability of purchase.

Between the output and the use of output lies the system boundary of the project. Up to this point, the project is directly influencing the results; thereafter the success of the intervention is influenced by the targeted groups and their behaviour. But the project is still responsible for achieving results, and therefore the importance of participation and ownership of the target groups becomes very crucial.

The increased use of the output, in this case improved biomass stoves, depends on the future users, the project environment and the levels of promotion provided by producers and sales personnel. Support to manufacturers and supplies should be established, or strengthened by the project.

The outcome of the whole exercise is an increase in access to modern cooking energy technologies. This is the target of the EAP. It enables stove producers to sell their technologies, to generate income and to facilitate the increased involvement of women. Provided that the users cook effectively and efficiently with their new stoves, there will be reduced biomass consumption, a positive impact on time and money, and a reduction in GHG emissions, indoor air pollution, and accidents.

Further results beyond the outcome are not isolated and cannot be attributed to only one project. This is where the attribution gap between outcomes and impacts is located. Impacts are the positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Positive impacts are for example less deforestation, less diseases, improved working conditions, more jobs and the creation of small/medium enterprises (SMEs). A negative impact might be less time for social interaction of women during firewood collection. Where the impacts of scaling up improved biomass stoves finally contribute to the MDGs, they are described as highly aggregated impacts.

Note: This is not a cast-in-stone system. The allocation of the different result levels may vary according to the project approach. The boundaries are not fixed, but fluent.


RBM is intended to measure the progress and success of the project towards achieving its objectives. To measure this progress, indicators have to be developed. They are quantitative or qualitative values that describe the real situation, and indicate the degree of change. Ideally, these indicators will be measured at the beginning of the project (baseline), during the project, at the end of the project, and perhaps several years later.

The baseline describes the situation prior to the development intervention. It provides a basis against which to monitor whether the intervention has achieved the desired results. Through indicators, project managers are able to trace back the achievement of set targets, to identify unexpected changes and unintended impacts. It is possible to determine whether the outputs achieved have really proved useful, and whether their use has really lead to a worthwhile outcome in terms of development . If necessary, the project strategy can be adjusted, additional activities can be included, or further key stakeholders can be involved.


Further Information

Results chain Ethiopia

File:Results Chains for Ethiopia.pdf

(Click twice for readable version)

Results chain Kenya


See hyperlink to the left



Monitoring of Access to Cooking Energy

To describe the outcomes (depending on the objective of the intervention), all information about access to cooking energy services should be monitored. In this instance, people with access to modern and clean cooking energy would be monitored by sales or construction figures, as well as the correct usage by households, social institutions and SMEs.
The two key results are usually considered to be stoves sales and correct stoves use, since all further outcomes and impacts depend on stoves on the market and their correct use. The stove’s effects very much depend on the capacity of the user to achieve maximum reduction of fuel use and emissions.

Therefore, the following is monitored on a regular basis by GIZ interventions:

  • production and sales figures (incl. costs and prices)
  • purchase and use data



Further Information

The GIZ programme Energising Development has developed an elaborate monitoring tool for energy access programmes: Technische Unterstützung für wirkungsorientiertes Monitoring: Arbeitshilfe für den Einsatz von Microsoft Excel




Monitoring of Impacts - Methodology

Cooking energy projects that enable access to improved energy, contribute to the achievement of the MDGs. A plausible hypothesis on the project’s contribution to the MDGs should be provided, even though the impacts cannot be monitored, and thus cannot be attributed directly to the intervention. However, after a certain time of implementation it is worthwhile to do an impact assessment to determine whether the assumptions/ hypotheses about intended impacts, on which the project was based, have been achieved.
With these impact assessments, very specific results can be monitored and compared with the baseline. These include: income generation from stove production, women engaged in new activities, time and expenses saved, perceived reductions in the levels of indoor air pollution, and reduction in the number of accidents. Conclusions regarding the contribution to the achievement of higher aggregated impacts, such as increased business development, women’s empowerment, decreased deforestation, decreased respiratory and eye diseases, and finally the MDGs can be plausibly demonstrated with the results of the impact assessment.

To collect empirical data on the impacts of energy projects, a wide range of methods and approaches is available. These may differ significantly in terms of time, money and expertise needed for implementation.


Qualitative Methods

Qualitative methods emphasise personal experience and interpretation. They are well suited to reveal background information and detailed opinions regarding particular themes. The information gained through qualitative methods is not necessarily valid for the total target population. This type of interview is generally conducted for a small number of sampling units. It is advisable to make use of semi-structured questionnaires containing several open questions.


Focus Group Discussions

Focus Group Discussions are one type of qualitative methods, allowing interviewers to study people in a more natural setting than in a one-to-one interview. Focus Group Discussions are low in cost, one can get results relatively quickly, and they provide several opinions by talking to various people at once. Care must be taken during these discussions to ensure that those who make the most noise do not dominate the groups with their views.


Observations

Observation of the target groups, villages and project interventions through local field staff is a cheap and simple way of gaining a first impression of energy impacts. Even though not as accurate and representative as structured surveys, observations can be especially useful for preparing further scientific research (e.g. questionnaires) by providing site-specific knowledge.


Quantitative Methods

Quantitative approaches are concerned to quantify social phenomena by collecting and analysing numerical data in a large number. Sampling and questionnaires have to fulfil certain criteria for allowing statistically reliable and valid data to be gained.


Experimental / Quasi-experimental Designs

The main challenge in determining impacts results from the fact that it is impossible to observe how people undergoing an intervention would have acted if this intervention had not occurred. Additionally, surveying control or comparison groups that have not been part of the project intervention helps to avoid wrong conclusions.
The control group is hence included in order to simulate how the target group would have developed if the intervention had not taken place. In order to effectively fulfil this task, this control group should be composed of individuals that have not undergone any intervention, but live under similar general conditions and exhibit – on average – similar socio-economic characteristics (e.g. educational background).
Please note that households without ICS in the target region should not serve as a control group since the decision to buy a stove might have been influenced by variables such as education, income, motivation and other household characteristics.


Interview Types and Questionnaires

Standardized interviews (also known as structured interviews) are used in quantitative surveys to obtain comparable information representative for the total target group. Structured interviews are the most common form used in impact assessments. The wording of questions and response items as well as their sequence is identical for all respondents. They use close-ended questions.
In semi-standardized questionnaires, additional qualitative information can be gathered by the use of open questions.



Sample

It is usually not advisable to survey the whole population in the project region. This would not only require a large amount of time and efforts, but with larger populations it is just not manageable. Conducting interviews with all people present in the study area is only advisable for small, manageable numbers of sampling units. In most cases it is sufficient to draw smaller samples, since beyond a certain number, results are repetitive and the significance is not really affected by increasing sample size.

The sample is fundamental for quantitative surveys. To be able to gather information from the interviewees valid for the total target population, the size of the sample as well as its representativeness is key. It has to be statistically significant. As a rule of thumb for household surveys, it is advisable to draw a sample of at least 40 to 50 households for each stratum (see below: stratified sample). In the case of very small numbers of sampling units, e.g. a very limited number of hospitals in the project area, it may be advisable to opt for qualitative approaches.
There are different ways of sampling.

Depending on the survey context, one of the following sampling methods may be chosen:


Random Sample / Systematic Sample

In a random sample, every individual unit in the population has an equal chance of being selected for the sample; selection occurs by chance, so knowledge of the total population group is needed to ensure that everyone is represented. A totally random selection of households is difficult in practice, and the systematic sample is a common technique to draw near-random samples. For a systematic sample, every xth unit is selected (e.g. interviews are conducted in every third household on the interviewers’ way, beginning from a randomly selected starting point).
Random / systematic samples provide information about distributions within the whole population. For example, to gain information about the percentage of efficient stoves existing in a village, a random sample should be drawn.


Stratified Sample

Stratified samples are used to ensure that appropriate numbers of small subgroups are included in the sample. This could apply to surveys focusing on technologies that have not yet been widely disseminated, such as Solar Cookers. In this case ownership/non-ownership of a solar cooker might be a typical variable. Others might include socio-economic groups, districts, professional groups, etc. The percentage drawn from each stratum might be identical or variable, according to the needs of the survey. In the case of different percentages drawn for each stratum, data has to be weighted for analysis to represent the population correctly. For example, in a sample stratified by two household income groups, with 10% of all high-income households and 30% of all low-income households being interviewed, the data for high-income households has to be weighted by a factor of three. To develop a stratified sample, and to structure the sample accurately requires a lot of reliable information about the total population group. This is a challenge in many cases, since available information is often limited.


Interview Partner

After drawing the sample on level of households, SMEs or social institutions, the sampling method for selecting the right interview partner should also be thought about . A pure random sample would e.g. be drawn by asking for the person in the household whose birthday was the most recent. In most cases it is the cook who is interviewed in cooking energy surveys. However interview partner can also be the head of the household or the manager of a SME or a social institution. Please consider that those persons may not be able to answer questions regarding the daily use of the stove.



Review of Documents and Statistics

Besides information that needs to be gathered from interviews or observations, data on particular indicators might be readily available from local databases, statistics and registers.


Procedure for Developing a Questionnaire and Interview Guidelines

  • Analyse the project strategy
  • Consider the projects results chains – do they cover all the relevant results hypotheses to be evaluated in the impact assessment? Identify relevant assessment fields for the impact assessment along results chains
  • Formulate results indicators and convert these into questions and answers, making sure both questions and categories from which an answer is to be chosen, are easy to understand.
  • To achieve the best response rates, questions should rank from the least sensitive to the most sensitive one. Questions regarding socio-economic data are thus placed at the end of the questionnaire.
  • Avoid suggestive questions- they force the respondents into a corner and reducing their freedom to answer the questions. Here is an example: "Many scientists believe that stove emissions can negatively impact your health. Do you think this view is correct?"
  • It may be necessary to translate the questionnaire into a local language.
  • Pilot test your questionnaire before rolling it out to all the involved households.


Please consider:
There can be a lot of undesired effects on the respondents and his/her answers, both from the questionnaire and the interviewer. These effects can lead to distorted data.

  • Sequence of items: Typical effects from questionnaires can arouse from the sequence of answer items, e.g. primacy-recency effects. Respondents remember first or last answer items best and forget about the items in the middle of a long list of answer possibilities.
  • Social-desirability: people tend to answer not according to their own opinion but according to social norms or what the respondent thinks would be the desired answer for the interviewer. In development projects where the interviewer is associated with money or other advantages this might be a serious problem.


Further Information on Methodology




Example Kenya

The Kenyan impact assessment was conducted together with national and regional counterpart staff with the intention to create capacity and increase ownership. Involvement of project staff, counterparts, local NGOs and students in various parts of the assessment (like preparation and data interpretation) was helpful for the exchange among all stakeholders. In Kenya, agricultural officers reported the enthusiasm of the targeted beneficiaries. One student who conducted the interviews in Kenya started his own stove business. Such examples show the secondary effect of these monitoring instruments, which might not have been anticipated.

In addition, a national consultant was involved in the whole process of impact assessment, from advising on the crucial indicators to questionnaire development, to taking part in the data entry, analysis, and interpretation. Enumerators (local students) were trained to conduct the interviews.

The target group representatives in districts and villages were selected using a set of selection criteria. Qualitative interviews using interview guidelines were conducted with selected local authorities. Selected women’s groups engaged in focus group discussions at village level using a set of tools developed for Participatory Rural Appraisal (PRA tools). These included elements such as trend analyses, activity lists, and influence matrices. Trained PRA experts were included in this exercise.

Finally structured interviews were conducted to produce a quantitative sample survey, administering questionnaires with the selected users (households, schools and restaurants) and producers. These interviews were carried out by carefully chosen and trained enumerators, in most cases local students.


Interview guidelines
PRA tools
File:Local Chiefs Interview Guidelines.pdf
Gtz Pra-tools women group Kenya.pdf

Housholds Schools Restaurants

File:Questionnaires for Households with and without Improved Cookstoves in Kenya.pdf

File:Questionnaire for Social Institutions in Kenya.pdf

File:Questionnaire for Productive Use of Improved Cookstoves in Restaurants in Kenya.pdf

The final report "Result assessment" in Kenya was published in 2009.


Model List of Impacts, Observation Fields and Indicators for IA Kenya, GTZ HERA 2007

Steps and Procedures - documented according to IA in Kenya. GTZ HERA 2007

These describe the procedure of IA in Kenya as well as recommendations for training for enumerators and lessons learnt during the IA.


Selection Criteria for IA in Kenya, GTZ HERA 2007

A set of guiding questions for data analysis and interpretation was developed to support for the consultant in Kenya.






Further Information

GTZ EnDev Guide Impact Assessment for Energising Development Projects, GTZ 2007


GIZ EnDev has developed Impact Monitoring Guideline - available on Energypedia

Impact_Monitoring_Guidelines


Webinar on Mobile Monitoring for Data Tracking and Management, available on http://www.pciaonline.org/webinars

GTZ EnDev Report on Impacts

A List of impact indicators was developed by GIZ HERA and is available under Energypedia


Mirt impact assessment rpt final.pdf

Impact assessment clay stoves malawi 2008.pdf

Impact Assessment of Chitetezo Mbaula, GTZ ProBEC Malawi, 2008

Impact Assessment of Mirt stoves, GTZ SUN-E, Ethiopia 2008



Participatory Impact Assessment – An experiment?

In selected SADC countries, where the Programme for Biomass Energy Conservation, ProBEC, is implemented, impact assessment interviews were conducted by local stove artisans. In 2004, the stove promoters and producers from Malawi, Mozambique, Zimbabwe, Zambia, Tanzania and Kenya were invited to Malawi for a ProBEC Workshop about experience exchange on low-cost clay and ceramics.

Very interesting feedback was given by producers and promoters during the workshop concerning the results of impact assessment. For many of them, it was the first time they had exchanged ideas with their customers, and thus they learnt firsthand about how the stoves were used, the way people had understood stove instructions, difficulties, problems and demands.

The stove producers, builders, and promoters considered this to be so important that they passed a resolution asking for training in monitoring and impact assessment as part of their regular training programme. They realized that by asking the questions themselves, it would increase their awareness of the quality of their stoves, improve their marketing skills and thus their access to customers.

This approach to impact assessment requires high involvement by the producers, and creates business awareness. It should be considered as complementary or additional to an external, unbiased impact assessment by an external person. The results of an assessment by the manufacturers should not be considered neutral.


Further Information

  • Brinkmann & Klingshirn: Participatory Impact Assessment, 2005:

Questionnaire assessment vb 052005.pdf


Cost-Benefit-Analysis (CBA)

The data from impact assessment can provide the starting point for an economic evaluation of the project, including a Cost-Benefit-Analysis (CBA) and a Cost-Effectiveness Analysis (CEA). Analysing both the economic efficiency of the investments, and the benefits deriving from energy efficient stoves on a macro and micro level can be helpful for further lobbying, public relations, and keeping control of the project.

A Cost Benefit Analysis of the Energy Advisory Project in Uganda demonstrated the economic value of the use of Lorena Rocket Stoves for individual households as well as for the public sector. The same was shown in a Cost Benefit Analysis in ProBEC Malawi for social institutions such as schools, hospitals and prisons.


GTZ Cost-Benefit Analysis – Tool

This tool is provided as a set of prepared excel sheets, where all data and information required for the conduction of an economic CBA needs to be entered into reserved cells. In addition GTZ HERA provides explanation on how to use the CBA tool.


Further Information

An evaluation of the costs and benefits of household energy and health interventions at global and regional levels as well as guidelines for conducting such cost-benefit analyses can be found at the WHO website

WHO used the Cost-effectiveness analysis (CEA) as a tool for assessing the cost-effectiveness ratio of different interventions regarding indoor air pollution. This was applied to identify the intervention that provides the highest ‘value for money’. CEA results can thus help policy makers to choose the interventions and programmes that maximize health benefits for the available resources.

Three scenarios have been assessed: providing the population with access to cleaner fuels (kerosene, liquid petroleum gas); providing the population with access to improved stoves; providing part of the population with access to cleaner fuels and part of the population with improved stoves.



References

This article was originally published by GIZ HERA. It is basically based on experiences, lessons learned and information gathered by GIZ cook stove projects. You can find more information about the authors and experts of the original “Cooking Energy Compendium” in the Imprint.



Top of the page

--> Back to Overview GIZ HERA Cooking Energy Compendium