Impact Monitoring Guidelines

Overview

The Impact Monitoring Group of Energising Development has developed recommendations for sound impact assessments.

We recommend to:

• use the recommended list of indicators
• conduct baseline studies before project intervention and impact studies after households, social infrastructure or SMEs received access to modern energy
• use a theory based approach by applying the result chain for developing the study design
• use a mixed methods approach: the logic of the comparative advantages of methods: i.e. the mix of quantitative (standardised household interviews) and qualitative methods (open / semi-structured interviews with key persons and households or focus group discussions)
• include, wherever possible, control groups (households which resemble those households with access to modern energy with the exception of this access) into your baseline and impact study
• consider research ethics

The following tables show in more detail typical steps for baseline and impact studies, which we furthermore recommend and support.

Introduction

Development interventions are executed with the ultimate aim of changing the living situation in less developed countries. What is often thoroughly investigated are the inputs to a development intervention and its immediate outputs - how much money is spent, how many children are vaccinated, business trainings conducted or latrines installed. Assessing the impacts of a development intervention though - the aforementioned amelioration of living standards that can be attributed to a particular intervention - is part of a movement towards evidence-based development policy making and accountability. The central question to all impact evaluations "What is the causal effect of a programme on the variable of interest?" can be applied to various policy contexts.

Initial but crucial steps to every evaluation is the establishment of

1. questions to be answered within the scope of the given evaluation;
2. the theory of change, depicting how, in theory, the intervention is envisaged to achieve results;
3. the result chain;
4. hypothesis to be tested by the evaluation;
5. performance indicators.

Taken together, the evaluators will hereby arrive at the specific questions to be phrased by the evaluation. Ideally, this will occur in the planning phase of the development intervention, thereby linking implementation and evaluation closely.

Preparation

Tasks	Materials/ Support	Library
Draft Terms of Reference for Consultants / Enumerators	Sample Terms for national / international consultants Sample Terms for enumerators	ToR Consultants: ToR Baseline Study (Rwanda PSP Hydro) ToR Survey french version
Selection of Consultant / Enumerators	Consultants
Study documents and statistics on national development goals and policies, national sector goals and policies, BMZ goals and concept for the partner country	Checklist on relevant development goals	UNDP Energy Services for the Millennium Development Goals BMZ Sustainable Energy for Development
Developing of Results Chains/ Indicators (consider undesired side effects being positive or negative)	Sample Results Chain Impact Criteria and indicators	Cooking: Energy Impacts, Observation Fields and Indicators IA Kenya: Result Chain IA Ethiopia: Result Chain IA Kenya: Model List of Impacts, Observation Fields and Indicators
Developing of Questionnaires, interview guidelines, PRA tools (MAPP), observation criteria and other tools (catalogue of methods)	Questionnaires for HH (stoves), stove producers, social institutions and restaurants using ICS PRA tools for focus group discussions (MAPP) Interview guidelines for village authorities Observation criteria for stoves in use (example lists of good cooking practices and of stove conditions)	Example Questionnaires (Impact Studies Material)
Sampling/ Selection target Group	Guidelines / Sampling & Selection Criterias Poverty criteria for selection of HH	Kenya: Selection Criteria for HH, SI, PU and Producers IA Kenya: Local Poverty criteria (example from local village authorities)
Training of Supervisors and Enumerators/ Pretesting	Training Material Training Agenda	Supervisors and Enumerators Example Training Agenda (french version) Kenya: ICS Training of Enumerators

Implementation

Task	Materials/ Support	Library
Logistics	Checklist on how to implement HH Interviews H1: Short Household Survey H2: Profound Household Survey H3: Qualitative Household Appraisal
Conducting Interviews		Manual Conducting Interviews (english version) Manual Conducting Interviews (french version)
Data entry

Data Analysis/ Reporting

Task	Materials/ Support	Library
Data analysis	Guiding Questions for Data Analysis	Kenya Stove HH: Guiding Questions for data analysis Kenya Stove Producer: Guiding Question for data analysis
Data interpretation – in exchange with the project team (in a workshop)	Checklist for data interpretation
Report writing (+ summary writing)	Example for report summary Example for ICS description (Annex)	Kenya Stove HH: Example for report summary
Handover of raw data