Control Groups

Evaluation designs can be broadly classified into three categories: experimental, quasi-experimental and non-experimental. The term control group is used when the evaluation employs an experimental design and the term comparison group is associated with a quasi-experimental design. In non-experimental design, program participants are compared to non-participants by controlling statistically for differences between participants and non-participants. These three evaluation designs vary in feasibility, cost, the degree of clarity and validity of results, and the degree of selection bias (see World Bank 2008).

The main challenge in determining impacts results from the fact that it is impossible to observe, how people undergoing an intervention would have acted if this intervention had not occurred. Only a well designed form of this so-called counterfactual situation allows for robust statements about the impact that is actually attributable to the intervention. The traditional way is to consider the situation before the intervention as this counterfactual. Therefore, households in the project area are to be investigated before and after the intervention. It is, however, inaccurate in most cases as soon as external factors affecting the mostly [quantitative] indicators change over time. Not least, qualitative approaches deliver indispensable insights into the various stages of the result chain and the transmission channels from outputs to impacts.

"Box 1": Options for Choice of Control Region

The obvious counterfactual situation for the stove and electrification projects would be Option A – a region where these interventions have not been taken place before. However, Option B provides as well for a viable counterfactual. While Option A investigates in how far the project sites outperform the control regions, Option B explores in how far the project sites catch up in relation to the control regions. The crucial point is that between the before and the after survey, no such intervention takes places. At this point, one disadvantage of Option A becomes evident: It is often not for sure that during that time period the control region actually remains non-electrified or without improved stoves inter­ventions respectively. It has already happened that unforeseen activities by other donors or partner country institutions in the course of the project period rendered the implementation of this approach impossible. On the other hand, Option A has the advantage that it can serve as an “ex-ante impact assessment”. This implies that the information gathered in the control regions helps to anticipate the beneficiary’s behaviour, potential difficulties and opportunities to be expected in the wake of the EnDev project before project implementation. These findings can then potentially be fed back into the project design.

Sources:

World Bank –Impact Evaluation (2008)

EnDev2 Evaluation & Monitoring Modules.