Impact evaluation: a luxury reserved for large projects?
A project can have met its operational objectives, kept to its schedule and delivered the planned outputs without being able to answer one essential question: what changes did the intervention actually contribute to producing? This question sits at the heart of impact evaluation. Yet impact evaluation remains associated with large programmes that have substantial budgets, long implementation periods and specialist teams.
In that picture, small and medium projects should limit themselves to tracking activities, outputs and a few outcome indicators, while analysis of effects is reserved for large-scale interventions.
The distinction is misleading. The issue is not whether a project is big enough to “deserve” an impact evaluation. It is to determine what level of evidence is needed, given the question asked, the nature of the intervention, the decisions to be taken and the resources available.
The question is therefore not how to reproduce, on a small scale, the evaluation systems of large programmes. It is how to produce a level of evidence credible enough to inform a decision.
The main barriers: cost, skills and time
Three obstacles mainly explain the reluctance of small and medium organizations.
Tracking activities is not enough to demonstrate effects
A monitoring system can work perfectly well and still be insufficient to answer questions about the effects of an intervention. It can tell you how many training sessions were held, how many people were reached, how many services were delivered, what achievement rate was reached or how many facilities were built.
That information matters for steering implementation. But it is not enough to appreciate the changes produced. A training session delivered does not on its own demonstrate improved practice. A service provided does not automatically demonstrate a durable improvement in beneficiaries’ situation. A facility built does not, by itself, allow any conclusion about whether it is actually used.
Monitoring mainly tells you what was done and what was observed. Evaluation seeks to go further: what do the observed changes mean, how can they be explained and, when the question warrants it, to what extent can they be associated with the intervention?
This distinction between activities, outputs, outcomes and effects is fundamental. Without it, a project can show excellent achievement rates while holding very little evidence on whether its intervention actually works.
Key point
Delivery is not effect
Carrying out an activity is information about implementation. It does not constitute, on its own, proof of the effect produced by the intervention.
Skipping evaluation creates a knowledge debt
Giving up on analyzing the effects of an intervention does not simply mean producing a less complete report. It can create a genuine knowledge debt.
This difficulty becomes particularly acute when a decision about continuation, adaptation or scale-up has to be taken.
Evaluation should therefore be seen not only as an accountability exercise, but also as an investment in the knowledge needed for steering and decision-making.
An impact evaluation is not necessarily a heavy experimental design
Systematically associating impact evaluation with randomization, control groups, large-scale surveys or econometric analysis makes the exercise intimidating. Those methods can indeed be relevant when the question concerns a causal relationship and the necessary methodological conditions are met. But they are not the only way to study the effects of an intervention.
Mixed methods can be particularly useful when quantitative data measure how a phenomenon evolves while qualitative data explore the mechanisms, perceptions and contextual factors that may explain it. Contribution analysis can also be relevant when the intervention sits in a complex environment. The choice is then built on the same logic as for any M&E method: from the question asked, not from the technique available.
The aim is therefore not to build the most complex methodological design possible. It is to produce the most credible and useful level of evidence given the context.
What should be proportionate is not whether the evaluation happens, but how complex the design needs to be.
Proxy indicators: useful, but not proof of impact
Methodological rigor therefore also means being able to say what the data allow you to assert, and what they do not.
A proportionate evaluation is prepared at design stage
Evaluation should not be thought of only as an activity carried out a few months before project closure. A proportionate approach can be prepared from the design of the intervention. Five questions help structure that thinking.
1. What change is expected?
The theory of change specifies the mechanisms assumed to lead to results and identifies the main assumptions the intervention rests on.
2. How will that change be observed?
The point is to identify genuinely useful indicators and evidence, rather than accumulating indicators that are hard to exploit.
3. Against what situation will change be appreciated?
Where possible, a baseline situation or earlier data help interpret the changes observed.
4. What other influences may come into play?
A change observed among beneficiaries may also be influenced by economic, institutional, social or environmental factors outside the intervention.
5. Which decision should the evaluation inform?
Adjusting an intervention, deciding on its continuation, documenting results, guiding a scale-up or drawing lessons. The level of evidence needed depends partly on how important the decision is.
The real issue: moving from compliance to decision
The question of impact evaluation is ultimately less technical than it looks.
When evaluation is treated solely as a donor requirement, it can become one more administrative activity: produce a report, populate indicators and document the results obtained. When it is integrated into steering, its function changes. It can establish whether an intervention should be continued, adapted, targeted differently or extended. It can also identify assumptions that do not hold, explain gaps between territories or beneficiary groups, and turn findings into learning.
It is this usefulness that should guide the level of methodological ambition.
Avoid knowledge debt from the design stage
Delta Monitoring lets teams define indicators and data sources when the project is designed, then feed them throughout implementation. At decision time, teams hold the data they would otherwise have had to reconstruct after the fact, whatever the evaluation budget.
Is impact evaluation really reserved for large projects? No. But that does not mean every project should put the same design in place.
The maturity of an M&E system is not measured by the number of indicators collected, the volume of reports produced or the sophistication of the methods used. It is measured above all by its capacity to provide, at the moment a decision has to be taken, evidence credible enough to understand the changes observed and act accordingly.
The question is ultimately not “is the project big enough to be evaluated?” but rather “what level of evidence is needed to take the next decision with more confidence?” It is from that question that an evaluation approach can be built that is rigorous, proportionate and useful for steering the project.