Evaluation of Humanitarian Action
Impact, alongside efficiency, effectiveness, relevance, and sustainability, is one of the criteria used to evaluate humanitarian interventions. In order to understand how impact is used to measure interventions, it is important to first be aware of how impact measurement fits within the evaluation of humanitarian interventions. This process best starts with an understanding of what the evaluation of humanitarian aid entails.
The OECD/DAC defines evaluation as “an examination, as systematic and objective as possible, of an on-going or completed project or programme, its design, implementation and results, with the aim of determining its efficiency, effectiveness, impact, sustainability and the relevance of its objectives”. This definition has become the standard for evaluating development and humanitarian programmes. The ALNAP defines evaluation of humanitarian aid as “the systematic and impartial examination of humanitarian action intended to draw lessons to improve policy and practice and enhance accountability”.
Evaluations offer a chance to reflect on the events leading to a humanitarian crisis and the responses to that crisis. As a tool, evaluation allows us to determine whether a response to the crisis can be improved by looking back at what has been done, and evaluating what worked, and what didn’t work during the intervention.
The evaluation of humanitarian interventions is usually initiated by, or is done in cooperation with, a humanitarian organization whose performance is being evaluated. It is usually carried out by a team of non-employees (external) or by a mixed team of non-employees (external) and employees (internal) from the commissioning organization, or the organisation being evaluated. The evaluation of humanitarian interventions assesses the project or program against recognized criteria (eg, the DAC criteria). Evaluations reports articulate findings, draws conclusions and make recommendations. To get more insight on Evaluation of Humanitarian Interventions using the DAC criteria, the ALNAP handbook is a great source, and can be found here.
A Closer Look at Evaluation Criteria: Impact
Both ALNAP and OECD/DAC evaluation criteria list “impact” as an evaluation criteria among effectiveness, efficiency, relevance and sustainability. Impact is defined as “the extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects”. We can thus understand impact as a measurement of what difference an intervention makes. Impact as a criteria for evaluation seeks to address the possibly transformative effects of a humanitarian response. It identifies the broader long term social, environmental, and economic effects of the intervention. It does so by probing the holistic changes in systems or norms, and potential effects on people’s well-being, human rights, gender equality, and the environment that an intervention as made. It is important to note that evaluation of an intervention’s “impact” doesn’t just look at one theme. Evaluation impact should ideally assess the impact of several of the interventions cross-cutting themes such as support to livelihoods, human rights, and gender equality. These issues often involve long-term and wider socioeconomic effects hence impact is well suited to assess these themes.
Why do we measure impact?
Organizations carry out evaluations in part to reflect on a response to a crisis or emergency to be able to draw lessons from the past to better inform future interventions. In a similar way, we measure to better inform learning and accountability from an impact to see what was changed, and to what extent.
When do we measure impact?
Impact can be assessed at several points along the intervention. For example, an analysis of the likely impact of an intervention can be conducted before the start of a project, in order to anticipate the wider consequences of an intervention. Impact can also be assessed as a part of the ongoing analysis process throughout a project. In this way, impact is embedded as part of management systems, in an attempt to adapt interventions or monitor performance. We can also conducted an analysis of the impact of interventions after the fact, as part of evaluations or research.
How is impact measured?
There are methodological approaches which can be implemented in drawing up an impact evaluation. Hallam (1998) outlines them as:
- the scientific approach, which is favoured by those wishing to generate quantitative measures of impact;
- the deductive/inductive approach, which is more anthropological and socio-economic; this approach relies on interviews with key informants, and draws on other similar or comparable cases; and
- participatory approaches, which depend on obtaining the views of those benefiting from a programme.
These methodological approaches use tools such as surveys, interviews, workshops and discussions, direct observation, participatory research, and case studies in order to better quantify impact. Most M&E practitioners argue that a mix of methods and/or approaches is desirable to meet the broad objectives of impact assessment.
Despite the number of ways in which we can measure impact, there are a number of methodological constraints which make impact measurement challenging. The nature of the humanitarian operating environment, the need to act quickly in situations of immediate crisis, an organisational culture that values action over analysis and the fact that there is little consensus around the core objectives of humanitarian aid all make impact analysis difficult.
Case Study: Evaluation Impact Assessment in Action
One example of an evaluation of a humanitarian project focusing on assessing impact is an impact assessment of Mercy Corps’ ongoing Emergency Cash-Based project in Aleppo and Idleb governorates, in North West Syria being conducted by Trust Consultancy and Development. The aim of the assessment was to determine whether the project has strengthened individuals’ and households’ capacities to cope more rapidly with shocks and the effects of conflict; reduced the vulnerability of individuals and households; and supported in the positive improvement of coping mechanisms, more information on the project is available on Trust’s website.
As noted, measuring the impact of humanitarian interventions is an exercise that falls within the bigger picture of evaluation of humanitarian aid. The main objective of measuring impact is to see if the intervention made any positive or negative effects. Humanitarian organisations carry out impact evaluations as a powerful way of drawing lessons from the response and also for purposes of accountability.
An introductory online course on evaluation of humanitarian interventions offered by ALNAP may be accessed here.
References & Ressources
- Hallam, A. (1998) Evaluating Humanitarian Assistance Programmes in Complex Emergencies. Good Practice Review 7. London: Overseas Development Institute. Available at https://www.alnap.org/help-library/evaluating-humanitarian-assistance-programmes-in-complex-emergencies-good-practice
- Evaluating Humanitarian Action using the OECD-DAC Criteria https://www.alnap.org/help-library/evaluating-humanitarian-action-using-the-oecd-dac-criteria
- What methods may be used in impact evaluations of humanitarian assistance? 3ie Working Paper https://www.3ieimpact.org/evidence-hub/publications/working-papers/what-methods-may-be-used-impact-evaluations-humanitarian
- Applying Evaluation Criteria Thoughtfully
https://www.oecd-ilibrary.org/development/applying-evaluation-criteria-thoughtfully_dedc34d7-en - Measuring the Impact of Humanitarian Aid – A Review of Current Practice
https://www.alnap.org/help-library/measuring-the-impact-of-humanitarian-aid-a-review-of-current-practice - EHA e-learning course | Unit 1: Introducing evaluation of humanitarian action
https://www.alnap.org/help-library/eha-e-learning-course-unit-1-introducing-evaluation-of-humanitarian-action
About the Author
Vincent Ndlovu graduated with a Bachelor of Laws (LLB) for his undergraduate studies and is currently pursuing his MA International Relations: International Humanitarian Action with the NOHA Network on Humanitarian Action at the University of Groningen in the Netherlands. He has a keen interest in the Protection Sector and Cash and Voucher Assistance (CVA) or Cash–Based Assistance (CBA).
Learn more about Vincent on LinkedIn.
