The idea of rational research carried out by objective, impartial, and neutral actors has traditionally permeated the consciousness of the international development community. Monitoring and evaluation (M&E) activities are necessary for international organizations to make informed and responsible decisions, encourage the improvement of the results of implemented projects, and promote reflexivity and learning. The work of the evaluator, however, is always interwoven within political contexts, where the high-stake interests of a variety of actors are mixed. The political nature of evaluation has been a topic of discussion since the 1970. Some saw it as a threat, while others treated it as an inevitable reality that should be accepted in order to increase the use of M&E activities.
At the same time, although M&E activities are taking an increasing place in the schedules and budgets of international organizations, “the utilization of evaluation findings is still disappointingly low”. Still too focused on short-term results (i.e. resource management, cost-effective rates, short-term impacts), contemporary discussions in the international development arena now insist on the expansion of M&E activities, allowing more judgment to be made on the sustainability of the results and political effects of projects. The literature describes growing pressure to make evaluation a central tool in the development of local actors, prompting international agencies to respect their political leadership and strengthen their capacity to exercise it. Traditional forms of evaluation typically designed by donors and development agencies are now being challenged to call for recognition of the political issues that overlap with a practice that was initially conceived to be neutral and impartial.
There is then a change taking place, a shift from a paradigm of the practice of technical, rational and scientific evaluation, to a recognition that politics count in the objectives of social change carried by development programs, with an understand that these must be carried out politically and locally. But how can the modern evaluator navigate between injunctions of impartial, neutral and rational exercise, and calls to navigate politics as fundamental factors in the success of development projects and the effectiveness of the evaluation results?
Informal influences and decision-making politics
Several sets of standards have been developed by international development actors in order to promote rigorous and independent practice of M&E activities. Most states have developed their own guides for various standards and assessment standards. At the international level, the United Nations Evaluation Group (UNEG) adopted in 2005 the UNEG’s Norms and Standards for Evaluation. The majority of the standards are commonly accepted, among which are:
- 1. Involving stakeholders;
- 2. Maintaining independence and disclosing conflicts of interest;
- 3. Using sufficient and credible evidence to support findings, conclusions, and recommendations;
- 4. Using a quality assurance process;
- 5. Keeping records to demonstrate that proper evaluation methods were used.
Despite the existence of such standards, evaluation research has since the early 1970s underlined the penetration of political, informal interests into evaluation processes. These interests are exercised by a large number of stakeholders, at national and international levels, at different stages of the project cycle: member states of international organizations, commissioning entity of the evaluation, donors, policy makers, but also program competitors, evaluation sponsors, beneficiaries or even the evaluators and the research community themselves. The competition for resources (material, human, normative) essential to political practice makes it a ground mine for unequal power relations between the actors. Moreover, the actors involved in evaluation are often also involved in decision-making processes, and their interests in evaluation are then intertwined with those of political practice.
Several reasons can then lead stakeholders to try to influence the evaluation process. They may seek to exert pressure to make the results appear more positive, or less negative, to justify the existence of a policy or to rally support, or to use the results to obtain funding.
Eckhard and V. Jankauskas (2019) identified different policy resources used by stakeholders to influence the evaluation process at different stages:
- 1. What they call the ‘agenda-setting power’ refers to a direct participation in the phase of evaluation initiation. As the evaluator isn’t free to decide upon the evaluation work plan, ‘the very ability to decide what is or what is not to be evaluated gives stakeholders the power of shaping the output of the evaluation’.
- 2. The ‘leverage over evaluation resources’ can directly influence the results of evaluation. Eckhard and Jankauskas cite a simulation study led by Azzam (2010) which ‘revealed that the more political power or influence stakeholders groups held over evaluation logistical factors, the more evaluators were willing to modify their design choices to accommodate perceived stakeholder concerns’.
- 3. Finally, the access to evaluation results before they get officially confirmed or published gives stakeholders the power to use ‘lobby-like strategies’ to translate their own interests into the final output during the drafting and finalising stage. Azzam (2010) in the simulation study cited above proved that a meeting between the stakeholders and the evaluators during the evaluation design stage led evaluators to modify the procedures afterwards. It gives opportunities for stakeholders to impact the interpretation of already gathered data or to shape the proposed recommendations.
In order to anticipate and respond to these pressures, Fitzpatrick (2011) encouraged evaluators to learn “about the political context, which includes the positions of various interest groups in regards to the program. More broadly, the evaluator [must] take time to learn the identity of the various groups who are interested in the program, who have some power or control over it, or who may be opposed to the program for whatever reason.” Some of his main recommendations to prevent informal influences during the evaluation process are to expand the design phase, to lean heavily on negotiation, to develop a dissemination strategy, or to think about the politics of the data collection, among others.
‘Thinking and Working Politically’
While the influence of policies is generally seen as a disruptive element in international development activities, some academic circles speak to the centrality of political processes and the need to guide them to maximize the sustainability and success of programs. The Developmental Leadership Program (DLP), led by Australian Aid, focuses on the role of aid development actors to support local political leadership which in turn promotes ‘developmental outcomes, such as sustainable growth, political stability and inclusive social development’. According to Roche, Hudson, Marquette and McLoughlin, DLP researchers, ‘Politics is not the obstacle, it is the way change happens’.
Development programs often have to deal with the spectrum of legitimacy, contestation and acceptation among the local actors responsible for the sustainability of the project outcomes. Understanding this is accepting the political intertwinements that can influence the outcome and the sustainability of a project. The evaluator must therefore find ways to measure this political activity. They can take this role by considering the criteria to reveal for these programs to grow alongside in which they are embedded. They may be less usual than donors’ criteria.
In order to explore these new needs and formulate recommendations, the Thinking and Working Politically Community of Practice was created by practitioners wishing to engage in discussion on the political practice of evaluation, underlining that ’political factors are usually much more important in determining developmental impact than the scale of aid funding or the technical quality of programming’. They point out that traditional results-based approaches are inadequate for the M&E activities of a program that seeks to support complex, unpredictable and non-linear changes. To this end, the authors advocate the systematic use of a mixed-methods approach to enable the measurement of these long-term impacts and to provide a better understanding of long-term policy processes that support sustainable and inclusive change. More specifically, they evoke a number of other methods and approaches, including:
- 1. Strategy testing
- 2. Action-research
- 3. Crowdsourcing and big data analysis of social media
- 4. Social network analysis
- 5. Qualitative comparative analysis
- 6. Micro-story collection and aggregation
- 7. Outcome mapping and harvesting
- 8. Process tracing
- 9. Real-time reporting
To understand the power relations and the role of politics at work in development projects between actors, the evaluator has the responsibility to succeed in communicating effectively the results to guide the decisions of the stakeholders and promote learning and innovation at the design level. Transmitting these observations tinged with political interests likely to clash with the normative contexts of international organizations also implies being aware of the informal influences at work there and knowing how to respond to them.
To meet these needs, UNEG’s Norms and Standards for Evaluation revised its guide in 2016. It now includes a section on professionalism, emphasizing on the intended use of evaluation as well as its role in the empowerment of local actors, in a manner “as impartial as possible”. This comprehensive document can be the first stone of a self-reflective global construction of evaluation standards, which could ultimately allow the politics of M&E to be considered at both decision-making and executive levels. From there to knowing if this very process can be impermeable to political influences, only time will allow to judge it and, why not, an external evaluation.
About the author
Lou is a Junior Officer in the Media and Communication Department at Trust. She has graduated from a Master degree in International Relations and Diplomacy in Lyon, France. She dedicates her free time to voluntering in organizations advocating for the defense of refugee and civil rights in France.
Read more about Lou on LinkedIn.
