Putting the pieces of the evaluation prep puzzle together: the ToR and beyond (part 1 of 2)

A well-written ToR, yes but…!

When preparing for the evaluation of a program or project, evaluation clients (read: program teams) tend to focus exclusively on the content to include in the related terms of reference (ToR). The thinking behind this? A well-written ToR – i.e., one that does a good job of “[spelling] out…[the] requirements and expectations related to an evaluation, review, or similar study” (World Bank, 2011) – will identify the most qualified evaluator and will necessarily result in a solid evaluation.

But is this really so?

You guessed it: no.

True, a ToR goes a long way towards assuring the quality of an evaluation. But there’s much more to evaluation prep than the content of the ToR itself.

In this article, the first of a 2-part series, I provide a recap of what we mean by ‘evaluation’ and ‘ToR’ and the purpose they serve. Then I outline the key elements that should be included in an evaluation ToR. In the next article in the series, we’ll dive into some of the non-ToR-specific elements of evaluation prep that can make or break an evaluation.

Quick recap of the ‘what’ and the ‘why’ of evaluations (skip this section if evaluation basics are old news to you!)

What is ‘evaluation’ (or ‘evaluative judgment-making’)?

Not everyone defines ‘evaluation’ the same way (and, personally, I tend to prefer the broader term ‘evaluative judgment-making’). However, for the purpose of this article, let’s use the excellent definition and related explanations offered by BetterEvaluation:

“[an evaluation is] any systematic process to judge merit, worth or significance by combining evidence and values” (BetterEvaluation, 2023)

The above definition allows for the inclusion of multiple types of evaluation under the same umbrella, including – but not limited to – problem diagnosis, root cause analysis, evidence mapping, ex ante economic evaluation, formative evaluation, process evaluation, real-time assessment, performance review or evaluation, impact evaluation, contribution tracing, cost-benefit analysis, value-for-money assessment…whether performed internally by the program team or commissioned to external evaluators.

[RESOURCE: For a more detailed definition of ‘evaluation’ along with an illustrative example of its application, see our Basics of M&E: A cheat sheet for beginners.]

Why do evaluation (or why make evaluative judgments)?

The importance of evaluations lies in their ability to “help determine what works well and what could be improved in a program or initiative” (Rural Health Information Hub, 2023). Program evaluations can, for example, be used to:

• [d]emonstrate impact to funders;
• [s]uggest improvements for continued efforts;
• [s]eek support for continuing the program;
• [g]ather information on the approach that can be shared with others; [and]
• [h]elp determine if an approach would be appropriate to replicate in other locations with similar needs (ibid).

[RESOURCE: A list of questions that evaluation helps answer is provided in the Basics of M&E: A cheat sheet for beginners guide.]

Overview of the ‘what’ and the ‘why’ of ToRs

What is a ToR?

The term ‘ToR’ refers to the document that lays out the scope of a specific task or assignment to be performed and any relevant background, contextual, or other details. In its plural form – i.e., ‘ToRs’ –, it refers to more than one such document.

When it comes to evaluations specifically, ToRs “[spell] out…[the] requirements and expectations related to an evaluation…” (World Bank, 2011). As such, evaluation ToRs speak to:

1.  [w]hy and for whom the evaluation is being done;
2.  [w]hat it intends to accomplish;
3.  [h]ow it will be accomplished;
4.  [w]ho will be in involved in the evaluation;
5.  [w]hen milestones will be reached and when the evaluation will be completed; [and]
6. [w]hat resources are available to conduct the evaluation (World Bank, 2011).

Why develop a ToR?

There are multiple benefits for developing a ToR for an evaluation. Most fundamentally, it puts everyone on the same page about what is expected from the evaluation, what the various roles and responsibilities are, and what the boundaries are. As such:

• it facilitates communication between the people with a stake in the program/project (who would, at a minimum, include the program/project team, members of the population affected by the program/project, the program/project’s funder, and the selected evaluator(s));
• helps avoid scope creep;
• makes it easier to hold people accountable.

Structure and content of a ToR for an evaluation

Digging deeper into the ‘why’ of ToRs: the ToR’s multiple functions…

The “[m]ultiple functions [that] are served by the ToR” include being to:

1. share background knowledge about the program/project, the sector, country, region, etc., in which it operates, and any relevant known effects of the program/project;
2. identify the specific evaluation questions, including why the evaluation is being conducted, its objectives, and its intended users;
3. define the scope of the evaluation (time period, depth, etc.) and, to a greater or lesser extent, the approach and methodology to be adopted;
4. articulate the governance, management, and coordination arrangements, specifically by outlining the roles and responsibilities of the various stakeholders engaged to facilitate or participate in the work and/or in related or parallel work;
5. establish the guiding principles and research ethics or procedures to be followed in the conduct of the evaluation;
6. identify the professional qualifications of the evaluator(s), including the required (or desired) experience, credentials, and competencies;
7. specify the expected deliverables (including details related to format, content, length, intended audience, and the expected review process) and related deadlines; and
8. provide an indication of the financial, human, and other resources available for the evaluation and what those resources cover (World Bank, 2011).

…which inform the ToR’s structure and content

It should come as no surprise to learn that the various purposes of the ToR help clarify what structure and content are warranted for the ToR document itself. In fact, structuring the ToR in the same sequence as the purposes presented in the previous sub-section works perfectly well. Each purpose could be represented as its own section in the ToR, populated with as much detail as necessary to fulfill the purpose.

It should be noted that it may not always be possible to lay out all the details related to the above-mentioned purposes of a ToR – and in some cases, such as defining the methodology, doing so at a too great level of specificity may even be counterproductive (believe me, I’ve had direct experience of this during my ‘[m]ost frustrating or challenging moment’ in my M&E work!). However, to the extent that these elements are at least considered and deliberated upon ex ante, the quality of the evaluation will likely be greatly enhanced.

In its final form, the ToR should be comprehensive yet concise. It’s tough to set a strict limit on the number of pages that a ToR should span (or, for more accuracy, the number of words that it should contain), but it should ideally not exceed 10 pages, plus appendices as needed to supplement the core information.

Conclusion – and then it’s over to you!

Developing a well-written ToR is a critical part of preparing for an evaluation. If you have an evaluation coming up for your program or project, get the juices flowing by outlining the ToRs and filling out each section with preliminary content (bullet points are fine at this stage) for further discussion with your program/project team, members of the population that the program/project affects, and the program’s/project’s donor(s) (and potentially other interested parties). If you’re a member of ‘Reimagine M&E: The Workshop’, you have exclusive access to a handy template with detailed section-by-section guidance from which to kick things off.

But don’t get too far ahead of yourself! A ToR is only one piece in the evaluation preparation puzzle. Watch this space for part 2 of this article where we’ll dive into the non-ToR-specific elements that can make or break an evaluation. (Spoiler alert! Engaging meaningfully with the stakeholders mentioned in the paragraph above is one of those elements.)

References:

BetterEvaluation (2023) What is evaluation? Web page. Available at: https://www.betterevaluation.org/getting-started/what-evaluation (accessed: 5 June 2023).

Rural Health Information Hub (2023) Importance of evaluation. Web page. Available at: https://www.ruralhealthinfo.org/toolkits/rural-toolkit/4/evaluation-importance (accessed: 5 June 2023).

World Bank (2011) Writing Terms of Reference for an Evaluation: A How-To Guide. Available at: https://documents1.worldbank.org/curated/en/209341599772583527/pdf/Writing-Terms-of-Reference-for-an-Evaluation-A-How-to-Guide.pdf (accessed: 5 June 2023).

Photo credit:

Gabriel Crismariu on Unsplash

Suggestion for how to cite this article (using APA 7 style):

Shejavali, K. (2023, June 5). Putting the pieces of the evaluation prep puzzle together: the ToR and beyond (part 1 of 2). Blog post. RM3 Consulting. Available at: https://www.rm3resources.com/blog/pieces-of-eval-prep-puzzle-part-1 (accessed: [insert the date that you last accessed this article at the link provided]).