## New Common Evaluation Framework for IMF Capacity Development

## Source details

**Canonical URL:** [New Common Evaluation Framework for IMF Capacity Development](https://www.imf.org/-/media/files/publications/pp/new-common-evaluation-framework-for-imf-capacity-development.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/pp/new-common-evaluation-framework-for-imf-capacity-development.pdf.md)
- [Structured JSON version](/-/media/files/publications/pp/new-common-evaluation-framework-for-imf-capacity-development.pdf.json)

---

### Executive summary — overview and objectives
- Framework purpose:
  - Adopt a common four-step process for all CD evaluations using OECD DAC evaluation criteria.
  - Produce shorter, more focused, and more comparable evaluations.
  - Improve information supporting evaluations.
  - Maintain current resource levels while using them more efficiently.
  - Use evaluation information to alter practices or shift CD targeting.
- Specific objectives and tools:
  - Specify clearly the objectives of all CD activities to be evaluated.
  - Apply a four-step common framework to all evaluations.
  - Ensure clarity and consistency in definition and application of OECD DAC evaluation criteria.
  - Use standardized Terms of Reference templates to reduce workload and sharpen focus of evaluations.

### Context, rationale, and recent practice
- Rationale:
  - Regular evaluation fosters learning and enhances accountability; Board discussion of The Fund’s Capacity Development Strategy—Better Policies through Stronger Institutions (2013) endorsed strengthening monitoring and evaluation.
  - Since 2013, ICD has developed proposals with departments to improve evaluation framework, discussing current practices with TA-providing, area, and other departments.
- Recent practice highlights:
  - Over the past three years, an average of about seven evaluations per year have been prepared, majority for externally-financed CD activities.
  - Recent IMF practice has been about 8-10 evaluations per year, majority external evaluations of donor-financed vehicles (e.g. RTACs, TTFs, some bilateral activities).
  - STA reviews averaging about one review per year since 2005 and incorporating RBM elements since 2011.
  - Triennial survey conducted every three years since 1995; most recent survey completed in 2015 covering the period 2012–14.

### Existing framework — TA evaluations (current practice)
- Types of TA evaluations:
  - Fund-wide evaluations (regular review of the Fund’s Capacity Development strategy expected every five years).
  - External evaluations of Regional Technical Assistance Centers (RTACs), Topical Trust Funds (TTFs), and some bilateral accounts.
  - Self-assessment evaluations conducted by TA departments.
- Common elements and methods in external evaluations:
  - Assessments of specific TA and training activities; planning, execution, and operations; case studies and desk-based research on workshops.
  - Typical methods: interviews with IMF HQ departments; documentation reviews (e.g., briefing papers and BTOs); detailed reviews of published and internal reports; field trips to RTACs; surveys of TA recipients; online surveys of beneficiaries, workshop participants, and Steering Committee members.
  - Most recent evaluations use OECD DAC criteria: Relevance, Effectiveness, Impact, Efficiency, and Sustainability.
- Self-evaluations:
  - Departments undertake internal evaluations (examples: FAD inspection missions; MCM ad-hoc ex post evaluations and assessment visits; LEG regular evaluations).
  - IEO assessment (2015) found considerable self-evaluation and many activities/reports of high technical quality; Executive Directors supported better distillation and dissemination of lessons.

### Existing framework — Training evaluations (current practice)
- Kirkpatrick four-level model used: Level 1 reaction, Level 2 learning, Level 3 behavior, Level 4 results.
- Instruments and practices:
  - End-of-course surveys (Level 1): administered final day with response rate close to 100 percent; online courses target a 50 percent response rate and have surveys open ~one week.
  - Pre- and post-course tests (Level 2): administered by all ICD training divisions with varying coverage; not yet standardized across divisions.
  - Follow-up surveys (Level 3): introduced in 2006; sent one year to eighteen months after a course to participants and sponsoring managers; coverage selective with eight courses per year surveyed since FY11.
  - Triennial survey of sponsoring agencies (combines Levels 1, 3, and 4).

### Strengths and weaknesses of the current system — key findings
- Strengths:
  - Regular evaluation of TA activities and procedures to allow lessons to influence future TA delivery (e.g., staff responses and action plans to mid-term RTAC and TTF evaluations).
- Weaknesses:
  - Lack of consistent and comparable methodologies across evaluations; wide variation in definitions, methodology, and substantive focus.
  - Definitions of DAC criteria have varied and the quality of evidence gathered has often been disappointing.
  - Insufficient agreement and clarity about CD objectives, indicators, and milestones inhibiting systematic assessment.
  - Need for agreed indicators and standardized process for collection and aggregation of information.
  - RTAC evaluations: about one-third of evaluations done in the past three years; reports often long, inconsistent in structure and key questions, objectives not stated clearly ex ante, and case studies costly with dubious merit when taken individually.

### Information and data requirements for good evaluation
- Minimum data requirements:
  - Baseline and ex-post information on the key indicators and control variables.
  - Suggested additional records: whether a TA request was initiated by the authorities or IMF staff.
- Complementary methods and practices:
  - More face-to-face interviews (suggested cadence: interviews with senior officials at 2-3 year intervals).
  - Inputs from staff in area departments.
  - “Flash” information drawing on RBM data to facilitate fast identification of problems.
- Survey limitations and remedies:
  - Current survey information helpful but perceived to have upward bias in ratings.
  - Extend pre- and post-course tests to all IMF training courses; tests should be standardized for each course across offerings (standard core list of questions with scope for revisions and region-specific customization).
  - Consider randomized control trials (e.g., quizzes for participants and non-participants to evaluate impact across modalities such as online vs. face-to-face).
  - Revise end-of-course and follow-up surveys to be more results-oriented:
    - New end-of-course questions on perceived usefulness for on-the-job performance, job relevance, whether training contained new information, and intention to use skills/knowledge.
    - Introduce a systematic and course-specific follow-up survey three to six months after training (instead of current one year to eighteen months); new follow-up questions to include extent of application of knowledge/skills, use of tools/techniques taught, and barriers/enablers to application.
  - Maintain a channel for regular reviews of training course content and occasional independent expert attendance at lectures.
  - Introduce a short annual questionnaire for IMF country teams/resident representatives (expected time to complete: no more than 15 minutes) to gather input on use of tools taught (examples: Financial Programming and Policies (FPP), Debt Sustainability Analysis (DSA), External Balance Assessment (EBA), Early Warning Exercise (EWE)).

### Four-step common evaluation framework — structure and standardization
- Four steps to be applied to all evaluations:
  1. Define the log frame or causal chain from inputs to outcomes.
  2. Indicate what is likely to have happened if the IMF did not deliver the CD (assess the counterfactual).
  3. Assess outcomes using OECD DAC evaluation criteria (relevance, effectiveness, impact, efficiency, sustainability).
  4. Discuss why achievement of the DAC criteria were low/high, explanatory factors, and whether alternative interventions might have provided better results.
- Standardization measures:
  - Ensure clarity and consistency in definition and application of OECD DAC criteria; Annex I provides definitions and typical evaluation questions.
  - Use standardized Terms of Reference (ToR) templates based on the four-step process and including sample questions to assess each DAC criterion; set page limits (e.g. no more than 25-30 pages) to ensure focused evaluations.
  - ICD’s SE division will review ToRs (for external and internal evaluations) to ensure consistency.

### Counterfactuals, impact assessment, and evaluation rigor
- Counterfactual use:
  - Assess impact against a counterfactual quantitatively (preferred, where feasible) or qualitatively.
  - If several “without-intervention” scenarios exist, provide likelihoods and select one for impact assessment; provide informed assessment of probabilities and confidence.
  - Note: a counterfactual is not the same as a baseline (baseline is a special case where pre-intervention variables are unchanged).
- Examples:
  - If IMF TA was unique and no alternative available → counterfactual would be no TA delivered.
  - In other cases, an alternative organization may have offered similar TA.

### Data collection planning, ex-post information, and CD-PORT/RBM integration
- Data planning and timing:
  - Ensure all CD activities have a data collection plan in place before the start of the CD activity; baseline data should be obtained before TA and training begins.
  - Through the IMF’s RBM system there will be agreement on indicators, data sources, and responsibilities for data collection.
- Ex-post information collection:
  - An “ex-post” survey to TA recipients should be administered immediately after the TA activity concludes (FAD already conducts such surveys; a revised version should be extended to all TA).
  - Develop a questionnaire/survey to collect information from providers, recipients, and area department staff on outcomes of the TA provided within 12 months after TA conclusion, where not already collected elsewhere (e.g. in CD-PORT). Store, organize, and use this information for future evaluations.
- Use of CD-PORT:
  - Implement systematic RBM information into CD-PORT to facilitate evaluations and lower costs.

### Allocating evaluation resources, planning, and flexibility
- Resource and planning principles:
  - Maintain reasonable accountability standards without evaluating everything; ensure a significant chance that a particular activity will be evaluated.
  - Implement the common evaluation framework expected to be budget neutral by using current resources more effectively.
  - Introduce a CD evaluation work plan with a rolling three-year horizon, revised annually; work plan to include all evaluations underway or planned and plans to collect better data and information for future evaluations.
  - The proposed work plan would be discussed and endorsed by the CCB at its July meeting, and updated annually.
  - Decide what and how to evaluate based on mandatory evaluations (e.g. RTAC/TTF/some bilateral subaccount evaluations mandated by partners), potential for answering important questions with data, perceived importance, and cost.
- Maintain flexibility:
  - Allow quick mid-term evaluations or rapid evaluations of urgent topics as part of the work plan (e.g., diagnose problems in a TA program or investigate causes of particularly favorable results).

### Using evaluation information to alter practices and shift resources
- Quantitative scoring and aggregation:
  - Introduce quantitative scoring to complement qualitative information; five DAC criteria will be scored on a 1-4 scale and averaged.
  - To obtain an aggregate score for activities with multiple objectives, assign weights to objectives (example: Objective or Outcome 1 gets a 30 percent weight; Objective or Outcome 2 a 70 percent weight).
  - For TA, such weights are expected to be included in the RBM system and used in evaluations; for training, weights could be agreed in the evaluation ToR.
  - For the overall score, compute a weighted average of criterion scores using objective weights; each evaluation will have a score attached to each objective/outcome and a summary score for the whole evaluation.
- Dissemination and use:
  - Ensure evaluations are easily accessible upon completion and findings are broadly shared among Fund staff.
  - ICD will produce a summary report of evaluation findings each year; the report would summarize performance scores and substantive lessons learned.
  - Annual summary report could be discussed at the July CCB meeting and posted on the CCB’s intranet page.
  - The November CCB meeting would be expected to use evaluation results to assist with setting CD priorities; CD priorities and activities for the coming three years are discussed annually at the November CCB meetings.
  - Responses to findings can range widely, including recommending different interventions, piloting different modes, or terminating interventions.

### Annex: The Four Steps — practical guidance and illustrative questions
- STEP 1. The Causal Chain:
  - Link inputs → activities → outputs → milestones → outcomes → objectives; RBM catalog in CD-PORT normally the source for log frames.
  - Example elements: Inputs, Activities, Outputs, Milestones, Outcomes, Objectives, Fund deliverables, Benefits to recipients, Indicators.
- STEP 2. Describe the counterfactual:
  - Assess what most likely would have occurred without the intervention, quantitatively where feasible; if multiple scenarios, provide likelihoods and select one for impact assessment.
- STEP 3. Assess outcomes using OECD DAC criteria:
  - Relevance, Effectiveness, Impact, Efficiency, Sustainability.
  - Selected example evaluation questions:
    - Relevance: How high did national authorities rank objectives on their list of priorities (scale 1-10)?
    - Effectiveness: To what extent were the objectives achieved or are likely to be achieved?
    - Impact: List all changes attributable to the CD compared to the chosen counterfactual; provide quantitative estimates if possible.
    - Efficiency: Provide estimates of costs and compare to value of impacts or alternative delivery methods where possible.
    - Sustainability: Assess likelihood outcomes/benefits will continue; assess knowledge dissemination and retention.
- STEP 4. Results and Alternatives:
  - Explain why DAC criteria achievement was low/high and evaluate alternative interventions with supporting evidence; recommendations should consider DAC criteria trade-offs (e.g., effectiveness vs. efficiency).

### Box 1 — theoretical underpinnings of training evaluations
- Kirkpatrick (1976) four-level model: reaction, learning, behavior, results.
  - Reaction: participant satisfaction and planned action; positive reaction does not indicate learning.
  - Learning: measured via pre/post tests and control groups where practical.
  - Behavior: changes in on-the-job behavior; best practice includes before-and-after appraisal by comprehensive group and statistical analysis with control group; appraisals should be three months or more after training.
  - Results: ultimate objectives of training programs; evaluation at this level is extremely difficult due to causality and confounding factors.
- Literature notes:
  - Phillips added a fifth level labeled return on investment (ROI) in the 1980s.
  - Kraiger et al (1993) proposed multi-dimensional learning (cognitive, affective, skill-based outcomes).
  - Alliger et al (1997) found utility-type reaction measures more predictive of transfer than affective-type reaction measures and even more predictive than some learning measures.

*International Monetary Fund — New Common Evaluation Framework for IMF Capacity Development (Executive Summary), April 7, 2017*

### EXECUTIVE SUMMARY

### EXECUTIVE SUMMARY

### Overview
- This document outlines a new common evaluation framework for the Fund’s capacity development (CD) activities.
- The framework adopts a common four-step process for all CD evaluations and uses the OECD Development Assistance Committee (DAC) evaluation criteria.
- Key objectives are grouped to: produce shorter/more focused/comparable evaluations; improve information supporting evaluations; maintain current resource levels while using them more efficiently; and use evaluation information to alter practices or shift CD targeting.

### Objectives of the new framework
- Produce shorter, more focused, and more comparable evaluations:
  - Specify clearly the objectives of all CD activities to be evaluated
  - Apply a four-step common framework (thought process) to all evaluations
  - Ensure clarity and consistency in definition and application of internationally-accepted OECD DAC evaluation criteria
  - Use standardized Terms of Reference templates to reduce workload and sharpen the focus of evaluations
- Improve the information supporting evaluations:
  - Ensure that all CD activities have a data collection plan in place before the start of the CD activity
  - Obtain ex-post information from TA providers and recipients and other stakeholders
  - Extend the pre- and post-course tests to all IMF training courses
  - Revise the end-of-course and follow-up survey questions for training to be more results oriented
  - Maintain a channel for regular reviews of training course content
  - Seek information from IMF country teams on the extent to which they observe tools learned in IMF training being applied at the country level
- Maintain the current level of resources allocated to evaluation but use these scarce evaluation resources more efficiently:
  - Establish reasonable accountability standards without evaluating everything
  - Introduce a CD evaluation work plan with a rolling three-year horizon, revised annually
  - Decide what and how to evaluate based on potential value of the information, cost, and achievement of accountability standards
  - Maintain the flexibility to conduct mid-term or rapid evaluations of TA projects
- Use the information from evaluations to alter practices and/or shift resources:
  - Introduce quantitative scoring to complement qualitative information in evaluations to facilitate comparisons and aggregation
  - Ensure evaluations are easily accessible upon completion and findings are broadly shared among Fund staff
  - The November CCB meetings would be expected to use evaluation results when setting CD priorities

### Context and rationale
- Regular evaluation is a crucial component of a sound CD strategy to foster learning from past experiences and enhance accountability (Board discussion of The Fund’s Capacity Development Strategy—Better Policies through Stronger Institutions (2013) endorsed strengthening monitoring and evaluation).
- Since 2013, ICD has worked with departments to develop proposals to improve the Fund’s evaluation framework for CD, discussing current practices with TA-providing, area, and other departments.
- Strengths of the current system include regular evaluation of both TA and training activities, regardless of funding source, and mechanisms for evaluation results to influence future CD delivery.
- Weaknesses identified include TA evaluations often being long and unfocused, and training evaluations sometimes leaving important questions unanswered.
- Proposals consider feasibility and cost; changes target preserving strengths while addressing weaknesses.
- The document reviews current framework (Section II), strengths/shortcomings (Section III), and presents the new common framework (Section IV). A companion guidance note will describe methodology and operational guidance for applying the four-step framework to TA and training.

### The existing framework — TA evaluations (summary of current practice)
- The Fund undertakes regular internal and external evaluations of its TA, including:
  - Fund-wide evaluations
  - External evaluations of Regional Technical Assistance Centers (RTACs), Topical Trust Funds (TTFs), and some bilateral accounts
  - Self-assessment evaluations conducted by TA departments
- Fund-wide evaluations assess overall CD policies and activities; the main vehicle for Executive Board assessment is the regular review of the Fund’s Capacity Development strategy, expected every five years.
- External evaluations of RTACs, TTFs, and some bilateral subaccounts:
  - Typically conducted midway through each RTAC’s five-year cycle
  - Evaluation subcommittee (including Fund staff) determines terms of reference and chooses the evaluator
  - Common elements: assessments of specific TA and training activities; assessments of planning, execution, and operations; case studies and desk-based research on workshops
  - Typical methods: interviews with relevant IMF HQ departments; documentation reviews (e.g., briefing papers and BTOs); detailed reviews of published and internal reports; field trips to RTACs; surveys of TA recipients; online surveys of beneficiaries, workshop participants, and Steering Committee members
  - Most recent evaluations use the OECD DAC criteria: Relevance, Effectiveness, Impact, Efficiency, and Sustainability
- TA departments’ self-evaluations:
  - Departments undertake evaluations to assess impact and extract lessons (examples: FAD inspection missions; MCM ad-hoc ex post evaluations and assessment visits; STA reviews averaging about one review per year since 2005 and incorporating RBM elements since 2011; LEG regular evaluations)
- IEO assessment:
  - The IEO published in 2015 an assessment of self-evaluation at the IMF, finding considerable self-evaluation and many activities/reports of high technical quality
  - Executive Directors supported better distillation and dissemination of self-evaluation lessons; the Managing Director supported this recommendation

### The existing framework — Training evaluations (summary of current practice)
- Fund training evaluations follow the four-level Kirkpatrick (1976) model:
  - Level 1: reaction (participants’ satisfaction)
  - Level 2: learning (knowledge acquisition)
  - Level 3: behavior (whether behavior changes following the course)
  - Level 4: results (final results or outcomes from attendance)
- Current training evaluation instruments include:
  - End-of-course surveys of participants (Level 1)
  - Pre- and post-course tests (Level 2)
  - Follow-up surveys of participants and sponsors (Level 3)
  - The triennial survey of sponsoring agencies (a combination of Levels 1, 3, and 4)
- End-of-course surveys:
  - Administered on the final day of every course with a response rate close to 100 percent
  - Reactions and suggestions collected via questionnaire and closing-session probing
  - ICD transmits information to senior managers via back-to-office reports and to ICD’s Curriculum Development Committee (CDC)
  - TA departments offering training under ICD include FAD, FIN, LEG, MCM, and STA; non-ICD training varies in survey practice
  - Online courses (SPOCs and MOOCs) have end-of-course surveys open for about a week targeting a 50 percent response rate
- Pre- and post-course tests:
  - Administered by all ICD training divisions with varying coverage across courses, time, and division
  - Tests are not yet standardized across divisions; ICD’s CDC is discussing plans for standardization
  - Online courses conduct tests at the beginning and end of each course
- Follow-up surveys:
  - Introduced in 2006; sent one year to eighteen months after a course to participants and sponsoring managers
  - Ask whether courses help participants do their job better, aid careers, and whether knowledge is used and shared
  - Conducted by an independent external firm to ensure anonymity
  - Coverage has remained selective; since FY11, eight courses per year have been surveyed with attention to representative sampling by region, topic, and language
- Triennial survey:
  - Conducted every three years since 1995; most recent survey completed in 2015 covering the period 2012–14
  - Sent to sponsoring government agencies to gather views on ICD training effectiveness and future needs; results summarized in a memorandum to management and posted on the intranet

### Strengths and weaknesses of the current system — Key findings
- Strengths:
  - Regular evaluation of TA activities is a key strength; over the past three years, an average of about seven evaluations per year have been prepared, majority for externally-financed CD activities
  - Procedures exist to allow lessons from evaluations to influence future TA delivery (e.g., staff responses and action plans to mid-term evaluations for RTAC and TTF evaluations)
- Weaknesses:
  - Lack of consistent and comparable methodologies across evaluations; definitions, methodology, and substantive focus vary widely, limiting cross-comparison
  - External evaluations and some internal evaluations use OECD DAC criteria, but definitions have varied and the quality of evidence gathered has often been disappointing
  - Insufficient agreement and clarity about CD objectives, indicators, and milestones has inhibited systematic assessment
  - Need for agreed indicators and a standardized process for collection and aggregation of information
  - IMF’s RBM system provides standardized objectives, outcomes, indicators, and systematic articulation of baselines for main TA work streams

_International Monetary Fund — New Common Evaluation Framework for IMF Capacity Development (Executive Summary), April 7, 2017_

### 19. Good evaluation will require better information and data. The minimum

### new-common-evaluation-framework-for-imf-capacity-development - 19. Good evaluation will require better information and data. The minimum

### Information and data requirements for good evaluation
- Minimum requirement: baseline and ex-post information on the key indicators and control variables.
- Suggested additional records: whether a TA request was initiated by the authorities or IMF staff working on the country.
- Current survey information: helpful but perceived to have a tendency toward upward bias in ratings.
- Recommended complementary methods:
  - More face-to-face interviews to bring out important issues and complement surveys (suggested cadence: interviews with senior officials at 2-3 year intervals).
  - Inputs from staff in area departments.
  - “Flash” information drawing on RBM data to facilitate fast identification of problems while time for remedies still exists.

### RTAC evaluations and methodological weaknesses
- About one-third of evaluations done in the past three years have been for RTACs.
- Analysis finds wide variation in use of DAC criteria, definitions, indicators, and overall methodology; no common methodology effectively applied.
- Key weaknesses identified:
  - little standardization in terms of structure of the documents or the data presented and inconsistency in key evaluation questions across reports, hampering comparison;
  - unnecessarily long evaluation reports;
  - the objectives of the entity being evaluated are not stated clearly or not stated at all, especially ex ante;
  - evaluations sometimes attempt to evaluate and also provide views on RTAC management and operational advice even when not relevant to TA delivery;
  - case studies tend to be costly and are often of dubious merit, particularly when taken individually.

### Counterfactuals and impact assessment
- Inadequate use of counterfactuals: assessments of impact and efficiency require information/judgment on what would have happened if the CD had not been delivered or had been delivered differently; this is rarely incorporated in Fund TA evaluations.
- The four-step common evaluation framework (Step two) provides scope for evaluators to assess likely outcomes in the absence of the CD delivery (i.e., the counterfactual).
- Examples:
  - If IMF TA was unique and no alternative available → counterfactual would be no TA delivered.
  - In other cases, an alternative organization may have offered similar TA.

### Training: current practice, shortcomings, and proposed improvements
- Current strengths:
  - Use of participant surveys administered at the end of all courses.
  - Lecturers and supervisors solicit feedback and frequently alter courses in response.
  - Participant ratings influence selection of course counselors and guest lecturers; lectures can be dropped or revised based on lower participant ratings.
- Shortcomings of current system:
  - Limited information on whether participants use tools taught when back on the job.
  - Reliance on participant surveys is inadequate for judging appropriateness of topics, fairness/comprehensiveness of evidence, or technical correctness of lectures.
  - Participant surveys are valuable mostly for measuring reaction and less valuable for learning, behavior change, or results.
  - Evaluations do not currently employ a control group or counterfactual; some before/after tests exist.
- Proposed data and design enhancements:
  - Extend pre- and post-course tests to all IMF training courses; tests should be standardized for each course across offerings (standard core list of questions with scope for revisions and region-specific customization).
  - Consider randomized control trials (e.g. quizzes for participants and non-participants to evaluate impact across modalities such as online vs. face-to-face).
  - Revise end-of-course and follow-up surveys to be more results-oriented:
    - New end-of-course questions on perceived usefulness for on-the-job performance, job relevance, whether training contained new information, and intention to use skills/knowledge.
    - Introduce a systematic and course-specific follow-up survey three to six months (rather than the current 12-18 months) after training. New follow-up questions to include extent of application of knowledge/skills, use of tools/techniques taught (for tool-focused courses), and barriers/enablers to application.
  - Maintain a channel for regular reviews of training course content (curriculum-level reviews and occasional independent expert attendance at lectures).
  - Introduce a short annual questionnaire for IMF country teams/resident representatives (expected time to complete: no more than 15 minutes) to gather input on use of tools taught at IMF training (examples listed: Financial Programming and Policies (FPP), Debt Sustainability Analysis (DSA), External Balance Assessment (EBA), Early Warning Exercise (EWE), and some forecasting techniques) and the level of expertise in using them.

### Four-step common evaluation framework and standardization
- Apply a four-step common framework to all evaluations (explained in Annex I; further detailed in forthcoming companion note on methodology). The four steps are:
  1. define the log frame or causal chain from inputs to outcomes;
  2. indicate what is likely to have happened if the IMF did not deliver the CD (assess the counterfactual) to help assess impact;
  3. assess outcomes using the OECD DAC evaluation criteria; and
  4. discuss why achievement of the DAC criteria were low/high, what factors explain it, and whether alternative interventions might have provided better results.
- Ensure clarity and consistency in definition and application of OECD DAC evaluation criteria; Annex I provides definitions and typical evaluation questions.
- Use standardized Terms of Reference (ToR) templates based on the four-step process and including sample questions to assess each DAC criterion; set page limits (e.g. no more than 25-30 pages) to help ensure focused evaluations.
- ICD’s SE division will review ToRs (for external and internal evaluations) to ensure consistency.

### Data collection planning and ex-post information
- Ensure all CD activities have a data collection plan in place before the start of the CD activity; baseline data should be obtained before technical assistance and training begins.
- Through the IMF’s RBM system there will be agreement on indicators, data sources, and responsibilities for data collection.
- Obtain ex-post information from TA providers and recipients and other stakeholders:
  - An “ex-post” survey to TA recipients should be administered immediately after the TA activity concludes (FAD already conducts such surveys; a revised version should be extended to all TA).
  - Develop a questionnaire/survey to collect information from providers, recipients, and area department staff on outcomes of the TA provided within 12 months after TA conclusion, where not already collected elsewhere (e.g. in CD-PORT). Store, organize, and use this information for future evaluations.

### Allocating evaluation resources and planning
- Principle: maintain reasonable accountability standards without evaluating everything; a significant chance that a particular activity will be evaluated is required.
- Recent practice: IMF has been doing about 8-10 evaluations per year, majority external evaluations of donor-financed vehicles (e.g. RTACs, TTFs, some bilateral activities).
- Implementing the common evaluation framework is expected to be budget neutral by using current resources more effectively and putting systematic RBM information into CD-PORT to facilitate evaluations and lower costs.
- Introduce a CD evaluation work plan with a rolling three-year horizon, revised annually:
  - The work plan will include all evaluations underway or planned and plans to collect better data and information for future evaluations (including RBM implementation).
  - Topics could include specific projects and broader issues (e.g. whether risks identified in surveillance are being sufficiently addressed in TA work).
  - The proposed work plan would be discussed and endorsed by the CCB at its July meeting, and updated annually.
- Decide what and how to evaluate based on:
  - which evaluations must be done (e.g. RTAC/TTF/some bilateral subaccount evaluations mandated by partners) and which would be beneficial;
  - potential for answering important questions with the data;
  - perceived importance of the question to be addressed;
  - cost.

*Source: NEW COMMON EVALUATION FRAMEWORK FOR IMF CD, INTERNATIONAL MONETARY FUND.*

### 40. Maintain the flexibility to conduct mid-term or rapid evaluations. It would be

### NEW COMMON EVALUATION FRAMEWORK FOR IMF CD

### Maintain the flexibility to conduct mid-term or rapid evaluations
- Allow quick mid-term evaluations or evaluations of topics that become urgent during the year as part of the evaluation work plan.
- Use cases:
  - If it is suspected that a TA program is running into problems, conduct a quick evaluation of causes and possible remedies.
  - If results from certain TA activities are very favorable, conduct a quick evaluation to determine causes of success and lessons learned to apply to current and future CD activities.

### D. Using the Information from Evaluations to Alter Practices and/or Shift Resources

- Introduce quantitative scoring in evaluations to facilitate comparisons and aggregation:
  - To obtain an aggregate score for evaluations of activities with multiple objectives or outcomes, weights must be assigned (e.g. Objective or Outcome 1 gets a 30 percent weight; Objective or Outcome 2 a 70 percent weight).
  - For TA, such weights are expected to be included in the RBM system and these will be used in evaluations.
  - For training activities, weights could be assigned and agreed as a prelude to any evaluation, such as in the evaluation ToR.
  - The five criteria in the DAC framework will be scored on a 1-4 scale and averaged.
  - For the overall score of the evaluation, a weighted average of these scores will be computed with the weights given to the objectives.
  - Outcome: each evaluation will have a score attached to each objective or outcome and a summary score for the whole evaluation.

- Ensure evaluations are easily accessible upon completion and findings are broadly shared among Fund staff:
  - ICD will produce a summary report of evaluation findings each year.19
  - The report would summarize performance scores from several evaluations, as well as substantive lessons learned.20
  - The report could be discussed at the July CCB meeting, where the outcomes for the previous fiscal year are discussed, and draw lessons for future delivery.21
  - This report, along with underlying evaluations, would be posted on the CCB’s intranet page.
  - Evaluation summaries will feed into the regular review of CD activities done every five years, with the objective of improving future CD delivery.
  - Note: As noted in paragraph 37, recently there have been about 8-10 CD evaluations produced per year on average.20 A similar report had been done previously by OTM, with the last one produced in 2010.21 Responses to findings from evaluations can range widely, including recommending different kinds of interventions, different modes of interventions on a pilot basis, and terminating interventions.
  - These activities would be part of the regular work of the CCB and will not involve additional resources.

- Use of evaluations in setting priorities:
  - The November CCB meeting would be expected to use evaluation results to assist with setting CD priorities.
  - CD priorities and activities for the coming three years are discussed annually at the November CCB meetings.
  - The annual evaluation report, discussed during the July CCB meeting, can provide additional information to help determine the medium-term CD priorities in November.

### Annex 1. The Four Steps in the Proposed Common Evaluation Framework

- STEP 1. The Causal Chain: Defining the Logical Framework from Input to Outcomes
  - A logical framework (log frame or results chain) links inputs (e.g. financial and human resources) to activities (e.g. missions, backstopping, delivering a training course) to outputs (e.g., a TA report) to outcomes (i.e. actual capacity improvements) and milestones en route to objectives.
  - Clarify multiple activities, exogenous variables, and whether activities are helpful, necessary, and/or sufficient; state ultimate objective even if not evaluated in all evaluations.
  - The RBM catalog in CD-PORT contains log frames and will normally be the most appropriate source for the log frames for TA and training.
  - Example Results Chain elements: Inputs, Activities, Outputs, Milestones, Outcomes, Objectives, Fund deliverables, Benefits to recipients, Indicators.

- STEP 2. Describe the counterfactual
  - Assess impact against a counterfactual: what would most likely have occurred without the intervention, either quantitatively (preferred, where feasible) or qualitatively.
  - If several “without-intervention” scenarios exist, provide likelihoods of major scenarios and select one for impact assessment; provide informed assessment of probabilities and confidence.
  - This step should be based on a-priori reasoning, client interviews, and/or other evidence.
  - Note: a counterfactual is not the same as a baseline, which is a special case of a counterfactual in which the pre-intervention variables are unchanged.

- STEP 3. Assessing Outcomes Using the OECD DAC Criteria
  - Use the five DAC criteria: relevance, effectiveness, impact, efficiency and sustainability.
  - Decompose the activity into interventions and objectives; for each intervention-objective pair, ask:
    - Was objective X relevant?
    - Did intervention Y achieve objective X:
      (a) Effectively?
      (b) With impact?
      (c) Efficiently?
      (d) Sustainably?
  - When an overall performance rating is desired for an entity or series of interventions with different objectives, assign weights for achievement of those objectives; for TA projects, such weights are expected in the Results Based Management system.
  - Definitions and example evaluation questions for each DAC criterion (selected examples from the table):
    - Relevance: extent to which CD activities served important objectives.
      - How high did the national authorities rank the objectives of the CD activity on their list of priorities (scale 1-10)?
      - To what extent were objectives derived from capacity gaps identified by others or international standards?
      - To what extent did objectives come from surveillance or program priorities for the country?
    - Effectiveness: extent to which CD activities attained their objectives.
      - To what extent were the objectives of the CD activity achieved or are likely to be achieved?
    - Impact: measures positive and negative changes attributable to the CD activity compared to the counterfactual.
      - List all changes that can be attributed to the CD activity, whether intended or not, compared to the counterfactual you believe would have been most likely.
      - Provide quantitative estimates of these impacts, if possible.
    - Efficiency: measures monetary value of outcomes or benefits compared to inputs or costs; an impact assessment is required to assess efficiency.
      - Provide estimates of the costs of the CD activity, to the fullest extent possible.
      - Estimate the value of impacts and compare to costs incurred, if possible.
      - Provide estimates of costs of alternative delivery methods, if possible.
      - If no monetary estimates, assess whether CD delivered is minimum cost via comparison with similar activities or examination of process and implementation (e.g., excessive staff turnover, unnecessary delays).
    - Sustainability: extent to which outcomes or benefits are likely to continue.
      - Assess degree to which transfer of knowledge will be further disseminated.
      - Assess extent to which skills and knowledge will be retained and used on the job.
      - Assess extent to which funding for CD will continue (no presumption that continued funding is necessarily desirable).
      - If objective was behavioral change or support for new policies or laws, assess persistence.

- STEP 4. Results and Alternatives
  - Examine why achievement of the DAC criteria was low/high and what factors explain it; and whether alternative interventions would have provided better results.
  - Possible explanatory factors for performance include:
    - The intervention was based on a sound diagnosis of the key problem
    - The mode of delivery was innovative
    - Background conditions required for success were implemented simultaneously
    - The original rationale for the intervention was poor
    - Unforeseen exogenous factors changed and undermined the success of the project
  - When suggesting alternative interventions, the evaluator must describe reasoning and supporting evidence.
  - Recommendations should consider the DAC criteria together; interventions that improve some criteria but fail others (e.g., increase effectiveness but reduce efficiency due to high costs) are not viable recommendations.

### Box 1. Theoretical Underpinnings of Training Evaluations
- IMF training logical framework builds on Kirkpatrick’s (1976) four-level model: reaction, learning, behavior, and results.
- Four levels:
  - Reaction: measures participant satisfaction and planned action; a positive reaction does not indicate learning.
  - Learning: determines objectively the amount of learning, typically using pre/post tests and control groups where practical.
  - Behavior: measures changes in on-the-job behavior; best practices include before-and-after appraisal by a comprehensive group and statistical analysis with a control group; appraisals should be made three months or more after training.
  - Results: ultimate specific objectives of training programs; evaluating at this level is extremely difficult due to causality and confounding factors.
- Notes from literature:
  - Phillips added a fifth level labeled return on investment (ROI) in the 1980s.
  - Kraiger et al (1993) proposed multi-dimensional learning (cognitive, affective, skill-based outcomes).
  - Alliger et al (1997) found utility-type reaction measures are more predictive of transfer than affective-type reaction measures and even more predictive than some learning measures.

*NEW COMMON EVALUATION FRAMEWORK FOR IMF CD — INTERNATIONAL MONETARY FUND*

---


_Source: https://www.imf.org/-/media/files/publications/pp/new-common-evaluation-framework-for-imf-capacity-development.pdf_