## jsa-2018

## Source details

**Canonical URL:** [jsa-2018](https://www.imf.org/-/media/files/capacity-developement/jsa-annual-reports/jsa-2018.pdf)

## Other formats

- [Markdown version](/-/media/files/capacity-developement/jsa-annual-reports/jsa-2018.pdf.md)
- [Structured JSON version](/-/media/files/capacity-developement/jsa-annual-reports/jsa-2018.pdf.json)

---

### Evaluation scope and purpose
- The evaluation assessed IMF capacity development projects supported by the Japanese Subaccount (JSA) in the period May 1, 2013 through April 30, 2017.
- Japan has contributed more than US$473 million since 1990 to IMF capacity development, enabling assistance to more than 120 member countries.
- Around 60 percent of the JSA CD budget is used in the Asia-Pacific region.
- Portfolio under review: 43 projects across 89 countries.

### Methodology and sample
- Evidence sources:
  - Extensive review of IMF project documentation and data.
  - Interviews with IMF HQ staff.
  - An electronic survey of IMF TA providers.
  - 12 case studies in four countries (Myanmar, Laos, Cambodia, Cameroon).
- Case studies represented 37 percent of expenditure and 28 percent of the number of projects in the JSA portfolio under review.
- Case study sample: 12 projects; expenditures sum to 23.851.557 (USD) and cover activity in 40 countries (on average in more than 3 countries in the region).

### Inputs — key statistics and allocations
- Total JSA-funded expenditure (IMF data for review period): USD 63,687,906.
- Total budget in review period (evaluators’ allocation approximation): USD 104,576,561 (100%).
- Distribution by IMF departments (expenditure shares, allocated expenditures through April 2017, USD):
  - FAD: 29.292.896 (46%), of which 11.527.714 not country-linked.
  - STA: 16.022.496 (25%), of which 8.799.385 not country-linked.
  - MCM: 14.124.760 (22%), of which 3.239.228 not country-linked.
  - ICD: 2.774.121 (4%), of which 1.926.984 not country-linked.
  - LEG: 1.473.633 (2%), of which 1.272.050 not country-linked.
  - Grand Total: 63.687.906 (100%), of which 26.765.360 not country-linked.
- Regional allocation (allocated expenditure through April 2017, USD):
  - Asia-Pacific: 47.518.363 (75%).
  - Eastern Europe / Caucasus: 5.304.779 (8%).
  - Sub-Saharan Africa: 10.864.764 (17%).
- Case study sample expenditures (IMF data through April 2017):
  - FAD: 7.556.260 (32% of sample).
  - ICD: 2.830.675 (12%).
  - MCM: 7.672.021 (32%).
  - STA: 5.792.602 (24%).
  - Grand Total sample: 23.851.557 (100%).

### Outputs — delivery modalities and cost shares
- Delivery modes and portfolio-level expenditure (USD, % of total):
  - Short-term missions (STX): 21.747.063 (34,1%).
  - Long-term deployments (LTX): 18.735.797 (29,4%).
  - Short-term missions by HQ staff (HQ): 13.461.593 (21,1%).
  - Seminars & study tours: 4.611.810 (7,2%).
  - Project backstopping: 2.348.294 (3,7%).
  - Project management: 1.656.924 (2,6%).
  - Language services: 881.403 (1,4%).
  - Local support: 219.971 (0,3%).
  - Miscellaneous: 25.051 (0,0%).
  - Total: 63.687.906 (100%).
- Portfolio observation:
  - IMF TA is delivered primarily through short missions of 1-2 weeks by STX or HQ staff.
  - More than 50 percent of the expenditure is directly related to STX and HQ short missions.
  - LTX accounts for 29 percent of costs; beneficiaries often prefer and find LTX/resident advisor support more effective and ultimately more cost-effective than STX/HQ support.
  - Some TA allocations were suboptimal (e.g., expensive experts for basic TA; TA to beneficiaries with insufficient commitment).

### Outputs quality, trainee selection, and follow-up
- General findings:
  - Outputs were generally and consistently very high; IMF is considered a “benchmark” and beneficiaries view its advice as “global best practice”.
  - Some TA or training activities were too complex or technically challenging for beneficiaries owing to low absorption capacity or suboptimal trainee selection.
  - In several cases short missions were too short for complex content and required follow-up that was often insufficient.
- Survey evidence (38 respondents):
  - 28 noted insufficient follow-up support.
  - 15 noted insufficient quantity of TA delivered.
- Common issues affecting training effectiveness:
  - Nomination and selection process of training participants.
  - Design of training programs vis-à-vis participant abilities.
  - Delivery considerations including language abilities.

### Outcomes — IMF reporting and evaluators’ findings
- IMF outcome achievement scale: 1=not achieved, 2=partially achieved, 3=largely achieved, 4=fully achieved.
- All 43 projects, outcome averages:
  - Active projects average outcome rating: 1,76; Expenditure: 31.686.379 (USD).
  - Completed projects average outcome rating: 2,77; Expenditure: 32.001.527 (USD).
  - Grand Total average outcome rating: 2,30; Expenditure: 63.687.906 (USD).
- Of the 43 projects, 23 were completed; IMF-reported average for completed projects: 2.77.
- Case study comparisons:
  - IMF self-reported average for the 12 case-study projects: 2.91 (completed subset); the evaluators’ ratings for the same projects were lower (evaluators’ exact averages reported in main report).
  - For completed case studies (n=8): IMF average 2,91; Evaluator average 2,46; Evaluator Overall Effectiveness score average 2,50.
  - For active case studies (n=4): IMF average 2,20; Evaluator average 1,88; Evaluator Overall Effectiveness score average 2,00.
- Caveats on IMF reporting:
  - Many outcome indicators and objectives were not sufficiently ‘SMART’, leading to subjective interpretation.
  - IMF often used composite scores per outcome, averaged over different countries and beneficiaries, complicating judgments about partial versus large achievement.
  - Outcomes frequently blurred outputs and outcomes; composite and non-SMART indicators likely lead to some overstatement of IMF self-reported outcome effectiveness.
- Thematic differences:
  - STA had the highest IMF-reported outcome effectiveness among thematic areas.
  - MCM had the lowest IMF-reported outcome effectiveness among thematic areas.
- Regional differences:
  - No significant difference in IMF self-reported ratings between regions for completed projects (Asia-Pacific 2,84; Eastern Europe / Caucasus 2,62; Sub-Saharan Africa 2,76; Grand Total 2,77 for completed projects).

### Impacts / overall objective achievement
- IMF overall objective ratings (43 projects):
  - Grand total average overall objective rating across all JSA projects: 2,30.
  - For completed projects (n=23) IMF reports an average overall objective (impact) rating of 2,30; IMF’s average outcome rating for the same completed projects was 2,77.
  - IMF’s own average rating for achievement of project objectives across all 23 completed JSA-supported projects during the evaluation period was 2.3.
  - Distribution for 23 completed projects: 12 projects had a rating of 3 or higher; 11 projects rated lower than 3 (only two lower than 2).
- Case-study impacts:
  - IMF average overall objective score for the 12 case studies: 2,56.
  - Evaluators’ average overall objective score for the 12 case studies: 2,17.
  - Completed projects among case studies (n=8): IMF average 2,94; Evaluators’ average 2,25.
- Specificity of objectives:
  - In 8 of the 12 case studies objectives were not defined specifically; vague formulations allowed subjective judgement in ratings.
- Conclusions on impacts:
  - IMF reporting considers that on average JSA-funded projects are only partially achieving their ultimate goals (average 2,30).
  - Evaluators generally rate impacts lower than IMF (2,17 vs 2,56 for the case-study sample).
  - Impacts are lower than average outcome achievements, suggesting Theories of Change do not always translate outcomes into impacts.

### Effectiveness — evaluators’ synthesis
- Overall conclusion:
  - IMF CD projects are between partially and largely effective.
  - IMF’s own ratings average 2.7 for all completed projects.
  - Evaluation team’s case studies measure an average effectiveness of 2.5.
  - In the 12 case studies, effectiveness was rated as 3 in 5 cases and rated at 2 or lower in 7 cases.
- Main constraint on effectiveness:
  - Institutional constraints: beneficiaries unable to (fully) absorb TA or training; intervention designs do little to address those constraints.
- Theories of Change (ToCs) quality (case study sample, n=12):
  - 5 coherent ToCs.
  - 3 ToCs with some issues.
  - 4 incoherent ToCs.
  - 8 of 12 cases have no specific goals defined.
  - 5 ToCs confuse outputs with outcomes or ill-define outcomes and objectives.
  - Only 5 cases seem to take into account institutional constraints.
- Recommendation implied: Develop coherent ToCs adapted to country-specific institutional and political constraints.

### Efficiency — evaluators’ findings
- Case-study summary on efficiency:
  - In 8 out of 12 cases there were no major efficiency concerns observed.
  - In 4 of the 12 cases there were noteworthy efficiency concerns.
- Average efficiency rating:
  - 2.67 (2.86 if weighted averages are used).
- Main efficiency issues:
  - Suboptimal selection of trainees.
  - Suboptimal duration, frequency, and timing of missions (too short, too infrequent).
  - Insufficient follow-up support.
  - Occasional use of high-cost expertise for relatively simple tasks.
- Suggested efficiency improvements:
  - Better alignment of short missions with beneficiaries’ needs.
  - Consider beneficiaries’ absorption capacity.
  - Improved follow-up support.
  - Reallocate some budgets from STX to LTX deployments where more effective.

### Attribution and sustainability
- Attribution (evaluators, 12-case sample):
  - TA made at least a difference in 3 cases.
  - TA was a critical factor in 7 cases.
  - TA was the direct cause of the observed change in 3 cases.
  - Evaluator-rated attribution average: 3 (on a 1–4 contribution scale).
- Sustainability (evaluators, case study sample, n=12):
  - Largely assured in 2 cases.
  - In all other cases only part of the results could be sustained.
  - Total average sustainability (evaluators): 2,00.
  - Main drivers of low sustainability: weak institutionalization, staff turnover, institutional weaknesses relative to task complexity.

### Additionality and donor coordination
- Evaluator average scores (case study sample, n=12):
  - Additionality: 3.3 out of 4 on average.
  - Donor coordination: 3.2 out of 4 on average.
- Interpretation:
  - JSA-supported CD projects were found to be highly additional.
  - Donor coordination was generally good and ensured complementarity with other donor projects.
  - Limitations: few cases of mutually reinforcing coordination (true synergies); donors would appreciate more sharing of IMF TA reports and information on TA results.

### TAOLAM delivery model — efficiency, effectiveness, relevance, visibility
- TAOLAM context:
  - In operation since 2012; core beneficiaries: Cambodia, Lao PDR, Myanmar, and Vietnam.
  - Current staffing: 1 Director and 5 resident advisors in TAOLAM office (support from Bank of Thailand and locally hired staff).
  - TAOLAM advisors work closely with IMF functional departments and regional partners.
- Expenditures by delivery type (expenditures through April 2017):
  - Asia, Non-TAOLAM (n=19): 26,986,223 (USD) — Average project size 1,420,328 (USD).
  - TAOLAM (n=8): 20,532,140 (USD) — Average project size 2,566,518 (USD).
  - TAOLAM-assisted (n=6): 17,758,019 (USD) — Average project size 2,959,670 (USD).
  - TAOLAM-led (n=2): 2,774,121 (USD) — Average project size 1,387,061 (USD).
  - Note: TAOLAM projects are significantly bigger in size in the sample (2,5 million USD vs. 1,4 million USD JSA Asia average as reflected in the sample).
- Efficiency:
  - TAOLAM allocates relatively more to LTX and relatively less to HQ-delivered STX (±10% less HQ delivery, ±10% more regionally-based TA delivery).
  - Operational cost trade-offs: additional office/housing/COLA costs vs lower travel costs for short-term missions.
  - Implementation cost details not available; qualitative findings show regional experts appreciated, more frequent shorter visits possible.
  - TAOLAM efficiency similar to HQ delivery on balance; RTAC could offer more economies of scale.
- Effectiveness and relevance:
  - Quantitative IMF scores show small differences: TAOLAM average overall objective scores 2.3; average outcome scores 2.2 (small sample caveat, n=8).
  - Qualitative case-study evidence indicates TAOLAM advisors provide more frequent, short visits enabling better follow-up, “hand-holding”, and adjustments to beneficiary needs (notably effective where basic advice and repeated short interactions are appropriate).
  - TAOLAM-led projects (n=2) show stronger improvements in design and re-adjustment where TAOLAM had fuller control.
  - Relevance of TAOLAM projects higher compared to HQ delivery and similar to RTAC delivery.
- Visibility of Japan:
  - Japan’s visibility is very high across the different countries and within the multilateral context.
  - The majority of projects’ direct beneficiaries are aware that the funding originates from Japan; only in three cases this was not known.
  - The delivery model (TAOLAM vs HQ vs RTAC) did not materially affect visibility of Japan.

### 5C capacity development effects (sample findings)
- 5C core capacities: C1 Commit and engage; C2 Carry out tasks; C3 Relate and attract resources; C4 Adapt and self-renew; C5 Maintain coherence.
- Key sample findings (case study beneficiary institutions, n=17; field impressions):
  - C2 — Carry out tasks:
    - 73% of Low core competencies improved.
    - 60% of Moderate core competencies improved.
    - C2 shows the most effects from IMF TA.
  - C1 — Commit and engage:
    - 71% of the “Low” category showed positive change.
    - If core competencies are moderate, only 29% improved further.
  - C3 — Relate and attract resources:
    - Where C3 is Low, IMF support tends to have a strong effect; not seen when capacity is moderate.
  - C4 — Adapt and self-renew and C5 — Maintain coherence:
    - Effects of IMF TA were much lower; improvements often require HR changes, re-organisation, or political top buy-in.
- Interpretation:
  - IMF TA strongest in improving task-carrying capacity (C2) and secondarily commitment/engagement (C1) and resource attraction (C3) where starting capacity is Low.
  - IMF TA weaker as an institutional change agent for C4 and C5 given delivery modalities and lack of routine institutional assessment.

### Visibility of Japan — detailed observations
- Visibility framework (V1–V4 across audiences D1–D4) applied in case studies.
- Summary findings:
  - The majority of projects’ direct beneficiaries are aware that IMF TA funding originates from Japan; only in three cases this was not known.
  - JSA-funded projects are seen as effective and adequate in the majority of cases.
  - The value added of Japan as a donor depends on Japanese actors/experts being deployed, JICA involvement, or Japanese trainings and scholarships — present in about half of cases.
  - Broader public in all four countries sees Japanese aid as an important contributor, though IMF TA and JSA are specialist and rarely present in mass media.
  - Visibility could be slightly improved by better “branding” (Japan mentioned on reports, debriefs) and better information sharing about which IMF projects are JSA funded.
- Country notes:
  - Myanmar: visibility at a maximum — every audience aware of funding origin.
  - Laos and Cambodia: generally high visibility but with isolated projects where funding origin was not known.
  - Cameroon: awareness lower; Japanese aid less well known compared to SE Asia.

### Recommendations (summary)
- More “design thinking”:
  - Use IMF technical expertise and country knowledge to develop comprehensive results chains and clearer Theories of Change.
  - Recognize that technical knowledge transfer alone may be insufficient; adopt tailor-made and non-standard interventions where needed.
- Operationalize institutional development:
  - Absorb existing concepts and tools for institutional assessment and development into IMF staff capabilities and processes.
  - Consider deploying institutional development experts alongside subject matter experts in some cases to improve institutional-level change, potentially more cost-effective than repeated partial TA.
- Political economy thinking:
  - Apply political economy analysis (PEA) at the design stage to better structure and operationalize political challenges.
  - Incorporate PEA findings into project ToCs to inform decisions about engagement and realistic objectives.
  - Deploy PEA expertise alongside technical experts to address frequent reasons for projects not achieving impacts.
- Cost implications:
  - Adding rigorous institutional and political economy assessments at identification stage is a relatively small expense (e.g., 20–30 person days) relative to project size and can prevent wasteful spending and improve results.
  - Incorporating institutional development and political economy into interventions should be largely budget neutral given potential offsets from reduced repetition and better targeting.
- Operational adjustments:
  - Improve trainee selection processes and match training design/delivery to participant abilities (including language).
  - Improve follow-up mechanisms after short missions; consider reallocating some STX budgets to LTX/resident or regional deployments where appropriate.
  - Improve sharing of IMF TA reports and results information with donors to foster synergies.

### Key statistics (preserved exactly)
- Total JSA-funded expenditure (review period): US$ 63,687,906.
- Portfolio: 43 projects; 89 countries.
- Thematic shares: FAD 46%, STA 25%, MCM 22%.
- Regional shares: Asia-Pacific ~75%, Africa 17%, (Eastern) Europe 8%.
- Delivery cost shares: STX and HQ missions >50%, LTX 29%, seminars/study tours 7%, project management/support 8%.
- Case study representation: 12 case studies = 37% of expenditure and 28% of projects.
- IMF outcome average (completed projects): 2.77.
- IMF case study average (completed sample): 2.91; evaluation team average for same completed case-study projects: 2.25 (case-study impacts) / 2.46 (case-study outcomes).
- IMF objectives rating across 23 completed projects: 2.3 average.
- Efficiency average rating (evaluators): 2.67 (2.86 weighted).
- Additionality score (evaluators): 3.3 out of 4.
- Donor coordination score (evaluators): 3.2 out of 4.

*Evaluation of IMF capacity development projects supported by the Japanese Subaccount (JSA), period May 1, 2013 through April 30, 2017.*

### EXECUTIVE SUMMARY ......................................................................................................

### EXECUTIVE SUMMARY

### The Japanese Subaccount (JSA) and purpose of the evaluation
- The evaluation assessed IMF capacity development projects supported by the Japanese Subaccount (JSA) in the period May 1, 2013 through April 30, 2017.
- Japan has contributed more than US$473 million since 1990 to IMF capacity development, enabling assistance to more than 120 member countries.
- Around 60 percent of the JSA CD budget is used in the Asia-Pacific region.

### Evaluation questions
- Have the JSA-supported projects been efficient, effective and relevant? And were the programmes sustainable and additional?
- What were the factors that have enhanced or detracted JSA-supported projects from reaching their objectives?
- How have the JSA-supported projects increased the visibility of Japan in the supported countries?
- How efficient and effective is the TAOLAM delivery mechanism, and has it led to more visibility of Japan?
- How effective has coordination of JSA-funded projects with other donors been?

### Methodology and sample
- Evidence sources: extensive review of IMF project documentation and data; interviews with IMF HQ staff; an electronic survey of IMF TA providers; 12 case studies in four countries (Myanmar, Laos, Cambodia, Cameroon).
- The case studies represented 37 percent of expenditure and 28 percent of the number of projects in the JSA portfolio under review.
- Portfolio under review: 43 projects across 89 countries.

### Inputs — key statistics and allocations
- Total JSA-funded expenditure, based on IMF data for the period under review, was US$ 63,687,906.
- Distribution by IMF departments (expenditure shares):
  - Fiscal Affairs Department (FAD): 46 percent.
  - Statistics Department (STA): 25 percent.
  - Monetary and Capital Markets Department (MCM): 22 percent.
- Regional allocation:
  - Asia-Pacific region: about three quarters of JSA-supported projects deployed.
  - Africa: 17 percent.
  - (Eastern) Europe: 8 percent.

### Outputs — delivery modalities and cost shares
- IMF TA is delivered primarily through short missions of 1-2 weeks by short-term experts (STX) or HQ staff.
- Expenditure shares of delivery types:
  - More than 50 percent of the expenditure is directly related to STX and HQ short missions.
  - Long-term experts (LTX): 29 percent of the cost.
  - Seminars and study tours: 7 percent.
  - Project management and support: 8 percent.
- Quality of outputs:
  - Outputs were generally and consistently very high; IMF is considered a “benchmark” and beneficiaries view its advice as “global best practice”.
  - Some TA or training activities were too complex or technically challenging for beneficiaries, owing to low absorption capacity or suboptimal trainee selection.
  - Beneficiaries often considered LTX support (regional advisors or resident advisors) more effective and ultimately more cost-effective than STX and HQ support.
  - Some TA allocations were suboptimal (TA to beneficiaries with insufficient commitment, or use of expensive experts for relatively basic TA).

### Outcomes / objectives achievement — IMF reporting and evaluators’ findings
- IMF outcome achievement scale: 1=not achieved, 2=partially achieved, 3=largely achieved, 4=fully achieved.
- Of the 43 projects in the JSA portfolio, 23 were completed and the average IMF-reported score for completed projects was 2.77.
- IMF self-reported average for the 12 case-study projects was 2.91; the evaluators’ ratings for the same projects were lower (evaluators’ exact average for the 12 case studies is reported later in the full report).
- The IMF-reported outcomes suggest outcomes were close to being “largely achieved” on average, but caveats include:
  - Many outcome indicators and objectives were not sufficiently ‘SMART’ (specific, measurable, achievable, relevant, and time-bound), making interpretation subjective.
  - IMF often used composite scores per outcome, averaged over different countries and beneficiaries, complicating judgments about partial versus large achievement.
- Thematic differences in outcome ratings:
  - STA had the highest outcome effectiveness among thematic areas.
  - MCM had the lowest outcome effectiveness among thematic areas.
- No significant difference in IMF self-reported ratings between regions.

### Key analytic observations on results reporting and attribution
- Outcome definitions frequently blurred outputs and outcomes; objectives were often not clearly defined.
- Composite and non-SMART indicators likely lead to some overstatement of IMF self-reported outcome effectiveness.
- Evaluators’ ratings were lower than IMF self-reported ratings in the case-study sample.
- Attribution and measurement challenges were identified (detailed findings and ratings presented in the main report).

### Delivery model observations (TAOLAM and delivery modalities)
- IMF TA delivery through STX/HQ short missions dominates expenditure, but beneficiaries often prefer and find LTX/resident advisor support more effective.
- TAOLAM delivery model, comparisons with RTAC and HQ models, operational and project portfolio, and cost categories (operational cost, implementation cost) are analyzed in the main report (detailed tables and comparisons presented there).
- Visibility of Japan under different delivery models (including TAOLAM) is assessed in the main report.

### Overall assessment highlights
- Outputs: high technical quality and global best-practice positioning of IMF advice; occasional mismatch between TA complexity and beneficiary capacity.
- Efficiency: a significant share of expenditure on short missions; evaluators note cases where LTX would be more cost-effective.
- Effectiveness: IMF reporting indicates outcomes close to “largely achieved” (IMF averages: 2.77 for completed projects; 2.91 for the 12 case studies as reported by IMF), but evaluators’ ratings are lower and outcome measurement limitations weaken confidence in the self-reported scores.
- Relevance and sustainability: assessed in detail in the main report (including sustainability ratings and 5C capacity development analysis for beneficiary institutions).
- Coordination and additionality: assessed in the main report (including donor coordination findings and implications for additionality).

*Source: Evaluation of IMF capacity development projects supported by the Japanese Subaccount (JSA), period May 1, 2013 through April 30, 2017.*

### 2.5 on average.

### jsa-2018 - 2.5 on average.

### Impacts / Achievement of overall project objectives
- The evaluation team found impact-level results (achievement of project objectives) to be less effective than IMF reporting.
- IMF’s own average rating for achievement of project objectives across all 23 completed JSA-supported projects during the evaluation period was 2.3.
- Rating distribution for the 23 completed projects:
  - 12 of the 23 completed projects had a rating of 3 or higher.
  - 11 projects were rated lower than 3, of which only two were lower than 2.
- For the sample of completed case studies:
  - IMF reported an average of 2.94.
  - The evaluation team rated these projects on average at 2.25.

### Assessments — Efficiency of JSA
- Case-study summary on efficiency:
  - In 8 out of 12 cases there were no major efficiency concerns observed.
  - In 4 of the 12 cases there were noteworthy efficiency concerns.
- Average efficiency rating:
  - 2.67 (2.86 if weighted averages are used).
- Observed areas for efficiency improvement:
  - Better alignment of short missions with beneficiaries’ needs.
  - Consideration of beneficiaries’ absorption capacity.
  - Improved follow-up support.
  - Reallocating some budgets from STX to LTX deployments (resident or regional) where more effective.

### Assessments — Effectiveness of JSA
- Overall effectiveness summary:
  - JSA-supported projects under review were between partially and largely effective.
  - IMF’s own ratings: 2.7 on average for all completed projects.
  - Case studies measure: 2.5 on average.
- Case-study breakdown (12 cases):
  - Effectiveness rated as 3 in 5 cases.
  - Effectiveness rated at 2 or lower in 7 cases.
- Primary reason for lower-than-expected outcomes:
  - Institutional constraints: beneficiaries unable to (fully) absorb TA or training; intervention designs do little to address constraints.
- Theories of Change (ToCs) assessment (12 cases):
  - 5 of the 12 ToCs can be considered coherent.
  - 4 are incoherent.
  - 3 have some issues.
  - 8 out of 12 cases have no specific goals defined.
  - 5 ToCs confuse outputs with outcomes or ill-define outcomes and objectives.
  - Only 5 cases seem to take into account institutional constraints.
- Recommendation from assessment:
  - Develop coherent ToCs adapted to country-specific circumstances and institutional and political constraints to improve results.

### Relevance and overall objective achievement of JSA-funded programs
- Overall project objective achievement: limited to partially achieving objectives (rating of 2 overall).
- Case studies show lower impact achievement (2.25) than IMF reporting (2.94) on completed projects.
- Relevance:
  - Projects are relevant to beneficiaries.
  - IMF TA does not typically address political constraints actively; political constraints are the main reason for partial impact achievement.
- Political constraints finding:
  - In 8 of the 12 cases, even when outcomes are achieved there is frequently insufficient political-level buy-in to use new capacity for policy decisions.
  - IMF assesses political feasibility through area departments and achieves a general “no objection” buy-in but political feasibility is not systematically assessed nor are political economy challenges incorporated into project designs.

### Attribution of JSA/IMF TA to observed changes
- Attribution findings for the 12 reviewed cases:
  - TA made at least a difference in 3 cases.
  - TA was a critical factor in 7 cases.
  - TA was the direct cause of the observed change in 3 cases.

### Sustainability of JSA support
- Sustainability assessment:
  - Largely assured in 2 of the cases.
  - In all other cases only part of the results could be sustained.
- Main reason for limited sustainability:
  - Institutional weaknesses relative to task complexity.

### Additionality and donor coordination
- Additionality rating: 3.3 out of 4 on average — JSA-supported CD projects were found to be highly additional.
- Donor coordination rating: 3.2 out of 4 on average — generally good and ensured complementarity with other donor projects.
- Limitations in donor coordination:
  - Few cases of mutually reinforcing coordination with other donors (true synergies).
  - Donors would appreciate more sharing of IMF TA reports and information on TA results.

### Specific research questions — TAOLAM delivery model
- Efficiency and effectiveness comparison:
  - Efficiency of the TAOLAM model is relatively similar to other models on balance.
  - Efficiency could be slightly improved with an RTAC model (more economies of scale possible).
  - Effectiveness of TAOLAM projects is higher than HQ delivery if regional advisors can be used; effectiveness could be somewhat improved in a larger RTAC by offering a wider bandwidth of technical expertise.
  - Relevance of TAOLAM projects is higher compared to HQ delivery and similar to RTAC delivery.
  - Visibility of Japan as donor would not be influenced by either delivery model.

### Specific research questions — Wider capacity development effects (5C methodology)
- The 5C results (sample findings):
  1) C2 — Carry out tasks:
     - 73% of Low core competencies improved.
     - 60% of Moderate core competencies improved.
     - C2 shows the most effects from IMF TA.
  2) C1 — Commit and engage:
     - In most (71%) of the “Low” category, things change to the better.
     - If core competencies are moderate, only 29% improve further.
  3) C3 — Capacity to attract resources:
     - Where C3 is Low, IMF support tends to have a strong effect (not seen when capacity is moderate).
     - IMF TA can “put beneficiaries on the map” and support arguments for more resources.
  4) C4 — Adapt and self-renew and C5 — Maintain coherence:
     - Effects of IMF TA were much lower.
     - Improvements often require HR changes, re-organisation, or political top buy-in, which did not frequently occur.
- Capacities needed:
  - To achieve more capacity development effects, IMF would require the same level of expertise and experience on institutional development as it has on monetary policy, fiscal affairs or statistics.

### Specific research questions — Visibility of Japan
- Visibility findings:
  - Japan’s visibility is very high across the different countries and within the multilateral context.
  - The majority of projects’ direct beneficiaries are aware that the funding of the IMF TA originates from Japan; only in three cases this was not known.
  - JSA-funded projects are seen as effective and adequate in the majority of cases.
  - The value added of Japan as a donor depends on factors such as Japanese actors/experts being deployed, JICA involvement as cooperating donor, or Japanese trainings and scholarships being provided — this occurs in about half of the cases.
  - Broader public in all four countries sees Japanese aid as an important contributor, though IMF TA and JSA are specialist and rarely present in mass media.
  - Visibility could be slightly improved by better “branding” (Japan mentioned on reports, debriefs) and better information sharing about which IMF projects are JSA funded.

### Recommendations
- More “design thinking”:
  - Use IMF technical expertise and country knowledge to develop comprehensive results chains and clearer theories of change.
  - Recognize that technical knowledge transfer alone may be insufficient; adopt tailor-made and non-standard interventions where needed.
- Operationalize institutional development:
  - Absorb existing concepts and tools for institutional assessment and development into IMF staff capabilities and processes.
  - Consider deploying institutional development experts alongside subject matter experts in some cases to improve institutional-level change, which may be more cost-effective than repeated partial TA.
- Political economy thinking:
  - Apply political economy analysis (PEA) at the design stage to better structure and operationalize political challenges.
  - Incorporate PEA findings into project ToCs to inform decisions about engagement and realistic objectives.
  - Deploy PEA expertise alongside technical experts to address the most frequent reasons for projects not achieving impacts.

*External Independent Evaluation of the Japanese Sub-account (JSA) at the IMF.*

### 1. Geographic distribution – representative

### 1. Geographic distribution – representative

### Geographic distribution: budgets and expenditures
- Selection based on available budget information; evaluators allocated budgets by approximation to countries and project. The estimated budget is higher than expenditure obtained in the review (USD 63,687,906).
- Total budget in review period (approximate, paragraph 3.2.1 assumptions):
  - Asia-Pacific: 72,112,949 (69%), # of countries in case study sample: 3
  - Eastern Europe / Caucasus: 12,749,526 (12%), # of countries in case study sample: 1
  - Sub-Saharan Africa: 19,714,087 (19%)
  - Total: 104,576,561 (100%), # of countries in case study sample: 4
- Country selection (highest estimated expenditures):
  1. Myanmar (largest expenditure in Asia)
  2. Lao (2nd largest expenditure in Asia)
  3. Cambodia (3rd largest expenditure in Asia)
  4. Cameroon (2nd largest expenditure in SSA/EE)
- Note: Many IMF projects are multi-country projects and use “composite” scorings for outcomes and impacts; calculation method for composite scores is not documented and cannot be validated.

### Thematic distribution: budgets and case study coverage
- Thematic distribution of JSA funded programs (budget in review period, approximate):
  - Fiscal Policy and Management (FAD): 16 programs, 40,579,033 (39%), # of programs in case study sample: 5
  - Monetary Policy and Financial Systems (MCM): 11 programs, 25,755,640 (25%), # of programs in case study sample: 3
  - Macroeconomic and Financial Statistics (STA): 10 programs, 23,296,756 (22%), # of programs in case study sample: 3
  - Institute for Capacity Development (ICD): 4 programs, 13,148,594 (13%), # of programs in case study sample: 1
  - Legislative Frameworks (LEG): 2 programs, 1,796,539 (2%), # of programs in case study sample: 0
  - Total: 43 programs, 104,576,561 (100%), # of programs in case study sample: 12
- Case study program selection aimed for distribution: 5 FAD, 3 MCM, 3 STA, 1 ICD (reflects distribution of capacity building budgets across functional departments).

### Status and size preferences in sample selection
- Sample biased toward larger expenditures and towards completed or almost completed programs to increase evaluability.
- In total, JSA funded 43 programs in 89 countries in the review period, with total budget USD 104,576,561.
- The sample contains more completed or finished programs than the total research population: 2 active, 10 completed in the sample (to increase evaluability).

### Case study sample — projects and expenditures (actual expenditures through April 2017, IMF data)
- Final case study sample (12 projects), expenditures sum to 23.851.557 (USD) as presented in Table 3 total.
- Selected project-level expenditures (through April 2017, USD, IMF data):
  - Developing Macroeconomic Management Capacity in Myanmar and Lao People's Democratic Republic (Myanmar) — Completed 2014, ICD, 1/Jul/13–30/Jun/15: 1.415.337
  - Central Bank Modernization in the Union of Myanmar (Myanmar) — Completed 2013, MCM, 1/May/12–30/Sep/15: 2.210.607
  - External Sector Statistics Resident Advisor to Lao People’s Democratic Republic and Myanmar (Myanmar) — Active 2014, STA, 1/Feb/14–31/Jul/17: 1.349.795
  - Developing Treasury Management and Financial Systems Modernization in Myanmar and Lao People's Democratic Republic (Laos) — Active 2013, MCM, 1/Mar/13–30/Jun/17: 4.321.068
  - External Sector Statistics Resident Advisor to Lao People’s Democratic Republic and Myanmar (Laos) — Active 2014, STA, 1/Feb/14–31/Jul/17: 1.349.795
  - Developing Macroeconomic Management Capacity in Myanmar and Lao People's Democratic Republic (Laos) — Completed 2014, ICD, 1/Jul/13–30/Jun/15: 1.415.337
  - Effective and Efficient Budget and Treasury Management for Southeast Asia (Cambodia) — Completed 2012, FAD, 1/Oct/11–30/Apr/16: 2.719.699
  - Regional Government Finance Statistics (Cambodia) — Completed 2012, STA, 1/Nov/11–30/Nov/15: 3.093.011
  - Strengthening Financial Stability Framework (Cambodia) — Active 2014, MCM, 1/May/13–5/Jan/18: 257.833
  - Implementing Tax Administration Reforms in Selected South East Asian Countries (Cambodia) — Completed 2012, FAD, 1/Aug/11–30/Jun/15: 2.716.308
  - Strengthening Budget Management and Customs Administration in the CEMAC (Cameroon) — Completed 2012, FAD, 1/May/11–30/Apr/16: 2.120.253
  - Economic and Monetary Community of Central African States (CEMAC): Strengthening Regional Financial Agencies (Cameroon) — Completed 2010, MCM, 1/Sep/09–30/Jul/15: 882.513
- Note: Actual expenditures per country are estimates where total project expenditures have been divided by the number of countries; this may not reflect actual intensity of TA operation in a given country.

### Research limitations (implications for representativeness and evaluability)
- Limitations on sample representativeness:
  - Sample is stratified and deliberately biased toward larger and completed programs.
  - Review allowed visits to only 4 countries while total number of countries was 89; many countries arise from a few programs with small budgets active across many countries (e.g., financial stability indicators in 48 countries), reducing bias magnitude.
- Evaluability varied by case/country:
  - Myanmar and Cambodia: authorities and beneficiaries were fully accessible; sufficient interviews.
  - Laos and Cameroon: authorities more reserved; fewer relevant staff interviewed; margin of error exists.
  - Case studies are not forensic audits but cross-checks of IMF documentation and triangulation with interviews/field visits.
- Survey among IMF experts/staff is not triangulated and cannot be used for accountability evidence; it supports learning aspects.

### Inputs — portfolio-level expenditures (review period, FY14–FY17: 1 May 2013–30 April 2017)
- Total expenditure based on IMF data for review period: USD 63,687,906 across 43 projects.
- Of total expenditure, USD 26.765.360 cannot be allocated to a specific country (includes HQ staff mission costs, staff time for backstopping/project management, seminar costs, language services).
- Thematic expenditure (allocated expenditures through April 2017, USD, and % of total; expenditures not specifically linked to a beneficiary country):
  - FAD: 29.292.896 (46%), of which 11.527.714 not country-linked
  - STA: 16.022.496 (25%), of which 8.799.385 not country-linked
  - MCM: 14.124.760 (22%), of which 3.239.228 not country-linked
  - ICD: 2.774.121 (4%), of which 1.926.984 not country-linked
  - LEG: 1.473.633 (2%), of which 1.272.050 not country-linked
  - Grand Total: 63.687.906 (100%), of which 26.765.360 not country-linked
- Geographic expenditure distribution (allocated expenditure through April 2017, USD, IMF data):
  - Asia-Pacific: 47.518.363 (75%)
  - Eastern Europe / Caucasus: 5.304.779 (8%)
  - Sub-Saharan Africa: 10.864.764 (17%)
  - Grand Total: 63.687.906 (100%)
- Status distribution (allocated expenditure through April 2017, USD; number of projects):
  - Active: 31.686.379 (49,75%), 20 projects
  - Completed: 32.001.527 (50,25%), 23 projects
  - Grand Total: 63.687.906 (100,00%), 43 projects

### Inputs — case study sample expenditures
- Case study sample: 12 case studies with 4 active and 8 completed projects, covering expenditure USD 23.851.557 (37% of expenditure and 28% of projects in portfolio under review).
  - Active projects account for 31% of the sample expenditure; completed projects account for 69%.
- Case study thematic expenditure through April 2017 (IMF data):
  - FAD: 7.556.260 (32%)
  - ICD: 2.830.675 (12%)
  - MCM: 7.672.021 (32%)
  - STA: 5.792.602 (24%)
  - Grand Total: 23.851.557 (100%)
- Regional distribution in sample: 10 cases in Asia, 2 in Africa; 87% of sample expenditures are in Asia. Projects are active in 40 countries and on average in more than 3 countries in the region.

### Outputs — delivery modalities and expenditure by output
- Portfolio-level output categories (expenditure, USD, % of total):
  - Short-term missions (STX): 21.747.063 (34,1%)
  - Long-term deployments (LTX): 18.735.797 (29,4%)
  - Short-term missions, by HQ staff (HQ): 13.461.593 (21,1%)
  - Seminars & study tours: 4.611.810 (7,2%)
  - Project backstopping: 2.348.294 (3,7%)
  - Project management: 1.656.924 (2,6%)
  - Language services: 881.403 (1,4%)
  - Local support: 219.971 (0,3%)
  - Miscellaneous: 25.051 (0,0%)
  - Total: 63.687.906 (100%)
- Case study sample output categories (expenditure, USD, % of total):
  - Short-term missions (STX): 6.631.057 (27,8%)
  - Long-term deployments (LTX): 9.940.016 (41,7%)
  - Short-term missions, by HQ staff (HQ): 3.405.704 (14,3%)
  - Seminars & study tours: 2.099.897 (8,8%)
  - Project backstopping: 958.526 (4,0%)
  - Project management: 503.524 (2,1%)
  - Language services: 135.475 (0,6%)
  - Local support: 161.528 (0,7%)
  - Miscellaneous: 15.830 (0,1%)
  - Total: 23.851.557 (100%)

### Output quality, allocation of expertise, and follow-up
- General finding on technical quality:
  - IMF TA delivery and deployed experts are generally assessed as high quality.
  - In areas where other donors are active, IMF TA is often considered the “best” and a “benchmark”; IMF advice seen as “global best practice”.
- Observations on short-term missions (STX) and long-term deployments (LTX):
  - STX/HQ missions (typically 1–2 weeks) are:
    - Sometimes supply driven; ToRs occasionally not shared beforehand; missions can be surprising to beneficiaries.
    - Often too short for complex content to be fully absorbed; require follow-up by same expert, which is often not possible unless a regional expert (e.g., TAOLAM) is available.
    - At times high-quality expertise used for less complicated tasks that could have been done with cheaper local resources.
  - LTX support is consistently valued; absence of LTX is a frequent reason for lower-than-expected effectiveness.
  - In the sample, LTX support is higher than portfolio average but concerns about insufficient LTX remain, suggesting a wider issue.
- Survey evidence (of 38 respondents):
  - 28 noted insufficient follow-up support.
  - 15 noted insufficient quantity of TA delivered.
- Evaluators cannot fully verify whether specific allocations of expertise could have been improved except in clear cases; observation suggests need for more attention to allocation choices.

### Trainee selection and training effectiveness
- Some off-site training is intensive and responsive to beneficiary needs; however, in at least three cases trainees were nominated by beneficiary institutions rather than IMF.
- Observed selection criteria by institutions sometimes prioritize internal HR/organizational concerns over qualification and need, leading to:
  - Trainees with insufficient English language skills or no prior education being sent to high-level trainings with limited effect.
- Survey responses identified key factors contributing to lower than expected effectiveness:
  1. Nomination and selection process of training participants: ensuring participants satisfy minimum requirements for the course.
  2. Design of training programs: extent to which design considered abilities and training needs of participants.
  3. Delivery of training programs: extent to which delivery considered abilities (including language abilities) and training needs of participants.
- Additional comments from 5 respondents on limited training effects:
  - “Lack of implementation of the knowledge gained from the training in the actual data compilation”
  - “Participants may be nominated on basis of rank from departments that are not working on the subject matter.”
- Conclusion: Improvements in selecting appropriate trainees and matching them to suitable training design and delivery appear feasible.

*Source: IMF evaluation of JSA-funded capacity development (FY14–FY17) by IMF.*

### 2.3 Outcomes

### 2.3 Outcomes

### Outcomes of the JSA portfolio under review
- For all 43 projects, outcome achievement ratings were manually collected and averaged across typically more than 3 outcomes per project.
- IMF reported averages (1=not achieved, 2=partially achieved, 3=largely achieved, 4=fully achieved):
  - Active projects average outcome rating: 1,76; Expenditure through April 2017 (IMF data): 31.686.379
  - Completed projects average outcome rating: 2,77; Expenditure through April 2017 (IMF data): 32.001.527
  - Grand Total average outcome rating: 2,30; Grand Total expenditure through April 2017 (IMF data): 63.687.906
- Selected project-level IMF average outcome ratings and expenditures (excerpt from the portfolio):
  - Banking Supervision and Support to the Reform of the State Owned Banks in Myanmar: 1,43; 689.827
  - Banking Supervision in ASEAN for Financial Stability: 1,67; 3.377.324
  - Developing Macroeconomic Management Capacity in CMLV Countries: 2,75; 1.358.784
  - Developing Treasury Management and Financial Systems Modernization in Myanmar and Lao People's Democratic Republic: 1,15; 4.321.068
  - Enhanced Data Dissemination in Countries in the Asia-Pacific Region: 1,33; 1.116.211
  - Financial Soundness Indicators: 2,33; 2.272.442
  - Promoting Priority PFM Reforms in Selected Asian Countries, with a Special Focus on Myanmar: 1,80; 4.401.731
  - Strengthening Macroeconomic Management in the Asia-Pacific Region (completed entry): 4,00; -
  - Improved External Sector Statistics in Asia Pacific region: 3,67; 3.825.284
  - Strengthening Budget Management and Customs Administration in the CEMAC: 3,29; 2.120.253
- Thematic differences in IMF reported outcome achievement for COMPLETED JSA projects (n=43) (Table 11):
  - FAD average of IMF outcome (objective) rating: 2,59; Expenditure through April 2017 (IMF data): 15.776.946
  - ICD average of IMF outcome (objective) rating: 3,88; Expenditure through April 2017 (IMF data): 1.415.337
  - LEG average of IMF outcome (objective) rating: 2,75; Expenditure through April 2017 (IMF data): 800.941
  - MCM average of IMF outcome (objective) rating: 2,08; Expenditure through April 2017 (IMF data): 4.993.514
  - STA average of IMF outcome (objective) rating: 3,25; Expenditure through April 2017 (IMF data): 9.014.790
  - Total Average: 2,77; Total Expenditure: 32.001.527
- Regional differences for COMPLETED JSA projects (n=43) (Table 12):
  - Asia-Pacific average of IMF outcome (objective) rating: 2,84; Expenditure through April 2017 (IMF data): 21.989.685
  - Eastern Europe / Caucasus average of IMF outcome (objective) rating: 2,62; Expenditure through April 2017 (IMF data): 5.304.779
  - Sub-Saharan Africa average of IMF outcome (objective) rating: 2,76; Expenditure through April 2017 (IMF data): 4.707.063
  - Grand Total: 2,77; 32.001.527
- Outcome priority distribution and average ratings (all JSA projects, n=43) (Table 13):
  - High: Average of IMF outcome (objective) rating 2,45; # of outcomes 226
  - Low: Average 1,00; # of outcomes 1
  - Medium: Average 2,07; # of outcomes 71
  - n/a: Average 1,68; # of outcomes 54
  - Other: Average 2,00; # of outcomes 10
  - Total Average: 2,24; Total # of outcomes 362
- Cautions on IMF-reported outcome ratings (summarized):
  - Most outcomes are not formulated as SMART objectives; distinguishing between ratings 2 and 3 is often subjective.
  - Nine out of 12 case study projects use composite scores across multiple countries; composite calculation methods are not transparent.
  - Outcomes are frequently outputs (e.g., number trained) and therefore more likely to be marked achieved.
  - Indicators for outcomes (or impacts) are sometimes the IMF output itself, so delivery of the output implies achievement of the outcome.
  - The practical meaning of ratings 1, 2, and 3 is unclear; "not achieved" (1) is rare in completed projects.

### Outcomes of the case study sample
- Methodological approach in case studies:
  - Evaluators assessed each stated outcome (objectives) in the country visited only (no composite scoring across countries) and rated using IMF’s 1–4 scale.
  - Evaluators also assigned an "overall" effectiveness score (which IMF reporting does not provide), taking relative importance of outcomes into account.
- COMPLETED projects, case study sample (n=8) — average outcome (objective) scores (Table 14):
  - Budget and Treasury Management: IMF average 3,00; Evaluator average 3,00; Overall Effectiveness score (Evaluator) 3,00
  - Budget Management and Customs Administration: IMF average 3,29; Evaluator average 2,71; Overall Effectiveness score (Evaluator) 2,00
  - Central Bank Modernization: IMF average 2,00; Evaluator average 2,00; Overall Effectiveness score (Evaluator) 2,00
  - Macroeconomic Management Capacity/Laos: IMF average 3,75; Evaluator average 3,25; Overall Effectiveness score (Evaluator) 3,00
  - Macroeconomic Management Capacity/Myanmar: IMF average 3,75; Evaluator average 2,25; Overall Effectiveness score (Evaluator) 3,00
  - Regional Financial Agencies: IMF average 2,33; Evaluator average 2,22; Overall Effectiveness score (Evaluator) 2,00
  - Regional Government Finance Statistics: IMF average 2,90; Evaluator average 2,00; Overall Effectiveness score (Evaluator) 2,00
  - Tax Administration Reforms: IMF average 2,25; Evaluator average 2,25; Overall Effectiveness score (Evaluator) 3,00
  - Total average for COMPLETED case study projects: IMF average 2,91; Evaluator average 2,46; Evaluator Overall Effectiveness score average 2,50
- ACTIVE projects, case study sample (n=4) — average outcome (objective) scores (Table 15):
  - Strengthening Financial Stability Framework: IMF average 1,80; Evaluator average 3,00; Overall Effectiveness score (Evaluator) 3,00
  - Treasury Management and Financial Systems Modernization: IMF average 1,50; Evaluator average 1,50; Overall Effectiveness score (Evaluator) 2,00
  - External Sector Statistics/Myanmar: IMF average 3,25; Evaluator average 1,50; Overall Effectiveness score (Evaluator) 2,00
  - External Sector Statistics/Laos: IMF average 2,25; Evaluator average 1,50; Overall Effectiveness score (Evaluator) 1,00
  - Total average for ACTIVE case study projects: IMF average 2,20; Evaluator average 1,88; Evaluator Overall Effectiveness score average 2,00
- Comparison of IMF reporting and evaluators’ ratings:
  - For the case study sample, evaluators’ outcome ratings are on average lower than IMF’s: evaluators average 2,5 versus IMF average close to 3 (2,91 for completed sample).
  - Overall difference in outcome ratings between IMF and evaluators across the sample: 11% (13% if a weighted average is used).
- Summary conclusion from the evaluation:
  - IMF-reported outcomes suggest completed projects are typically close to "largely achieved" with an average of 2,7.
  - Evaluator assessments in the case studies are lower, with an average of 2,5, interpreted as between partially and largely achieved.
  - Given non-SMART outcomes, composite scoring, and other reporting practices, the numerical ratings remain multi-interpretable despite not necessarily being inaccurate.

*Source: IMF evaluation report (jsa-2018), Section 2.3 Outcomes.*

### 2.4 Impacts/overall objective achievement

### 2.4 Impacts/overall objective achievement

### Impacts — key findings
- Impacts (project objectives) can have one or more ratings per project; many projects include two or more different interventions packed into one project.  
- Objective scores in IMF reporting are frequently composite scores, referring to different countries with different results on impact level.  
- Few objectives are measurable given the formulations, making clear-cut judgements difficult or impossible.  
- Impact is defined here as the results achieved on a goal level in an intervention; the goal level in an IMF program is the overall objective (first level impacts), with higher-level impacts such as GDP growth or poverty reduction not defined in program documents. The review therefore considered the overall impact as the first level impact.

### Overall objective ratings (IMF reporting)
- Based on 43 projects in the JSA portfolio:
  - For completed projects (n=23), IMF reports an average overall objective (impact) rating of 2,30 (close to partially achieved).
  - For the same completed projects, the average outcome rating was 2,77.
  - The most common IMF objective rating is 3 – largely achieved: 12 of the 23 completed projects have a rating of 3 or higher; 11 projects are rated lower; only two are lower than 2 (on average).
- Active projects’ average rating (1,82) reflects elapsed time since project start and aligns with the average outcome rating IMF gives these projects (1,80 on average) until projects near completion.
- Grand total average overall objective rating across all JSA projects (n=43): 2,30.

### Case studies — IMF vs evaluators
- Case study sample (n=12) findings:
  - IMF average overall objective score for case studies: 2,56.
  - Evaluators’ average overall objective score for case studies: 2,17.
  - Completed projects among case studies (n=8):
    - IMF average for completed case-study projects: 2,94 (considered “largely achieved”).
    - Evaluators’ average for completed case-study projects: 2,25 (close to partially achieved).
    - Of the 8 completed case-study projects, 6 had an IMF rating of 3 or higher; evaluators found only two projects qualified for a 3 and none higher.
- Specific IMF and evaluator scores for case-study projects (selected examples):
  - Active projects IMF/Evaluator averages: 1,80 / 2,00.
  - External Sector Statistics/Laos: 2,00 / 1,00.
  - Strengthening Financial Stability Framework: 1,80 / 3,00.
  - Budget and Treasury Management (completed): 3,00 / 3,00.
  - Tax Administration Reforms (completed): 3,33 / 2,00.
  - Total average (case studies): IMF 2,56; Evaluators 2,17.

### Specificity of objective formulations (case studies, n=12)
- In 8 of the 12 case studies the objectives were not defined specifically (Yes/No specificity indicator):
  - Projects with non-specific objective formulations (examples):
    - Macroeconomic Management Capacity/Myanmar: N — "improve effectiveness of local capacity..."
    - Central Bank Modernization: N — "establish a new framework; strengthen capacity..."
    - External Sector Statistics/Myanmar: N — "availability of accurate data..."
    - Treasury Management and Financial Systems Modernization: N — "increased capacity..."
    - Budget Management and Customs Administration: N — "sound PFM...," "vision strengthened..."
  - Projects with specific objective definitions:
    - Regional Government Finance Statistics: Y
    - Strengthening Financial Stability Framework: Y
    - Tax Administration Reforms: Y
    - Regional Financial Agencies: Y
- Implication: Vague objective wording ("improve", "strengthen", "sound", "strengthened vision of") allows subjective judgement in assigning ratings; small changes can be interpreted as 1 (not achieved), 2 (partially achieved), or 3 (largely achieved). In several cases the indicator for impact achievement is an IMF-generated output (e.g., trained staff), creating tautological measures of impact.

### Conclusions on impacts
- IMF reporting considers that on average JSA-funded projects are only partially achieving their ultimate goals (average 2,30).  
- The evaluators generally rate impacts lower than IMF (2,17 vs 2,56 for the case-study sample).  
- Impacts are lower than average outcome achievements, suggesting theories of change do not always translate outcomes into impacts.  
- Lack of specificity in objectives makes interpretation multi-interpretable — what constitutes partially or largely achieved remains an opinion.

### Assessments — Efficiency (evaluators’ findings)
- Overall statement: IMF CD projects are, on average (case-study basis), largely efficient; in 8 of 12 cases there are no major efficiency concerns; in 4 of 12 cases there are efficiency concerns that deserve attention.
- Efficiency definition: value of outcomes/benefits (outputs) relative to inputs/costs incurred to achieve them.
- Aggregate IMF efficiency ratings are not available; the evaluators assessed efficiency for each case study.
- Table of expenditures through April 2017 (IMF data) and evaluators’ average efficiency score (scale 1–4):
  - Budget and Treasury Management: 2.719.699 USD — 2,00
  - Budget Management and Customs Administration: 2.120.253 USD — 3,00
  - Central Bank Modernization: 2.210.607 USD — 3,00
  - External Sector Statistics/Laos: 1.349.795 USD — 3,00
  - External Sector Statistics/Myanmar: 1.349.795 USD — 3,00
  - Macroeconomic Management Capacity/Laos: 1.415.337 USD — 2,00
  - Macroeconomic Management Capacity/Myanmar: 1.415.337 USD — 2,00
  - Regional Financial Agencies: 882.513 USD — 3,00
  - Regional Government Finance Statistics: 3.093.011 USD — 3,00
  - Strengthening Financial Stability Framework: 257.833 USD — 1,00
  - Tax Administration Reforms: 2.716.308 USD — 4,00
  - Treasury Management and Financial Systems Modernization: 4.321.068 USD — 3,00
  - Grand Total expenditure: 23.851.557 USD — Grand Total average efficiency score 2,67
- Evaluators’ 1–4 efficiency scale:
  - (1) inefficient (major inefficiencies noted)
  - (2) significant inefficiencies noted
  - (3) minor inefficiencies noted
  - (4) efficient (no issues noted)
- Average efficiency rating: 2,67 (2,86 if weighted averages are used), interpretable as minor efficiency issues noted.

### Main reasons for efficiency ratings (case-study examples)
- Suboptimal selection of trainees; training on basic topics given by expensive LTX noted in multiple projects (e.g., Macroeconomic Management Capacity/Myanmar and /Laos).  
- Suboptimal selection of trainees and LTX location issues (e.g., LTX based in Thailand but mostly active in Myanmar).  
- Mismatch between TA expectations, needs, and perceived quality of TA implementation by recipients (Treasury Management and Financial Systems Modernization).  
- Insufficient follow-up and insufficient understanding of the context of short missions (Budget and Treasury Management).  
- Inefficient timing of TA missions and limited consideration of absorption capacity of trainees (Strengthening Financial Stability Framework).  
- Supply-driven TA on occasion and insufficient coordination with beneficiaries (Budget Management and Customs Administration).  
- Underspent budgets where more LTX support might have been useful (Regional Financial Agencies).  
- Tax Administration Reforms: no major efficiency issues noted.

### Efficiency implications and recommendations (evaluators’ synthesis)
- Allocation of CD resources is not always optimal; recurrent issues include:
  - Suboptimal selection of trainees (not sufficiently based on needs assessments; no systematic testing of participants; insufficient control over selection criteria used by authorities).
  - Suboptimal duration, frequency, and timing of missions (too short, too infrequent, insufficient follow-up).
- Suggested improvements:
  - Better alignment of TA and training with beneficiaries’ needs.
  - Greater consideration of absorption capacity when designing interventions.
  - Improved follow-up support after missions.
  - Reallocating some budgets spent on less effective STX to LTX deployments (resident or regional) where appropriate.

*Source: IMF JSA 2018 — 2.4 Impacts/overall objective achievement; assessments and efficiency ratings reported by the evaluators.*

### 3.2 Effectiveness

### 3.2 Effectiveness

### Overall conclusion on effectiveness
- The evaluation concludes that IMF CD projects are between partially and largely effective.
- The IMF’s own ratings average 2.7 for all completed projects, suggesting projects are closer to “largely effective”.
- The evaluation team’s case studies measure an average effectiveness of 2.5, exactly between partially and largely effective.
- In the 12 case studies, effectiveness was rated as 3 in 5 cases, and rated at 2 or lower in 7 cases.
- The main reason for lower-than-expected outcomes is institutional: beneficiaries are unable to (fully) absorb the TA or training, and intervention designs do not sufficiently address that.

### Quality of the Theories of Change (ToCs)
- For each project the evaluation team derived the implicit Theory of Change (ToC) and assessed it on: (a) overall coherence, (b) clear definition of outputs, outcomes and impacts, (c) consideration of institutional context, and (d) consideration of political constraints.
- Summary findings from the case study sample (n=12):
  - 5 of the 12 ToCs are coherent.
  - 3 of the ToCs have some issues.
  - 4 of the ToCs are incoherent.
  - 8 out of 12 cases have no specific goals defined.
  - 5 ToCs confuse outputs with outcomes, or do not clearly define (and sometimes mix) outcomes and objectives.
  - Only 5 cases seem to take into account institutional constraints.
  - 6 cases consider political constraints (to some extent) in the design of the project.
- The projects rated “largely relevant” (relevance score of 3) all have ToCs that are (a) coherent, and (b) have considered institutional and political constraints adequately.
- The evaluation team found that the quality of the (documented) Theories of Change can be improved.
- The transition to Results Based Management (RBM) is expected to be a step forward in improving ToC quality.

### Institutional and political constraints
- All except two of the 12 cases have ToCs designed for multiple countries, despite differing institutional and political contexts across countries.
- The 12 case studies did not include an example where institutional (absorption) constraints were addressed with measures other than “training”.
- The four coherent strategies are all in Cambodia, where absorption capacity is highest.
- In countries where absorptive capacity is typically a constraint (Myanmar, Lao, and Cameroon), there was typically no clear ToC strategy on how to address this.
- Best practice among other donors: conduct a deep institutional assessment ex ante to gauge feasibility of knowledge transfer and identify institutional constraints (leadership buy-in, organization, HR motivation).
- Following assessment, develop an institutional development strategy that identifies fundamental changes needed (e.g., ability to attract and retain highly qualified staff) and devise strategies to achieve them (e.g., leadership engagement, organizational change plans, use of other donor resources).
- If IMF TA cannot address crucial constraints, two choices are proposed: (1) downgrade objectives, or (2) consider dropping the project.
- IMF advisors often attempt to address constraints “on the fly”; if outcomes are difficult, TA may be reduced or reallocated to other countries in the program.
- Survey and interview evidence:
  - Institutional and political constraints are seen as major issues by 26 of 41 respondents.
  - 26 of 40 respondents deem political constraints important or most important factors for success.
  - Only 7 of 40 would attribute TA constraints as a cause for limited effectiveness.
  - Respondents (nearly all TA providers) often do not see the link between intervention design and institutional/political constraints.
- The evaluation team found political and institutional constraints are often treated as a “given” external constraint rather than actively addressed through project design.
- Recommendation implicit in findings: develop coherent ToCs adapted to each country’s institutional and political constraints.

### Effectiveness ratings (evaluators vs IMF documentation)
- The evaluation team’s overall effectiveness rating across the 12 case studies averages 2.33.
- The IMF’s average outcome rating for the same sample is 2.73.
- The difference is minus 0.40 points on a scale of 1-4 (minus 14%).
- Main reasons listed by evaluators for effectiveness ratings (case study sample, n=12) include:
  - Institutional constraints and limited absorption capacity (7 of 12 cases).
  - Political constraints preventing adoption of reforms.
  - Mismatch between TA needs and delivery.
  - Resource limitations and selection/suboptimal selection of trainees.
  - Delays that nonetheless led to eventual achievement in some cases.
- Selected case comparisons (Effectiveness score by evaluators / IMF average / Difference / Primary reason):
  - Macroeconomic Management Capacity/Myanmar: 3 / 4 / -1 — Highest level outcome not achieved, limited absorption capacity.
  - External Sector Statistics/Laos: 1 / 2 / -1 — Absorption capacity too limited; political constraints; bureaucratic hurdles.
  - Strengthening Financial Stability Framework: 3 / 2 / +1 — Delays, but ultimately achieved most outcomes.
  - Tax Administration Reforms: 3 / 3.3 / -0.3 — Political constraints prevented adoption of key reforms.
- Total average effectiveness: evaluators 2.33; IMF 2.73; difference -0.40.

### Relevance and overall objective achievement
- On average, final objectives were “partially achieved” (3) in the 12 case studies.
- Case studies show a lower overall objective achievement average of 2.25 than the IMF’s own rating for all completed JSA projects (2.94).
- Projects are generally relevant to beneficiaries, but IMF TA does not typically address political constraints actively, which are the main reason for partial impact achievement.
- The evaluation distinguishes relevance (extent CD activities served important objectives of beneficiary countries) from overall objective achievement (positive and negative changes brought about by CD activity).
- The evaluation team rated impact achievement (overall objectives) lower than IMF ratings by 0.5 points on average (evaluators 2.17; IMF 2.67; difference -0.50).
- The most cited reason for not fully achieving impact: insufficient buy-in at the political level to use new capacity (8 of 12 cases).
- Reasons for lower overall objective achievement include: political economy constraints not considered, insufficient attention to macroeconomic policy relevance, severe institutional constraints, lack of resources devoted to the most relevant theme/agency, and no clear change-management strategy to address political economy constraints.
- Examples of case-specific overall objective scores and reasons:
  - Macroeconomic Management Capacity/Myanmar: overall objective score 2 (Evaluator) vs 3 (IMF) — Political economy constraints not considered; insufficient attention to macroeconomic policy relevance.
  - External Sector Statistics/Laos: overall objective score 1 vs 2 — Resources not devoted to the most relevant theme/agency; no efforts to address political economy constraints.
  - Budget Management and Customs Administration: overall objective score 2 vs 4 — Changes occurred but have not been implemented yet; no political priority at the time.
- Survey findings on likely impacts reported by respondents:
  - Nearly 80% of respondents believe projects typically (with probability of at least 60%) lead to a detectable increase in knowledge/skills of individual participants.
  - Around 1/3 of respondents does not believe TA results lead to changes in the daily work of individual participants.
  - 63% believe there is a good chance (60% likelihood) that improvement at the individual level leads to change at the institutional level.
  - Only 40% believe that change at the institutional level leads to changes in macro-economic policy making.

### Relevance and political constraints in project lifecycle
- IMF CD projects are generally aligned with recipient authorities’ priorities; projects are not started where there would be outright disagreement upfront.
- Political constraints are considered in early identification phases; area departments, mission chiefs, and Resident Representatives are well aware of political feasibility.
- Typical pattern: obtain a general “no objection” buy-in and engage; later lack of full buy-in may emerge, and TA may continue at reduced scale in the hope of future political change.
- Political economy analyses (PEAs) are not formally conducted in the cases reviewed; PEAs could inform motivations and strategies to alter political behavior or prompt decisions to set less ambitious goals or not engage.
- Risk: continuing TA without addressing political constraints can result in no impact or impact dependent on external political change.

*The evaluation team, JSA Annual Report 2018 (IMF).*

### 3.4 Attribution

### 3.4 Attribution

### Attribution of IMF CD projects
- The evaluation finds that the attribution of IMF CD projects to the observed changes is high: in the reviewed cases, the IMF "made at least a difference" in 3 cases, "was a critical factor" in 7 cases, and "was in fact the direct cause" of the observed change in 3 of the 12 cases.
- The evaluation assessed attribution by means of a contribution analysis and used a four-level scale: 1 = no contribution, 2 = made some difference, 3 = critical factor, 4 = direct causal link.
- Evaluator-rated attribution levels for the 12-case sample:
  - Macroeconomic Management Capacity/Myanmar — 3
  - Central Bank Modernization — 3
  - External Sector Statistics/Myanmar — 4
  - Treasury Management and Financial Systems Modernization — 2
  - External Sector Statistics/Laos — 2
  - Macroeconomic Management Capacity/Laos — 3
  - Budget and Treasury Management — 4
  - Regional Government Finance Statistics — 3
  - Strengthening Financial Stability Framework — 3
  - Tax Administration Reforms — 3
  - Budget Management and Customs Administration — 3
  - Regional Financial Agencies — 4
- Total average attribution (evaluator rating) — 3
- Interpretation: IMF TA is contributing to the changes observed and is at least a critical factor in 10 of the 12 cases. The assessment would be negative only in case there is no contribution; in all other cases the TA was usefully deployed. With (some) low capacity beneficiaries the contribution is naturally higher; in other cases the contribution is lower, but never absent.

### Methodological note on attribution
- The CEF of IMF uses “the positive and negative changes brought about by CD activity, compared to the most likely counterfactual” as the definition of attribution; the evaluation notes that this ex ante counterfactual definition is not feasible as an objective ex post criterion because a factual counterfactual cannot be observed or triangulated in this context.

### Sustainability (related finding)
- The evaluation finds limited sustainability of CD projects: in 2 of the 12 cases sustainability is largely assured; in all other cases prospects are that only part of the results can be sustained.
- The IMF does not rate sustainability in its M&E system; this conclusion draws on the case studies and the survey.
- Evaluator-rated sustainability ratings and main reasons (case study sample, n=12):
  - Macroeconomic Management Capacity/Myanmar — 1 — Not institutionalized, dependent on continued support by IMF experts
  - Central Bank Modernization — 2 — Not institutionalized, dependent on continued support by IMF experts
  - External Sector Statistics/Myanmar — 2 — Capacities not institutionalized, while staff turnover is expected
  - Treasury Management and Financial Systems Modernization — 2 — Few results, unlikely to sustain on their own
  - External Sector Statistics/Laos — 2 — Insufficient buy-in created to maintain new system
  - Macroeconomic Management Capacity/Laos — 1 — Not institutionalized, dependent on continued support by IMF experts
  - Budget and Treasury Management — 3 — Good prospects that new skills will be shared internally, by the beneficiary
  - Regional Government Finance Statistics — 2 — Staff capacity is limited and staff turnover is high
  - Strengthening Financial Stability Framework — 2 — Staff attrition, combined with low absorption capacity
  - Tax Administration Reforms — 2 — Insufficiently internalized capacity, no political buy-in
  - Budget Management and Customs Administration — 2 — Very low number of qualified staff to sustain new capacity
  - Regional Financial Agencies — 3 — Most changes are sustainable, but HR cuts are limiting the effects longer term
- Total average sustainability (evaluators) — 2,00
- Main drivers of low sustainability: staff that received support not institutionalizing changes; staff turnover; weak institutions relative to task complexity; absence of comprehensive action plans addressing institutional challenges.
- Survey finding: 63% of respondents (predominantly TA providers) believe that an “improvement at the individual level leads to a change at the institutional level” (with a 60% likelihood); the evaluation notes this belief is not supported by the case study observations.
- Recurrent pattern: projects are sometimes successors or repeats of similar projects, suggesting repeated TA or continued “hand-holding” may be required for internalization.

### Additionality and Donor coordination
- Average evaluator scores (case study sample, n=12):
  - Additionality — 3.3 out of 4 on average
  - Donor coordination — 3.2 out of 4 on average
- Interpretation: IMF CD delivery is highly additional; TA and training are considered valuable and irreplaceable in many cases. Donor coordination is good and ensures IMF CD is generally complementary with other donor projects. Few cases exhibit mutually reinforcing coordination (synergy), which could be improved.
- Evaluator definitions used:
  - Additionality: whether a government could have self-funded and outsourced the TA, or whether other donors could have been better able to implement the TA.
  - Donor coordination: extent to which there was sufficient coordination with other donors to avoid overlap or to achieve positive synergy between IMF and others.
- Case-level results (selected examples):
  - Macroeconomic Management Capacity/Myanmar — Additionality 3; Donor coordination 3; Donors worked with: WB primarily, some JICA (hardware provision). Comment: good information sharing but "people-dependent", IMF reports not shared with WB for confidentiality reasons.
  - Central Bank Modernization — Additionality 4; Donor coordination 3; Donors worked with: JICA, WB.
  - External Sector Statistics/Myanmar — Additionality 4; Donor coordination 3; Donors worked with: EU/COMPASS.
  - Treasury Management and Financial Systems Modernization — Additionality 2; Donor coordination 2; Donors worked with: JICA, Nomura Institute, ISCA. Comment: potential overlap.
  - External Sector Statistics/Laos — Additionality 3; Donor coordination 3; Donors worked with: EU, ESCAP, WB, BoT. Comment: good coordination overall, through TAOLAM.
  - Budget and Treasury Management — Additionality 4; Donor coordination 4; Donors worked with: WB. Comment: excellent collaboration with WB.
  - Regional Government Finance Statistics — Additionality 2; Donor coordination 2; Donors worked with: n/a. Comment: no coordination visible.
  - Tax Administration Reforms — Additionality 3; Donor coordination 4; Donors worked with: WB. Comment: very active PFM working group with good exchange of information.
  - Budget Management and Customs Administration — Additionality 4; Donor coordination 4; Donors worked with: WB, EU, AFD, JICA, GIZ. Comment: good coordination, some synergy.
  - Regional Financial Agencies — Additionality 4; Donor coordination 4; Donors worked with: WB. Comment: some coordination, complementary actions.
- Aggregate interpretation: In 10 of the 12 cases there are no major additionality issues; donor coordination shows at least cooperation in 10 of the 12 cases; only two cases show synergy as assessed by the evaluators.
- Observations on donor coordination practice: IMF typically informs other donors working on the same theme/beneficiary but often stops short of deeper joint design because of differing timing, systems, priorities and objectives. Survey respondents identify “exploiting synergies” as a possible improvement (46%) and list lack of donor coordination as the third most frequent cause of “TA limitations.” Recommended improvement frequently mentioned: “sharing information” on TA recommendations made and implemented.
- Practical barriers to sharing: IMF confidentiality agreements and time needed to review and approve reports; reports may be perceived as “too technical” by partner donors. The evaluation notes an absence of a fast, intelligible means to share key information (one positive example cited: Cameroon, where each mission debriefs the working group and leaves a short summary of findings).

### TAOLAM delivery model (contextual findings)
- The evaluation compared three TA delivery models in principle:
  1. HQ delivery model: short-term (usually 2 week) missions fielded by IMF HQ staff from functional departments, complemented with (changing) STX short-term advisors and resident advisors where feasible.
  2. RTAC delivery model: Regionally based long-term experts in a Regional Technical Assistance Center delivering repeated TA missions (typically from 2 to 10 days) to a fixed group of countries, complementable by HQ short-term staff and in-country resident advisors.
  3. TAOLAM delivery model: TAOLAM functions effectively as an RTAC but exclusively delivers JSA-funded projects and is smaller in size than a typical RTAC, serving initially two, now four countries.
- TAOLAM operation and portfolio:
  - TAOLAM in operation since 2012; core beneficiaries: Cambodia, Lao PDR, Myanmar, and Vietnam.
  - Japan and Thailand considered external donors to TAOLAM. Bank of Thailand hosts the TA Office and provides in-kind support; Japan funds TA and training activities through JSA in areas: (1) public financial management, (2) monetary and foreign exchange operations, (3) government finance statistics, (4) external sector statistics, and (5) macroeconomic management.
  - Current staffing: 1 Director and 5 resident advisors in TAOLAM office; supported by three staff seconded from Bank of Thailand and two locally hired staff (office manager and IMF HQ administrators), with bulk of this support funded by non-JSA resources.
  - TAOLAM advisors work closely with IMF functional departments, TA missions and training providers, long-term resident advisors and other short-term experts, and collaborate with regional development partners including IMF-Singapore Regional Training Institute (STI) and the Regional Office for Asia and the Pacific in Tokyo (OAP).
- TAOLAM role in the review sample:
  - Of the total 19 projects in the review sample, 8 are supported by TAOLAM; only 2 of the 8 are “TAOLAM-led” (TAOLAM in charge of implementation); the other 6 are “TAOLAM-assisted” (projects managed by functional departments remotely, with regional experts housed in TAOLAM).
- Expenditures by delivery type (expenditures through April 2017):
  - All JSA projects in review sample (n=35)* — 63,687,906 (USD) — Average project size 1,819,654 (USD)
  - Asia, Non-TAOLAM (n=19) — 26,986,223 (USD) — Average project size 1,420,328 (USD)
  - TAOLAM (n=8) — 20,532,140 (USD) — Average project size 2,566,518 (USD)
  - TAOLAM-assisted (n=6) — 17,758,019 (USD) — Average project size 2,959,670 (USD)
  - TAOLAM-led (n=2) — 2,774,121 (USD) — Average project size 1,387,061 (USD)
  - Note: * for which expenditure data is available
- Interpretation: TAOLAM projects are significantly bigger in size (2,5 million USD vs. 1,4 million USD JSA Asia average as reflected in the sample). Although only eight TAOLAM projects are in the review population, these comprise a third of the total JSA expenditure in the review period, and nearly half of the expenditure in Asia; in Asia TAOLAM is the premier delivery model for JSA-funded programs.

_Attribution, sustainability, additionality and TAOLAM delivery model findings as evaluated in the 2018 JSA evaluation by the IMF_

### 4.2 Efficiency of the TAOLAM delivery model

### 4.2 Efficiency of the TAOLAM delivery model

### 4.2.1 Expenditure categories
- IMF data on expenditure categories show differences between all JSA projects, JSA projects in Asia, and TAOLAM projects within Asia.
- TAOLAM projects allocate relatively more to Long-term experts (LTX) — the Bangkok-based regional experts — and relatively less to HQ-delivered short-term experts (STX).
- The data demonstrate that the differences between TAOLAM and non-TAOLAM are not sweeping:
  - ±10% less HQ delivery, in return for ±10% more regionally-based TA delivery.
  - All other categories remain similar to other JSA-funded projects.

### 4.2.2 Operational cost
- Detailed operational expense figures for TAOLAM were not available to the evaluators; findings are based on interviews with TAOLAM and TAOLAM advisors.
- Reported additional costs compared to HQ delivery:
  - An additional office location in Bangkok.
  - Additional costs for expatriate staff stationed in Bangkok that would not be paid in Washington DC (such as housing allowance, cost of living adjustment (COLA)).
  - The additional position of a TAOLAM Director.
- Reported cost savings compared to HQ delivery:
  - Lower travel cost for short-term TA missions (refers to flight ticket cost and time spent on travel, but not Daily Subsistence Allowance (DSA), which is the same in all scenarios).
- Other operational observations:
  - Where TAOLAM leads the intervention (2 cases), approval of expenses and similar processes is faster and more efficient because approvals are done in Bangkok, rather than from HQ.
  - Likely less HQ staff time needed for project management due to the presence of a TAOLAM Director.
  - Compared to a RTAC, the office is larger which may (or may not) lead to a reduction in operational cost (i.e. lower overhead).
  - Much administration for TAOLAM projects is still done from HQ, rather than delegated to the TAOLAM office, likely increasing administrative processing cost and decreasing efficiency.
  - All other cost components appear essentially the same compared to HQ delivery or a RTAC delivery model.

### 4.2.3 Implementation cost
- Data detailing implementation cost (cost per delivered person day, training day, or travel expenses) is not available; direct comparison with other delivery models is not possible.
- Qualitative findings from interviews and 10 case studies:
  - Use of regionally-based experts in all TAOLAM-supported programs was highly appreciated and considered a more efficient alternative than HQ delivery.
  - Geographic proximity allows for more frequent, shorter visits: TAOLAM advisors minimum mission duration typically 2-4 days versus +/- 10 days for an HQ mission. Beneficiaries in earlier stages of development consider this essential for follow-up visits.
  - Much TA (e.g., on ESS) does not justify an LTX deployment; repeated short missions (“on the job training”, “trouble-shooting”-like TA) are appropriate where a typical HQ mission of 10 days would be inefficient.
  - Shorter communication lines and developed relationships yield quicker reaction times: TAOLAM advisors can visit within weeks, whereas beneficiaries perceive HQ missions would typically arrive within months.
  - Hypothetically, an RTAC model would likely have regionally-based experts serving a larger population of countries and projects relative to the number of advisors, which could diminish flexibility to some extent but retain ability to deliver shorter and more frequent visits intact.

### 4.3 Effects of the TAOLAM delivery model on performance

#### Quantitative performance scores
- Source: IMF Assessment Reports; IMF Scoring: 1-not achieved / 2-partially achieved / 3-largely achieved / 4-fully achieved.
- Table 29: Average effectiveness and overall objective scores, TAOLAM and non-TAOLAM delivery models
  - Asia, Non-TAOLAM (n=19): Average overall objective scores 2.0; Average outcome (objectives) scores (=effectiveness) 2.1
  - TAOLAM (n=8): Average overall objective scores 2.3; Average outcome scores 2.2
  - TAOLAM-assisted (n=6): Average overall objective scores 2.1; Average outcome scores 1.9
  - TAOLAM-led (n=2): Average overall objective scores 3.0; Average outcome scores 3.3
- Caveats:
  - Scores are self-reported IMF figures and are composite scores (averages over groups of countries).
  - Differences are small and only significant to a limited extent; n=2 for TAOLAM-led is too small to draw general conclusions.
  - Analysis includes assessments of both completed and active programs; it was not possible to separate active and completed programs.

#### Results by TAOLAM country
- Table 30: Average effectiveness and overall objective achievement scores in TAOLAM countries, per delivery model (Source: IMF Assessment Reports; IMF Scoring as above)
  - Cambodia
    - Non-TAOLAM: # of projects in the country 9; Average Overall Objectives Score 2,2; Average Outcome Score 2,2
    - TAOLAM: # of projects in the country 4; Average Overall Objectives Score 2,5; Average Outcome Score 2,3
  - Laos
    - Non-TAOLAM: # of projects in the country 7; Average Overall Objectives Score 2,5; Average Outcome Score 2,3
    - TAOLAM: # of projects in the country 6; Average Overall Objectives Score 2,1; Average Outcome Score 2,1
  - Myanmar
    - Non-TAOLAM: # of projects in the country 6; Average Overall Objectives Score 1,8; Average Outcome Score 2,0
    - TAOLAM: # of projects in the country 5; Average Overall Objectives Score 2,3; Average Outcome Score 2,3
  - Vietnam
    - Non-TAOLAM: # of projects in the country 3; Average Overall Objectives Score 2,5; Average Outcome Score 2,5
    - TAOLAM: # of projects in the country 4; Average Overall Objectives Score 2,5; Average Outcome Score 2,3
- Interpretation:
  - No clear consistent trend across countries due to small project counts and small deviations from average scores.
  - Only in Myanmar and Laos is there a difference of about 20%; TAOLAM scores better in Myanmar but worse in Laos.

#### 4.3.1 Effectiveness of TAOLAM projects (qualitative)
- TAOLAM advisors provide a service model more tailored to beneficiary needs through more frequent, short visits (e.g., 10 times 2 days or 5 times 4 days per year versus 2 missions of 10 days per year).
- The TA is often focused on “hand-holding”, “training on the job”, and “troubleshooting”, which suits situations where authorities need time to incorporate advice or await new data updates. This effect is more visible in countries like Myanmar.
- Recurrent visits by the same experts enable:
  - Avoiding repeated introductions to institutional and country specifics.
  - More concrete assistance due to deeper understanding of institutional realities and political sensitivities.
  - Faster identification of new TA needs and quicker, better judgment of those needs.
- Beneficiaries tend to prefer an LTX, but where workload does not justify a resident advisor, TAOLAM advisors are viewed as the next-best option compared to one-off HQ STX visits.
- Possible disadvantages compared to HQ and RTAC:
  - TAOLAM office is small (5 advisors), limiting “bandwidth” of TA versus a larger RTAC or HQ where more advisors are present.
  - Fewer peer learning/collaborative opportunities compared to a larger office, though peer learning can occur across distances.
  - If new TA needs arise not available among current advisors, recruiting new advisors in TAOLAM is cumbersome; RTACs may more readily supply required expertise from existing capacity.
  - RTACs typically have longer funding cycles (up to 5 years) versus TAOLAM (2-3 years), which affects planning security and prolonged presence needed in low-capacity countries.

#### 4.3.2 Relevance of TAOLAM projects (qualitative)
- Quantitative data do not show TAOLAM projects achieve planned impact much more than non-TAOLAM projects, but field visits yield qualitative observations:
  - More frequent contact and deeper involvement by TAOLAM advisors and the director lead to deeper understanding of needs, obstacles, and technical/institutional challenges; geographic proximity and recurrent engagement enable more intense contact comparable to RTAC potential.
  - Examples:
    - In Myanmar and Laos, macroeconomic management TA was adjusted to a more basic level after frequent interactions revealed gaps in participants’ background knowledge; an introductory course “quantitative methods for financial programming” (basic Excel skills) was introduced.
    - In Myanmar, the monetary operations advisor was highly valued for understanding the country’s unique situation and adjusting TA to constraints on independent monetary policy; regional advisor provided more relevant advice (e.g., on reserve money targeting and implementing a new definition of reserve requirements) than some HQ-fielded MCM missions.
  - TAOLAM-led projects (2 cases) show stronger improvements in design and re-adjustment where the director has strong engagement and fuller control over resources; when TAOLAM is only supporting implementation, involvement at design stage is limited and informal, making effects discrete and harder to establish with certainty.
  - Coordination with other donors is not intense or often synergetic, but TAOLAM’s Bangkok location and more frequent field visits make more coordination possible (anecdotal coordination between STA and EU-ASEAN COMPASS on external sector statistics TA in Myanmar and Laos).
  - Risk: TAOLAM has one funder; interdependency of TA projects across IMF-funded pieces means gaps may not be filled by TAOLAM funding and filling gaps from HQ is complex and may not enable use of regional experts critical to success (e.g., real sector statistics).

### 4.3.3 Visibility of Japan
- Japan is highly visible as a donor across case studies; management-level staff at beneficiary institutions is fully aware that funding originates from Japan, and staff directly receiving TA are typically aware.
- In the three SE Asia countries visited (Myanmar, Cambodia, Laos), Japan is generally the largest donor and is the default answer for funder recognition.
- Visibility of Japan is not lower for non-TAOLAM projects.
- Work-floor staff are typically unaware of what TAOLAM is or how it differs from other IMF support funded by Japan, even when management-level staff know the TA comes through TAOLAM.

*Source: IMF Assessment Reports and IMF data summarized in "4.2 Efficiency of the TAOLAM delivery model", JSA 2018.*

### 4.4 Conclusions

### 4.4 Conclusions

### Efficiency of TAOLAM
- Compared to an HQ delivery model TAOLAM offers similar efficiencies.
- Whether or not the cost savings (lower travel cost and –time) offset the additional cost (overseas benefits, possibly additional overhead) is not possible to ascertain with the given cost data.
- Given the type of expenditure it is likely that the net effect (i.e. the net increase, or net decrease in operational cost) is relatively small.
- The main efficiency advantage of an RTAC would likely be more economies of scale, possibly leading to lower overhead, if one presumes that more project management functions could be shifted from IMF HQ to the TA centre.
- It is certain that the efficiency of shorter, more frequent visits is higher than if the same was delivered through an HQ mission; HQ delivery would be less efficient. Delivery through a possible RTAC model would be equally efficient.
- TAOLAM efficiency could be increased if more administrative responsibility was delegated from HQ to TAOLAM, which is currently not the case.
- The increased flexibility is in principle efficiency-enhancing, and is observed with the TAOLAM model. In relative terms, HQ delivery would offer the least flexibility, and RTAC probably slightly less flexibility, while TAOLAM delivery is the most flexible.
- In all, as the relatively small changes in expenditure categories underline, the efficiency of TAOLAM as a delivery model is likely not significantly different from other models.

### Effectiveness and sustainability
- The effectiveness scores in IMF’s system do not show a significant deviation between TAOLAM-delivered and other projects.
- Regional advisors based in the TAOLAM office are visiting their projects much more frequently, and in so doing develop better understanding and better relationships with beneficiary institutions; that affects the TA delivery positively, as could be seen in case studies.
- Recurrent visits are more effective, as the advisors need understand the (institutional, political and technical) environment only once, and most TA delivered requires many but short and flexible inputs from the advisors. Longer HQ missions would be less effective.
- This effect is most expressed where relatively basic advice is need, such as is the case in e.g. Myanmar.
- The TAOLAM delivery model enables a more tailor-made support, and is an effective “go-between” if deployment of a resident advisor is not economic.
- In all, there is likely a positive effect of the delivery model on the effectiveness of the TA; largely because regional advisors can cater better to the needs of beneficiaries.
- Average IMF rating: 2,2

### Relevance and impact
- The quantitative scores on relevance, as with effectiveness, do not show that TAOLAM projects are more relevant than non-TAOLAM (i.e. HQ-delivered projects).
- Given the more frequent (and recurrent) involvement of TAOLAM advisors, liaison, communication and subsequently understanding of the political, institutional and technical challenges is deeper than in case of HQ delivery.
- This improves the quality of the design, and enables re-adjustments of the TA, increasing – potentially - the relevance of the TA delivered through TAOLAM, compared to HQ delivery.
- It is likely that the more TA projects are managed actively by TAOLAM, the more expressed this advantage would be.
- A limitation of the current TAOLAM set-up is that it is challenging to maintain coherence and coverage of the country TA program, if a specific TA project is not accepted by TAOLAM’s sole funder.
- In all, the TAOLAM delivery model is likely to increase relevance of JSA-funded TA (compared to HQ-delivery) as it enables more “tailor-made” designs, and more flexible response to changing (political and institutional) changes.
- Average IMF rating: 2,3

### Visibility of Japan
- Visibility of Japan as a donor is generally high; all beneficiaries are aware that Japan is the (sole) funder of the TA.
- There is no difference noticeable whether or not the TA is delivered through TAOLAM or HQ; in other words, the delivery model as such does not matter for visibility of Japan.
- Visibility score: 4 (highest rating)

### Comparison of TAOLAM with RTAC and HQ delivery models
- Efficiency
  - No major inefficiencies noted for TAOLAM.
  - Compared to TAOLAM, the efficiency of HQ delivery is likely to be similar.
  - Compared to TAOLAM, the efficiency of RTAC delivery is likely to be higher.
  - Cost-efficiency, economies of scale, speed of decision making, cost effectiveness:
    - Compared to HQ: (Possibly) additional overhead and additional allowances (e.g. housing, COLA), but lower travel cost.
    - Compared to RTAC: lower economies of scale (RTAC is assumed to be significantly bigger than current TAOLAM office).
    - Compared to HQ: faster decision making possible, more flexible delivery (shorter missions possible).
    - Compared to HQ: more cost-effective delivery of repeated ST missions.
- Effectiveness & sustainability
  - Compared to TAOLAM, the effectiveness of HQ delivery is likely to be lower.
  - Compared to TAOLAM, the effectiveness of RTAC delivery is likely to be similar.
  - Outcome achievement, including sustainability:
    - Compared to HQ: where LTX is not feasible, STX from HQ are less appreciated by beneficiaries, while regional experts visiting the same projects more frequently leads to higher effectiveness.
    - Compared to RTAC: TAOLAM has a lower bandwidth of TA expertise to offer within the delivery model.
- Synergies/coordination with other IMF and Japan-funded programs
  - Compared to HQ: in principle, more frequent and personal contacts are possible with a regional office.
  - JICA and other Japan-funded programs are generally well aware of JSA activities, but limited options exist.
- Synergies/coordination with other donor programs
  - Few examples of synergetic cooperation visible, not majorly different with TAOLAM (IMF-wide issue).
- Peer learning opportunities
  - Compared to HQ and RTAC: the small size of TAOLAM (5 advisers) limit the possibility of peer learning/collaboration within TAOLAM, compared to an RTAC or HQ delivery.
- Relevance & impact consistency with own strategic priorities (countries, issues)
  - Compared to HQ: more frequent contact with beneficiaries possible, adding to the design and relationship management of RR and MC.
  - TAOLAM has no defined role in the TA design process, but possibly informal influence.
- Visibility of Japan
  - Compared to TAOLAM, the visibility of Japan of HQ delivery is likely to be similar.
  - Compared to TAOLAM, the visibility of Japan of RTAC delivery is likely to be similar.
  - Visibility is high, unlikely that visibility of Japan as donor would change with the delivery model.

### Summary conclusions
- The efficiency of the TAOLAM model is relatively similar to other models (on balance), but could be slightly improved if an RTAC model was deployed (more economies of scale are possible).
- The effectiveness of TAOLAM projects is higher than HQ delivery if regional advisors can be used, and could be somewhat improved (though a wider bandwidth of technical expertise) in a larger RTAC.
- The relevance of TAOLAM projects is higher compared to HQ delivery, and similar to RTAC delivery.
- The visibility of Japan as donor would not be influenced by either delivery model.

*Source: JSA-2018, 4.4 Conclusions (IMF).*

### Annex 9.4. In summary, the following table describes the way in which we adapt the concept to IMF CD

### Annex 9.4. In summary, the following table describes the way in which we adapt the concept to IMF CD

### 5C definitions (adapted to IMF context)
- Core Capacities and adapted definitions:
  - C1 - Commit and engage
    - The organisation has a mandate to carry out its tasks
    - There is leadership buy-in for the organisation
    - Rating scale: Low; Low/moderate; moderate/high; high
  - C2 - Carry out tasks
    - Staff has the necessary skills and is carrying out the tasks required
    - Rating scale: Low; Low/moderate; moderate/high; high
  - C3 - Relate and attract resources
    - The leadership of the organisation has developed functional relationships with key stakeholders
    - The organisation has sufficient means - in its context – to fund its operations
    - Rating scale: Low; Low/moderate; moderate/high; high
  - C4 - Adapt and self-renew
    - Staff is able to adapt to new circumstances without outside (donor) assistance
    - Rating scale: Low; Low/moderate; moderate/high; high
  - C5 - Maintain coherence
    - The organisation is able to ensure that its objectives are not counteracted by other organisations, but can establish (policy-) coherence that reinforces its objectives
    - Rating scale: Low; Low/moderate; moderate/high; high

- Caveats on the evidence:
  - Results are impressions evaluators gained in the field based on interviews alongside the evaluation protocol and “cannot be seen as ‘hard evidence’.”
  - A much deeper institutional research would be needed for a more secure assessment.
  - The assessment serves “purely a ‘learning purpose’ only and is not an ‘accountability’ item.”

### 5C assessment summary (case study beneficiary institutions, n=17)
- Methodology notes:
  - For each project (which may have more than one beneficiary institution) the rating is shown and whether any change was attributable to IMF support.
  - Change notation: “=” means no change; “+” means one level up; “++” means two levels up; “-” means one level down.
- Representative findings from Table 33 (selected entries preserved verbatim as in source):
  - Macroeconomic Management Capacity — Myanmar — CBM: Low = | Low + | Low ++ | Low = | Low =
  - Macroeconomic Management Capacity — Myanmar — MoF: Low + | Low + | Low = | Low = | Low =
  - External Sector Statistics — Myanmar — CBM FEMD: Low ++ | Low/moderate - | Low/moderate - | Low = | Low +
  - External Sector Statistics — Myanmar — DICA: Low/moderate + | Low + | Low + | Low = | Low +
  - Treasury Management and Financial Systems Modernization — Laos — BoL: Moderate = | Low = | Low = | Low/moderate = | Low =
  - External Sector Statistics — Laos — BoL: Low = | Low ++ | Low ++ | Low ++ | Low +
  - (Additional project-level rows captured in the assessment for 17 beneficiary institutions; full table contains detailed per-project C1–C5 ratings and changes.)

### Analysis of 5C assessment (aggregate from Table 34)
- Occurrence (number of cases reviewed): 17 for each core capacity.
- Number of Low Capacity / positive improvement attributable to IMF:
  - C1: 7
  - C2: 11
  - C3: 8
  - C4: 11
  - C5: 3
- Number of Moderate Capacity / positive improvement attributable to IMF:
  - C1: 7
  - C2: 5
  - C3: 3
  - C4: 6
  - C5: 3
- Number of High Capacity (occurrence n/a entries preserved as in source):
  - C1: 3
  - C2: 1
  - C3: 2
  - C4: 0
  - C5: 0
- Percentages (preserved verbatim):
  - % of Low Capacity/positive improvement attributable to IMF:
    - C1: 71%
    - C2: 73%
    - C3: 67%
    - C4: 27%
    - C5: 46%
  - % of Moderate Capacity/positive improvement attributable to IMF:
    - C1: 29%
    - C2: 60%
    - C3: 0%
    - C4: 50%
    - C5: 25%
  - % of High Capacity:
    - C1: 18% n/a
    - C2: 6% n/a
    - C3: 12% n/a
    - C4: 0% n/a
    - C5: 0% n/a

- Interpretation and key observations:
  - The data are impressions from visits and interviews with 17 institutions or departments; the number of cases is too low to draw wide-ranging quantitative conclusions.
  - Trends coincide mostly with qualitative case-study findings.
  - Relative strengths of IMF TA in improving capacities:
    - C2 – Carry out tasks: strongest effects. “73% of Low and 60% of Moderate core competencies are improved.”
    - C1 – Commit and engage: second-best performance. “71% of the ‘Low’ category, things change to the better,” but only “29%” improvement for Moderate categories.
    - C3 – Relate and attract resources: strong effect where capacity is Low (not when Moderate).
    - C4 – Adapt and self-renew and C5 – Maintain coherence: much lower effects; often require deeper HR changes, re-organisation, or higher-level political buy-in.

- Explanatory context from field impressions:
  - IMF is seen as a provider of authority and technical expertise:
    - Authority: IMF seen as benchmark for best practices and prestigious to beneficiaries.
    - Technical expertise: HQ staff and experts are considered deep; resources they can tap into are seen as deep.
  - IMF less seen as an effective institutional change agent, for two reasons highlighted:
    - Institutional assessments are not routinely done and incorporated in theories of change (which are often the same across many countries).
    - Profile and delivery mode of IMF experts: HQ or STX missions are short and focus on technical state rather than institutional constraints; LTX deployments are more attentive to institutional realities but are one person with no budgets to resolve constraints like IT or multi-institutional coordination.
  - Where more fundamental institutional change occurred, cooperation with another donor (typically the WB) was often intense and built into program design.

### Visibility of Japan (Specific research questions and synthesis from case studies)
- Visibility framework:
  - Four indicators: V1 Awareness; V2 Seen as effective & adequate; V3 Added value of Japan as donor; V4 Positive image of Japan as donor.
  - Four audiences: beneficiary directly involved; beneficiary management; key outside stakeholders; broader public.
  - Scale used in synthesis examples: Y = Yes/visible, Somewhat, N = No/not visible, n/a = not applicable.
- Key summary findings from case-study synthesis:
  1. “The majority of projects’ direct beneficiaries are aware that the funding of the IMF TA originates from Japan. Only in three cases this was not known.”
  2. “JSA-funded are seen as effective and adequate in fewer, but still the majority of cases.”
  3. “The value added of Japan as a donor appears to depend on factors like Japanese actors/experts being deployed on the project, JICA involvement as cooperating donor, or Japanese trainings and scholarships being provided. This is the case in about half of the cases.”
  4. “JSA-funded IMF TA does transport a positive image of Japan; only in the three cases where the funding origin was not known to the beneficiaries, this could not be the case.”
  5. “The broader public in all four countries sees Japanese aid as an important contributor; however, in no case was there a publication that could be retrieved which IMF’s support was discussed and Japan was mentioned as the funder.”
- Country-level impressions:
  - Myanmar: visibility at a maximum — every audience aware of funding origin; aid seen as positive across the board.
  - Laos and Cambodia: generally similar to Myanmar but with one project in each country where funding origin was not known.
  - Cameroon: awareness lower (two projects reviewed); Japanese aid is less well known compared to SE Asia and funding origin is largely “insider knowledge.”
- Identified limitations to Japan’s visibility:
  - Difficulty for recipients to find out which mission or project is JSA-funded.
  - No “branding” of JSA on reports, events or publications visible to evaluators in the reviewed cases.
  - Suggested implication: if feasible within a multilateral context, branding could increase Japan’s visibility.
- Aggregate frequency totals (as preserved in the source table):
  - Frequency Yes, visible: 85 7 7 8  7 5 4  5 4 4  7 7 8 12
  - Frequency Somewhat visible: 21 1 2 0  3 4 3  1 2 1  2 2 0 0
  - Frequency No, not visible: 39 3 3 4  2 3 2  4 5 4  3 3 3 0
  - Frequency n/a: 11 1 0 0  0 0 3  2 1 3  0 0 1 0
  - Total (Y=3,S=2,N=1, n/a=0) 156
  - Totals reported: V1 total:   V1 total: 82     V2 total: 75     V3 total: 60     V4 total: 91

### Conclusions and evaluation questions addressed
- Scope of evaluation:
  - Period assessed: May 1, 2013 through April 30, 2017.
  - Focus: performance of IMF capacity development projects supported by the Japanese Subaccount (JSA).
- Key evaluation questions listed in the annex:
  1. Have the JSA-supported projects been efficient, effective and relevant? And were the programmes sustainable and additional?
  2. What were the factors that have enhanced or detracted JSA-supported projects from reaching their objectives?
  3. How have the JSA-supported projects increased the visibility of Japan in the supported countries?
  4. How efficient and effective is the TAOLAM delivery mechanism, and has it led to more visibility of Japan?

- Overall evaluative synthesis (preserved as in source):
  - IMF TA shows strongest effects in improving C2 – Carry out tasks, followed by effects on C1 – Commit and engage, and on C3 – Relate and attract resources when starting from Low capacity.
  - IMF TA shows weaker effects on deeper institutional capacities C4 – Adapt and self-renew and C5 – Maintain coherence.
  - IMF is perceived as a provider of authority and technical expertise but less as an institutional change agent; delivery modalities and lack of routine institutional assessment help explain this pattern.
  - Japan’s visibility through JSA-funded IMF TA is generally high among direct beneficiaries and other audiences in the sample, but visibility is limited by lack of clear labeling/branding and by the specialist nature of IMF TA.

*Evaluation annex from the IMF Japanese Subaccount (JSA) evaluation, Annex 9.4, JSA 2018.*

### 5. How effective has coordination of JSA-funded projects with other donors been?

### 5. How effective has coordination of JSA-funded projects with other donors been?

### Evaluation approach and scope
- The evaluation team:
  - Conducted an extensive review of IMF project documentation and data.
  - Conducted interviews with IMF HQ staff.
  - Designed and implemented an electronic survey under IMF TA providers.
  - Visited a sample of 12 case studies in four countries (Myanmar, Laos, Cambodia, Cameroon).
  - The 12 case studies were equivalent to 37% of expenditure, or 28% of the number of projects in the JSA portfolio under review.

### Inputs and portfolio composition
- Total JSA-funded expenditure for the period under review: US$ 63,687,906.
- Portfolio coverage:
  - 43 projects.
  - 89 countries.
- Thematic allocation of expenditure:
  - Fiscal Affairs Department (FAD): 46% of expenditure.
  - Statistics (STA): 25% of expenditure.
  - Monetary and Capital Markets (MCM): 22% of expenditure.
- Regional deployment:
  - Asia-Pacific: about three quarters of JSA-supported projects.
  - Africa: 17%.
  - (Eastern) Europe: 8%.

### Outputs and delivery modalities
- Delivery modes and expenditure shares:
  - Short missions by short term experts (STX) or HQ staff (1–2 weeks): more than 50% of expenditure.
  - Long-term experts (LTX): 29% of cost.
  - Seminars and study tours: 7%.
  - Project management and support: 8%.
- Quality and appropriateness:
  - Output quality: generally and consistently very high; IMF considered a “benchmark” and beneficiaries view its advice as “global best practice”.
  - Challenges: some TA and training were too complex for beneficiaries due to low absorption capacity and suboptimal trainee selection.
  - Resource allocation concerns: TA sometimes provided to beneficiaries with insufficient commitment or used expensive experts for relatively basic TA.
  - Perceived cost-effectiveness: beneficiaries often considered LTX (regional or resident advisors) more effective and ultimately more cost-effective than STX and HQ support.

### Outcomes and objective achievement
- IMF project outcome measurement:
  - IMF measures achievement on a scale of 1 to 4 (1=not achieved, 2=partially achieved, 3=largely achieved, 4=fully achieved).
  - Of 43 projects, 23 were completed; average IMF-reported outcome score: 2.77.
  - Departmental differences: STA most successful, MCM least successful.
  - Evaluation team concerns: many outcomes were not clearly defined or were outputs rather than outcomes; indicators were often not sufficiently ‘SMART’.
  - Composite scoring issues: IMF composite scores averaged over countries/beneficiaries complicated interpretation.
- Case study comparisons:
  - IMF average rating for the 12 case studies (8 completed projects): 2.91.
  - Evaluation team average rating for the same projects: 2.5.

### Impacts, overall objective achievement, and attribution
- IMF average rating for achievement of project objectives across all 23 completed projects: 2.3 (close to “partially achieved”).
- Distribution of objective ratings among 23 completed projects:
  - 12 projects rated 3 or higher (“largely achieved” or better).
  - 11 projects rated lower than 3, of which only two were lower than 2.
- Case study averages:
  - IMF reported average: 2.94 for completed case studies.
  - Evaluation team rated average: 2.25 for the same projects.
- Attribution in the 12 reviewed cases:
  - TA made at least a difference in 3 cases.
  - TA was a critical factor in 7 cases.
  - TA was the direct cause of observed change in 3 cases.

### Efficiency of JSA delivery
- Case study findings:
  - In 8 out of 12 cases there were no major efficiency concerns.
  - In 4 out of 12 cases noteworthy efficiency concerns were observed.
- Average efficiency rating: 2.67 (2.86 if weighted averages used), suggesting mostly minor inefficiencies.
- Suggested efficiency improvements:
  - Better alignment of short missions with beneficiary needs.
  - Consideration of beneficiary absorption capacity.
  - Improved follow-up support.
  - Reallocating some budgets from less effective STX to LTX (resident or regional) deployments.

### Effectiveness and relevance
- Overall effectiveness:
  - JSA-supported projects under review were between partially and largely effective.
  - IMF’s own ratings: 2.7 on average for all completed projects (closer to “largely effective”).
  - Case study measure: 2.5 on average (between partially and largely effective).
  - In 5 of 12 cases effectiveness rated as 3; in 7 of 12 rated 2 or lower.
- Main cause of lower than expected outcomes: institutional constraints limiting absorption of TA or training.
- Theories of Change (ToCs):
  - Only 5 of 12 ToCs considered coherent.
  - 4 ToCs incoherent.
  - 3 ToCs have some issues.
  - 8 of 12 cases have no specific goals defined.
  - 5 ToCs confuse outputs with outcomes or ill-define outcomes and objectives.
  - Only 5 cases take institutional constraints into account.
  - Individual IMF experts make ad hoc efforts to address institutional constraints; systematic ToC development adapted to country circumstances is recommended.

### Relevance and political economy considerations
- Relevance: Projects generally align with beneficiary country priorities.
- Political feasibility assessment:
  - IMF assesses political feasibility through area departments and achieves general “no objection” buy-in.
  - Political feasibility is not systematically assessed, nor are political economy challenges incorporated into project designs.
- Political constraints:
  - In 8 of 12 cases, even when outcomes are achieved, insufficient political-level buy-in prevents use of new capacity for policy decisions.
  - IMF TA does not typically address political constraints actively; these are a main reason for partial impact achievement.

### Sustainability and wider capacity development effects
- Sustainability:
  - Largely assured in 2 cases.
  - In all other cases only part of the results expected to be sustained.
  - Main reason: institutional weaknesses relative to task complexity.
- 5C methodology findings (effects on five core organisational functions):
  - C2 – Carry out tasks: strongest effects; 73% of Low and 60% of Moderate core competencies improved.
  - C1 – Commit and engage: second-best effects; 71% of the “Low” category improved, 29% improvement when core competencies were moderate.
  - C3 – Capacity to attract resources: strong effect where capacity is Low; not seen when capacity is moderate.
  - C4 – Adapt and self-renew and C5 – Maintain coherence: much lower effects; these often require HR change, re-organisation, or political buy-in, which did not frequently occur.
  - To achieve more capacity development effects IMF would require similar expertise on institutional development as it has on monetary policy, fiscal affairs, or statistics.

### Additionality, donor coordination, and visibility of Japan
- Additionality:
  - Delivery of JSA-supported CD projects rated highly additional: 3.3 out of 4 on average.
  - In many cases TA or training considered valuable and irreplaceable.
- Donor coordination:
  - Generally good: 3.2 out of 4 on average.
  - Ensured complementarity with other donor projects.
  - Few cases of mutually re-enforcing coordination (true synergies).
  - Donors would appreciate more sharing of IMF TA reports and results information.
- Visibility of Japan:
  - Japan’s visibility: very high across countries and within context of a multilateral institution.
  - Majority of projects’ direct beneficiaries aware that IMF TA funding originates from Japan; only in three cases this was not known.
  - JSA-funded IMF TA conveys a positive image of Japan where funding origin is known.
  - Value added of Japan as donor often depends on Japanese actors/experts deployed, JICA involvement, or Japanese trainings/scholarships (present in about half of cases).
  - Recommendation: better “branding” (Japan mentioned on reports, debriefs) and improved information sharing about which IMF projects are JSA funded to improve visibility slightly.

### TAOLAM delivery model assessment
- Comparative assessment:
  - Efficiency of TAOLAM: relatively similar to other models on balance; could be slightly improved with an RTAC model (more economies of scale possible).
  - Effectiveness: TAOLAM more effective than HQ delivery if regional advisors can be used; could be improved further with a wider technical expertise bandwidth in a larger RTAC.
  - Relevance: TAOLAM relevance higher compared to HQ delivery, similar to RTAC delivery.
  - Visibility of Japan: would not be influenced by choice of delivery model.

### Key statistics and exact figures
- Total JSA-funded expenditure: US$ 63,687,906.
- Projects: 43.
- Countries: 89.
- Thematic shares: FAD 46%, STA 25%, MCM 22%.
- Regional shares: Asia-Pacific ~75% (about three quarters), Africa 17%, (Eastern) Europe 8%.
- Delivery cost shares: STX and HQ missions >50%, LTX 29%, seminars/study tours 7%, project management/support 8%.
- Case study representation: 12 case studies = 37% of expenditure and 28% of projects.
- IMF outcome average (43 projects, 23 completed): 2.77.
- IMF case study average (8 completed): 2.91; evaluation team average for same projects: 2.5.
- IMF objectives rating across 23 completed projects: 2.3 average.
- Efficiency average rating: 2.67 (2.86 weighted).
- Additionality score: 3.3 out of 4.
- Donor coordination score: 3.2 out of 4.

### Recommendations (summarized)
- More “design thinking”:
  - Use IMF technical and contextual expertise to develop comprehensive results chains and realistic theories of change that reveal institutional and political challenges.
  - Adopt tailor-made, non-routinized interventions when needed.
- Operationalize institutional development:
  - Absorb and apply existing concepts and tools for institutional assessment and institutional development planning.
  - Consider deploying institutional development experts alongside subject matter experts; upfront costs likely offset by reduced need for repeated TA.
- Political economy thinking:
  - Apply political economy analyses (PEA) at the design stage to identify political constraints and inform ToCs.
  - Use PEA to decide whether to engage, change goals, or support institutional advocacy for political change.
- Cost implications:
  - Adding rigorous institutional and political economy assessments at identification stage is a relatively small expense (e.g., 20–30 person days) relative to project size and can prevent wasteful spending and improve results.
  - Incorporating institutional development and political economy into interventions should be largely budget neutral given potential offsets from reduced repetition and better targeting.
- Suggested capacity-building reading (examples provided in the annexes of the evaluation).

*The evaluation team, JSA 2018*

### 9.1 Terms of Reference

### 9.1 Terms of Reference

### Background and objectives of the evaluation
- Japan is the largest contributor to IMF capacity development (CD), including technical assistance (TA) and training, and since 1990 Japan-financed IMF CD has helped country authorities build capacity in fiscal, monetary, financial, related statistical fields, and legal and administrative capacity.
- Contributions are provided through the Japan Subaccount for Selected Fund Activities (JSA); since 1990 Japan has contributed more than $449 million for IMF projects and programs.
- JSA funds cover program and project costs, including salaries and travel costs of staff and experts and costs associated with organizing seminars and workshops.
- Since April 2010, JSA financed TA activities are delivered under a programmatic approach to provide region-wide and medium-term assistance, encourage synergies, and enhance effectiveness and sustainability.
- Previous independent external evaluations:
  - First evaluation: February 2010 (all projects completed through April 2008).
  - Second evaluation: June 2011 (covered 150 projects totaling $25 million completed between May 2008 and April 2010).
  - Third evaluation: initiated in FY14 (covering programs and projects from April 2009 to April 2013).
- Sponsorship and coordination:
  - The evaluation is requested and sponsored by the Japanese Ministry of Finance under article 3(b) of Annex II of the Letter of Understanding with Japan.
  - Contact with the Ministry of Finance will be coordinated through the IMF Office of the Executive Director for Japan (OED-Japan).
- Purpose and scope:
  - A team of external evaluators will be selected through competitive bidding, consistent with IMF procurement policy, to conduct the fourth independent evaluation.
  - Objectives of the evaluation:
    - Assess the degree to which JSA-supported projects achieved their objectives, including relevance, effectiveness, efficiency, sustainability, and impact, using the DAC criteria and the IMF’s Common Evaluation Framework (Annex 1).
    - Identify factors that enhanced or detracted from achieving objectives, key lessons learned, and make recommendations for improvement, ensuring improvements on some DAC criteria are not at the expense of others.
  - Coverage:
    - Programs approved in FY13, FY14 and FY15, and programs approved in FY12 completed after the last evaluation.
  - Additional Japan-specific issues to be examined:
    - Coordination in the field between JSA-financed IMF CD and other donors, particularly Japan’s other ODA initiatives through strengthened information sharing with Japanese authorities.
    - Increasing visibility for Japan on JSA-financed CD activities.

### Scope and special focus: TAOLAM
- The evaluation will include the delivery model of the Technical Assistance Office for the Lao People's Democratic Republic and the Republic of Myanmar (TAOLAM), since its inception in 2012.
- TAOLAM-specific evaluation questions:
  - Efficiency: To what extent were CD activities delivered by TAOLAM efficient in comparison with alternative CD delivery models (standalone JSA programs, Regional Technical Assistance/Training Centers)?
  - Effectiveness: Compared with alternative delivery models, to what extent was TAOLAM effective in achieving objectives, identifying and addressing implementation risks, and responding to recipient country needs?
  - Impact/sustainability: To what extent is delivery by TAOLAM likely to affect (marginally) the impact of the CD activity and sustainability of its outcomes and benefits?
  - Coordination in the field: How has the existence of TAOLAM and presence of its coordinator in Bangkok affected coordination with other external partners, including Japan’s other ODA initiatives?
  - Visibility of Japan as donor: To what extent has the TAOLAM model been effective in enhancing the visibility of Japan as donor partner to recipient countries?
  - Country coverage: How has inclusion of Cambodia and Vietnam as TAOLAM beneficiary countries affected TAOLAM’s performance, including effectiveness, resource allocation, and visibility of Japan?
- The TAOLAM evaluation will identify lessons learned, assess longer-term viability, and make recommendations to improve TAOLAM’s ability to deliver JSA-sponsored programs efficiently and effectively.

### Governance of the evaluation
- Management:
  - The Global Partnerships Division of the Institute for Capacity Development (ICDGP) will manage the evaluation process in consultation with an evaluation committee (EC).
- Evaluation Committee (EC) responsibilities:
  - Review and agree on draft Terms of Reference to ensure stakeholder-relevant issues are covered.
  - Review and comment on the Inception Note prepared by evaluators.
  - Review and comment on the Draft Evaluation Report.
- EC composition:
  - Eight members: three from IMF area departments, three from IMF TA departments, and two from the IMF Institute for Capacity Development.
- Coordination with OED-Japan:
  - ICDGP will work closely with OED-Japan; each deliverable will be circulated to OED-Japan. Comments by the EC and OED-Japan on deliverables may be considered by the evaluator at its discretion.

### Evaluation methodology
- Approach:
  - The evaluation team will follow guidelines in the IMF’s Common Evaluation Framework, which provides a common core methodology and allows incorporation of additional materials as appropriate, in consultation with OED-Japan.
- DAC evaluation questions (examples provided in Table 1) address:
  - Relevance: e.g., beneficiary ranking of objectives; derivation from international standards or complementarity with other CD providers; linkage to IMF surveillance or program priorities.
  - Efficiency: e.g., provide estimates of costs of CD activity; estimate value of results and compare to costs; estimate costs of alternative delivery methods; assess minimum-cost delivery where monetary estimates are unavailable.
  - Effectiveness: e.g., extent objectives were achieved or likely to be achieved; identification and mitigation of implementation risks; relevance to capacity needs identified in country surveillance.
  - Impact: e.g., assess all changes attributable to CD activity compared to most likely counterfactual; provide quantitative estimates where possible.
  - Sustainability: e.g., degree to which transfer of knowledge is likely to be retained and disseminated; continued funding for CD; persistence of behavioral change or policies/laws supported by CD.
- Evaluation criteria and ratings:
  - A quantitative rating scheme will be used: the five DAC criteria will be scored on a 1-4 scale and averaged (equal weights to each DAC criterion).
  - For the overall score, a weighted average of these scores will be computed with weights given to the objectives, ensuring each evaluation has a score attached to each objective/outcome and a summary score for the whole evaluation.
- Information sources (each criterion should be assessed using at least two sources):
  - Document and data analysis: project/program proposals, work plans, previous assessments; financial information provided by ICD and TA departments.
  - Interviews: semi–structured interviews with country authorities as appropriate and cost effective; meetings in Washington with IMF technical assistance and area departments and the Institute for Capacity Development.
  - Survey tools: evaluators may conduct a survey to consult a wider range of beneficiaries.
  - Case studies: evaluators expected to visit four to five countries for in-depth field investigations; countries to be visited will be discussed and agreed during the Inception Phase and outlined in the Inception Note.

### Timeline and deliverables
- Timeline summary:
  - The evaluation team’s work is expected to take about 20 weeks, beginning in the first quarter of 2017 and ending in the fall of 2017.
  - The contract with evaluators will be for a maximum of 100 person–working days (including travel) during that period.
  - Three phases: Desk Phase, Field Phase, and Synthesis Phase.
- Phase details and person-day estimates:
  - Desk Phase:
    - Deadline: At the latest, 4 weeks after contract signing and before the field phase begins.
    - Tasks: desk review of documents; visit IMF Headquarters to interview staff in GP, TA departments, and Area Departments; prepare an Inception Note finalized in consultation with GP, the EC, and Japan-OED; brief IMF representatives before field phase.
    - Total work time: about 25 person days.
  - Field Phase:
    - Evaluators will visit 4-5 beneficiary countries, ensuring consultation with a variety of stakeholders and working closely with national authorities and partner offices where relevant.
    - Total work time: about 30 person days, including travel time.
  - Synthesis Phase:
    - Focused on preparation of the draft report, with necessary follow-up interviews with IMF staff.
    - Draft evaluation report to present main findings, lessons learned, and recommendations, with a summary of information gathered.
    - Draft report to be prepared in English and submitted electronically in about 3 weeks; IMF and Japan-OED will provide comments within a 5-week period; team will consider comments at their discretion and submit final report 2 weeks later.
    - Total work time: about 45 person days.
- Indicative week-by-week timeline:
  - Week 1-5: Desk review of materials, submission of the Inception Note, and travel planning.
  - Week 6: Approval of Inception Note and meetings at IMF headquarters with TA managers in IMF functional departments, ICDGP, and OED-Japan staff; draft and send out the survey.
  - Week 7-10: Field work (including discussions with resident advisors, and representatives of beneficiary countries and other donors).
  - Week 11-14: Survey analysis, preparation of the draft evaluation report.
  - Week 15-18: Comments by IMF and Japan to the draft evaluation report.
  - Week 19-20: Comments incorporated and final evaluation report submitted.
- Report timing and language:
  - Final report in English expected to be printed in mid-2017.
  - Japanese translation, also covered by the JSA, will follow.
- Deliverables (evaluators will produce):
  - An Inception Note that sets out:
    - (i) an overview of how the evaluation will be conducted;
    - (ii) methodology for information collection and analysis (including criteria for selecting the case studies);
    - (iii) draft interview guidelines;
    - (iv) a detailed plan for data collection;
    - (v) a list of potential interviewees;
    - (vi) plans for field visits and meetings;
    - (vii) outline of a quality control mechanism to ensure draft deliverables are of appropriate quality.
  - The survey and semi-structured interview instruments for feedback from CD project experts and beneficiary country authorities.
  - A draft evaluation report in English (3-4 page executive summary; full draft 40-50 pages).
  - A final evaluation report in English.
- Report presentation expectations:
  - Draft and final reports will present main findings and recommendations taking into account the scope and objectives of the evaluation, with all interviews and meetings listed in an appendix.
  - Recommendations should be concise, prioritized, grouped by time horizon and target audience, and as targeted as possible to facilitate implementation.
  - Evaluators will work with ICDGP to ensure accurate Japanese translation of the final report.

### Evaluator qualifications
- Team composition:
  - An experienced independent team consisting of a lead and at least two other professionals with backgrounds in macroeconomics, financial management, CD evaluation, or related fields.
  - Team should also have or be augmented by consultants with expertise on Japanese ODA policy and survey management.
  - Team members will not have worked on any JSA programs under evaluation.
- Desired qualifications and experience:
  - Extensive knowledge of IMF issues, including capacity development and training.
  - Some capacity and background in macroeconomic policymaking desirable.
  - Experience in the region and countries covered by JSA CD, especially Asia.
  - Experience in evaluation; experience in evaluation of Capacity Development and training is an asset.

*Terms of Reference for the 2017 External Independent Evaluation, RFP 1128 Japan Technical Assistance Subaccount (JSA), January 2017 — Japan Ministry of Finance–sponsored evaluation coordinated through the IMF Office of the Executive Director for Japan.*

### 9.2 JSA Data

### 9.2 JSA Data

### Project inventory and timeline highlights
- Projects listed include: FAD_APD_2015_01; FAD_MCD_2014_01; FAD_AFR_2013_01; FAD_AFR_2016_01; FAD_APD_2016_01; FAD_MCD_2013_01; FAD_EUR_2010_01; FAD_APD_2011_01; FAD_APD_2012_06; FAD_EUR_2014_01; FAD_MCD_2010_02; FAD_EUR_2015_01; FAD_APD_2012_03; FAD_CE6_2012_01; FAD_APD_2014_01; FAD_CE6_2015_01; IMF_APD_2014_01; INS_STI_2013_01; ICD_STI_2016_02; IMF_APD_2016_01; LEG_MMR_2013_01; LEG_MMR_2016_01; MCM_AFR_2010_03; MCM_MMR_2013_01; MCM_APD_2013_01; MCM_AFR_2015_01; MCM_EAC_2012_01; MCM_APD_2011_03; IMF_APD_2013_01; MCM_MMR_2015_01; MCM_KHM_2014_01; MCM_IND_2016_01; MCM_IDN_2014_01; STA_APD_2012_18; STA_IMF_2014_20; STA_APD_2014_21; STA_APD_2013_19; STA_EUR_2012_18; STA_APD_2011_17; STA_APD_2016_10; STA_APD_2015_10; STA_APD_2014_20; STA_IMF_2012_15.
- Start dates range from 1/Nov/09 (FAD_EUR_2010_01, FAD_MCD_2010_02, STA_APD_2012_18 start entry) through 29/Jun/15 (ICD_STI_2016_02). End dates include 31/Jul/14, 30/Sep/16, 30/Apr/17, 30/Apr/18, 30/Jun/15, 30/Jun/17, 5/Jan/18 among others.

### Allocated budgets and expenditures (May 2013 - Apr 2017 review period; IMF expenditures until Sept 2017)
- FAD_APD_2015_01: Allocated budget 4.885.014; Expenditure until Sept 2017 2.516.249.
- FAD_MCD_2014_01: Allocated budget 3.501.532; Expenditure until Sept 2017 3.213.266.
- FAD_AFR_2013_01: Allocated budget 2.749.178; Expenditure until Sept 2017 1.704.297.
- FAD_AFR_2016_01: Allocated budget 2.614.003; Expenditure until Sept 2017 1.755.204.
- FAD_APD_2016_01: Allocated budget 1.986.293; Expenditure until Sept 2017 925.464.
- FAD_MCD_2013_01: Allocated budget 1.966.125; Expenditure until Sept 2017 1.786.813.
- FAD_EUR_2010_01: Allocated budget 1.532.226; Expenditure until Sept 2017 -.
- FAD_APD_2011_01: Allocated budget 1.473.975; Expenditure until Sept 2017 1.211.610.
- FAD_APD_2012_06: Allocated budget 2.004.538; Expenditure until Sept 2017 2.716.308.
- FAD_EUR_2014_01: Allocated budget 1.000.001; Expenditure until Sept 2017 -.
- FAD_MCD_2010_02: Allocated budget 948.757; Expenditure until Sept 2017 304.701.
- FAD_EUR_2015_01: Allocated budget 600.000; Expenditure until Sept 2017 -.
- FAD_APD_2012_03: Allocated budget 3.268.598; Expenditure until Sept 2017 2.719.699.
- FAD_CE6_2012_01: Allocated budget 3.004.055; Expenditure until Sept 2017 2.120.253.
- FAD_APD_2014_01: Allocated budget 4.944.738; Expenditure until Sept 2017 4.401.731.
- FAD_CE6_2015_01: Allocated budget 4.100.000; Expenditure until Sept 2017 3.917.302.
- IMF_APD_2014_01: Allocated budget 2.086.916; Expenditure until Sept 2017 1.415.337.
- INS_STI_2013_01: Allocated budget 4.719.481; Expenditure until Sept 2017 -.
- ICD_STI_2016_02: Allocated budget 4.220.511; Expenditure until Sept 2017 -.
- IMF_APD_2016_01: Allocated budget 2.121.686; Expenditure until Sept 2017 1.358.784.
- LEG_MMR_2013_01: Allocated budget 961.478; Expenditure until Sept 2017 800.941.
- LEG_MMR_2016_01: Allocated budget 835.061; Expenditure until Sept 2017 672.692.
- MCM_AFR_2010_03: Allocated budget 1.238.972; Expenditure until Sept 2017 882.513.
- MCM_MMR_2013_01: Allocated budget 2.114.314; Expenditure until Sept 2017 2.210.607.
- MCM_APD_2013_01: Allocated budget 3.448.785; Expenditure until Sept 2017 3.377.324.
- MCM_AFR_2015_01: Allocated budget 3.155.825; Expenditure until Sept 2017 485.195.
- MCM_EAC_2012_01: Allocated budget 2.852.054; Expenditure until Sept 2017 -.
- MCM_APD_2011_03: Allocated budget 2.277.444; Expenditure until Sept 2017 1.605.128.
- IMF_APD_2013_01: Allocated budget 4.592.155; Expenditure until Sept 2017 4.321.068.
- MCM_MMR_2015_01: Allocated budget 2.141.651; Expenditure until Sept 2017 689.827.
- MCM_KHM_2014_01: Allocated budget 1.293.613; Expenditure until Sept 2017 257.833.
- MCM_IND_2016_01: Allocated budget 1.465.252; Expenditure until Sept 2017 -.
- MCM_IDN_2014_01: Allocated budget 1.175.573; Expenditure until Sept 2017 295.266.
- STA_APD_2012_18: Allocated budget 3.138.766; Expenditure until Sept 2017 3.093.011.
- STA_IMF_2014_20: Allocated budget 2.527.019; Expenditure until Sept 2017 2.272.442.
- STA_APD_2014_21: Allocated budget 1.832.529; Expenditure until Sept 2017 1.349.795.
- STA_APD_2013_19: Allocated budget 3.744.927; Expenditure until Sept 2017 3.825.284.
- STA_EUR_2012_18: Allocated budget 3.200.885; Expenditure until Sept 2017 -.
- STA_APD_2011_17: Allocated budget 2.370.930; Expenditure until Sept 2017 1.800.593.
- STA_APD_2016_10: Allocated budget 2.264.133; Expenditure until Sept 2017 1.343.891.
- STA_APD_2015_10: Allocated budget 2.100.196; Expenditure until Sept 2017 1.116.211.
- STA_APD_2014_20: Allocated budget 1.392.498; Expenditure until Sept 2017 925.366.
- STA_IMF_2012_15: Allocated budget 724.874; Expenditure until Sept 2017 295.902.

### Regional coverage, number of countries, and TA involvement
- Examples of coverage and totals (as reported for each project):
  - FAD_APD_2015_01: Nr of countries 7; Region Asia-Pacific; Total original budget, all years 4.885.014; Costs/country (of budget in review period) 697.859; TAOLAM involvement T-assisted.
  - FAD_MCD_2014_01: Nr of countries 7; Region Eastern Europe / Caucasus; Total original budget, all years 3.501.532; Costs/country 500.219; TAOLAM involvement Non-Asia.
  - FAD_AFR_2013_01: Nr of countries 11; Region Sub-Saharan Africa; Total original budget, all years 4.125.653; Costs/country 249.925; TAOLAM involvement Non-Asia.
  - FAD_APD_2014_01: Nr of countries 4; Region Asia-Pacific; Total original budget, all years 4.944.738; Costs/country 1.236.185; TAOLAM involvement T-assisted.
  - IMF_APD_2014_01: Nr of countries 2; Region Asia-Pacific; Total original budget, all years 2.086.916; Costs/country 1.043.458; TAOLAM involvement T-led.
  - INS_STI_2013_01 and ICD_STI_2016_02: Nr of countries 0; Region Asia-Pacific; Total original budget, all years 6.900.001; TAOLAM involvement Non-Asia.
  - STA_IMF_2014_20: Nr of countries 48; Region Asia-Pacific; Total original budget, all years 2.527.019; Costs/country 52.646; TAOLAM involvement Asia, Non-TAOLAM.
- Many projects specify TAOLAM involvement as T-assisted, T-led, T-assisted/T-led, Asia, Non-TAOLAM, or Non-Asia.

### Delivery-category cost breakdowns (selected examples)
- FAD_APD_2015_01: STX_DELIVERY 1.099.803; LTX_DELIVERY 233.789; HQ_DELIVERY 788.698; SEMINAR_STUDYTR 232.283; PROJECT_BKST 70.300; PROJECT_MNGT 28.206; LANG_SERVICES 62.921; MISCELLANEOUS -; LOCAL SUPPORT 249.
- FAD_MCD_2014_01: STX_DELIVERY 867.660; LTX_DELIVERY 1.196.268; HQ_DELIVERY 557.642; SEMINAR_STUDYTR 124.715; PROJECT_BKST 130.345; PROJECT_MNGT 51.160; LANG_SERVICES 284.395; LOCAL SUPPORT -; MISCELLANEOUS 1.081.
- FAD_APD_2014_01: STX_DELIVERY 1.605.260; LTX_DELIVERY 1.611.463; HQ_DELIVERY 826.844; SEMINAR_STUDYTR 58.273; PROJECT_BKST 142.819; PROJECT_MNGT 70.706; LANG_SERVICES 1.924; LOCAL SUPPORT 80.105; MISCELLANEOUS 4.337.
- IMF_APD_2013_01: STX_DELIVERY 952.342; LTX_DELIVERY 2.484.715; HQ_DELIVERY 481.487; SEMINAR_STUDYTR 49.497; PROJECT_BKST 208.253; PROJECT_MNGT 103.180; LANG_SERVICES 29.029; LOCAL SUPPORT 11.099; MISCELLANEOUS 1.466.
- STA_APD_2012_18: STX_DELIVERY 1.009.979; LTX_DELIVERY 721.548; HQ_DELIVERY 606.169; SEMINAR_STUDYTR 548.284; PROJECT_BKST 135.687; PROJECT_MNGT 71.344; LANG_SERVICES -; LOCAL SUPPORT -; MISCELLANEOUS -.
- STA_IMF_2014_20: STX_DELIVERY 358.405; LTX_DELIVERY -; HQ_DELIVERY 993.516; SEMINAR_STUDYTR 729.985; PROJECT_BKST 53.631; PROJECT_MNGT 134.029; LOCAL SUPPORT -; MISCELLANEOUS 2.876.

### Performance ratings (IMF average Objective Score, Outcom e Score, and Composite status)
- Scores reported per project (IMF: Av. Objective Score; IMF: Av. Outcome Score; IMF: Composite score?):
  - FAD_APD_2015_01: 1,0; 1,0; n.
  - FAD_MCD_2014_01: 3,0; 2,6; y.
  - FAD_AFR_2013_01: 2,5; 2,7; y.
  - FAD_AFR_2016_01: 2,0; 1,8; n.
  - FAD_APD_2016_01: 1,7; 1,5; y.
  - FAD_MCD_2013_01: 1,5; 1,8; y.
  - FAD_EUR_2010_01: 2,8; 3,2; n.
  - FAD_APD_2011_01: 2,5; 3,1; y.
  - FAD_APD_2012_06: 2,6; 1,6; y.
  - FAD_APD_2012_03: 3,0; 3,0; y.
  - FAD_CE6_2012_01: 4,0; 3,3; y.
  - FAD_APD_2014_01: 2,0; 1,8; y.
  - IMF_APD_2014_01: 3,0; 3,8; y.
  - INS_STI_2013_01: 4,0; 4,0; n/a.
  - ICD_STI_2016_02: 4,0; 4,0; n/a.
  - IMF_APD_2016_01: 3,0; 2,8; y.
  - LEG_MMR_2013_01: 2,0; 2,8; n.
  - LEG_MMR_2016_01: 1,4; 1,4; n.
  - MCM_AFR_2010_03: 2,2; 2,3; y.
  - MCM_MMR_2013_01: 2,0; 2,0; n.
  - MCM_APD_2013_01: 1,5; 1,7; n.
  - MCM_AFR_2015_01: 1,7; 1,4; y.
  - MCM_EAC_2012_01: 1,9; 2,1; y.
  - IMF_APD_2013_01: 1,4; 1,2; y.
  - STA_APD_2012_18: 3,0; 2,9; y.
  - STA_IMF_2014_20: 2,0; 2,3; y.
  - STA_APD_2014_21: 2,0; 1,8; y.
  - STA_APD_2013_19: 4,0; 3,7; y.
  - STA_EUR_2012_18: 3,0; 3,5; y.
  - STA_APD_2011_17: 3,5; 3,3; y.
  - STA_APD_2016_10: 2,0; 1,8; y.
  - STA_APD_2015_10: 1,0; 1,3; y.
  - STA_APD_2014_20: 2,0; 1,5; y.
  - STA_IMF_2012_15: 3,0; 3,0; y.

*Source: JSA Data (9.2) from the cited JSA compilation.*

### 9.3 Survey results

### 9.3 Survey results

### Research population & sampling: definition of review period & eligible programs
- The total population of CD programs approved in FY10-FY17 is 62 programs with a total budget of USD 199,920,012.
- This evaluation covers JSA-funded programs in the period FY14-FY17, defined as 1 May 2013 until 30 April 2017.
- Inclusion criteria for the research population:
  - Programs approved in FY13 to FY16. Programs approved (started) after May 2016 are not considered.
  - Programs approved before the review period but active at least 1 year in the review period (end date on or after May 1st, 2014).
  - All effects realized of the above programs are covered with a cut-off date of May 2017 (end of FY 2017).
- Approximations for expenditure (per country, per project) used for selection:
  - Total budgets are assumed to be proxies for real expenditures.
  - All budgeted costs are assumed to be distributed equally between participating countries and equally between years of implementation (example: a USD 3M program in 2 countries over 3 years → USD 0.5M per country per year).
  - Exact expenditures will be retrieved for each project in the case studies.
- Research population summary (based on assumptions above):
  - 43 programs in 89 countries.
  - Total budget in the review period of USD 104,576,561.

### Research sample selection: criteria and composition
- The research sample is limited to four countries and 12 projects to be reviewed in the field.
- Stratified-sample selection criteria:
  - Geographic distribution – representative.
  - Thematic distribution – representative.
  - Size of expenditure – preference for larger expenditure.
  - Status – preference for completed or almost completed programs.
- Geographic distribution (estimated budgets in review period):
  - Asia-Pacific: USD 72,112,949 — 69% — 3 countries in case study sample.
  - Eastern Europe / Caucasus: USD 12,749,526 — 12% — 1 country in case study sample.
  - Sub-Saharan Africa: USD 19,714,087 — 19%.
  - Total: USD 104,576,561 — 100% — 4 countries.
- Countries selected (highest estimated expenditures):
  - Myanmar (largest expenditure in Asia)
  - Lao (2nd largest expenditure in Asia)
  - Cambodia (3rd largest expenditure in Asia)
  - Cameroon (2nd largest expenditure in SSA/EE; CAR had largest budget but minor actual expenditures <100,000 USD and was replaced by Cameroon)
- Country portfolio distribution by country type (budget spent in review period, approximates):
  - Low-income developing countries (LIDCs): USD 60,219,145 — 63% — 4 countries in case study sample.
  - Developing markets: USD 30,828,342 — 32% — 3 countries.
  - Fragile states: USD 20,232,949 — 21% — 1 country.
  - Frontier markets: USD 17,580,808 — 18%.
  - Total budget in review period: USD 95,636,569 — 100%.
  - Note: Percentages don’t add up to 100% as some categories overlap.
- The four selected case study countries representation:
  - All four are LIDCs.
  - Two are developing markets (Lao, Cambodia and Cameroon).
  - One is categorized as a fragile state (Myanmar).

### Thematic distribution and program status
- Thematic distribution (number of programs, budget in review period, percentage of budget, target # in case study sample):
  - Fiscal Policy and Management (FAD): 16 programs — USD 40,579,033 — 39% — target 5 programs.
  - Monetary Policy and Financial Systems (MCM): 11 programs — USD 25,755,640 — 25% — target 3 programs.
  - Macroeconomic and Financial Statistics (STA): 10 programs — USD 23,296,756 — 22% — target 3 programs.
  - Institute for Capacity Development (ICD): 4 programs — USD 13,148,594 — 13% — target 1 program.
  - Legislative Frameworks (LEG): 2 programs — USD 1,796,539 — 2% — target 0 programs.
  - Total: 43 programs — USD 104,576,561 — 100% — target 12 programs.
- Active vs. completed in the research population (budget in review period, approximates):
  - Active: USD 54,773,016 — 52% — 20 programs.
  - Completed: USD 49,803,545 — 48% — 23 programs.
  - Total: USD 104,576,561 — 100% — 43 programs.
  - “Completed” implies, in principle, a final assessment report exists (though some programs may be past formal end date without a final assessment report).
- Case study program selection aimed to maximize total budgets and preference for completed/near-completed projects.

### Case study sample: selected programs and budgets
- The evaluators selected 12 programs in Myanmar, Lao, Cambodia, and Cameroon. Selected program examples (as of April 2017, budgets are approximates for review period):
  - FAD_CE6_2012_01 Strengthening Budget Management and Customs Administration in the CEMAC — CAM — Completed — FAD — Start 1-May-11 — End 30-Apr-16 — Budget in Review Period USD 3,004,055 — Total Budget USD 5,009,501.
  - FAD_APD_2012_06 Implementing Tax Administration Reforms in Selected South East Asian Countries — CAMB, LAO — Completed — FAD — Start 1-Aug-11 — End 30-Jun-15 — Budget in Review Period USD 2,004,538 — Total Budget USD 3,625,930.
  - FAD_APD_2012_03 Effective and Efficient Budget and Treasury Management for Southeast Asia — CAMB, LAO — Completed — FAD — Start 1-Oct-11 — End 30-Apr-16 — Budget in Review Period USD 3,268,598 — Total Budget USD 4,993,940.
  - STA_APD_2012_18 Regional Government Finance Statistics — CAMB — Completed — STA — Start 1-Nov-11 — End 30-Nov-15 — Budget in Review Period USD 3,138,766 — Total Budget USD 4,959,450.
  - MCM_MMR_2013_01 Central Bank Modernization in the Union of Myanmar — MYA — Completed — MCM — Start 1-May-12 — End 30-Sep-15 — Budget in Review Period USD 2,114,314 — Total Budget USD 2,989,286.
  - IMF_APD_2013_01 Developing Treasury Management and Financial Systems Modernization in Myanmar and Lao People's Democratic Republic — LAO, MYA — Active, close to finished — MCM — Start 1-Mar-13 — End 30-Jun-17 — Budget in Review Period USD 4,592,155 — Total Budget USD 4,975,883.
  - FAD_APD_2014_01 Promoting Priority PFM Reforms in Selected Asian Countries, with a Special Focus on Myanmar — MYA — Active, close to finished — FAD — Start 1-May-13 — End 30-Apr-17 — Budget in Review Period USD 4,944,738 — Total Budget USD 4,944,738.
  - STA_IMF_2014_20 Financial Soundness Indicators — CAMB — Active, close to finished — STA — Start 1-May-13 — End 30-Apr-17 — Budget in Review Period USD 2,527,019 — Total Budget USD 2,527,019.
  - MCM_KHM_2014_01 Strengthening Financial Stability Framework — CAMB — Active — MCM — Start 1-May-13 — End 5-Jan-18 — Budget in Review Period USD 1,293,613 — Total Budget USD 1,515,122.
  - STA_APD_2014_21 External Sector Statistics Resident Advisor to Lao People’s Democratic Republic and Myanmar — LAO, MYA — Active — STA — Start 1-Feb-14 — End 31-Jul-17 — Budget in Review Period USD 1,832,529 — Total Budget USD 1,974,921.
  - FAD_CE6_2015_01 Strengthening Customs Administration in African CEMAC region and selected LIC in Asia — CAMB, CAM, LAO, MYA — Active, close to finished — FAD — Start 1-May-14 — End 30-Apr-17 — Budget in Review Period USD 4,100,000 — Total Budget USD 4,100,000.
  - IMF_APD_2014_01 Developing Macroeconomic Management Capacity in Myanmar and Lao People's Democratic Republic — LAO, MYA — Completed — ICD — Start 1-Jul-13 — End 30-Jun-15 — Budget in Review Period USD 2,086,916 — Total Budget USD 2,086,916.
- Coverage and expenditure representation:
  - Case studies cover an estimated 6% of the total estimated expenditure (USD 104,576,561 total for research population; USD 6,364,546 estimated for the case study sample in the country to be reviewed).
  - Total budget spent in the review period of the case study programs (estimated for all countries where the program is active): USD 34,095,297.
- Active vs. completed in the case study sample (as of 30 June 2017):
  - Active: 2 programs — Budget in review period USD 3,126,142 — 9%.
  - Completed OR finished: 10 programs — Budget in review period USD 30,933,156 — 91%.
  - Total: 12 programs — USD 34,059,297 — 100%.
  - If projects past their formal end date as of 30 June 2017 are counted as “finalized”, 10 of 12 (91% of the expenditure) are with “finalized” projects.

### Case study approach and stakeholders interviewed
- Case studies will include interviews with the following stakeholders:
  1. Involved IMF project/program manager, IMF country desk representatives.
  2. Involved IMF expert(s).
  3. Representative(s) of the beneficiary organisation:
     - Trained or supported staff.
     - Leadership of the organisation supported.
  4. Other key stakeholders (case-specific).
- Interviewee types 1–3 are compulsory; type 4 is case-specific.
- Data sources: documents from IMF and other sources cross-checked where possible with interviewees.
- Case study format will include:
  1. Brief description of the program’s original goals, activities and budgets.
  2. Description of the (re-constructed) theory of change.
  3. Findings: reported impacts, outcomes, outputs and inputs, vs. observed and triangulated findings on the results.
  4. Assessment using DAC criteria, including:
     - 4.1. Efficiency
     - 4.2. Effectiveness
     - 4.3. Relevance
     - 4.4. Attribution (using contribution analysis)
     - 4.5. Additionality (using donor coordination as key determinant)
     - 4.6. Sustainability (using 5C assessment as key determinant)
  5. Specific findings on learning questions:
     - 5.1. Delivery model (TAOLAM)
     - 5.2. Donor coordination (qualitative notes)
     - 5.3. Visibility of Japan
     - 5.4. Capacity development effects, using 5C assessment

### Surveys & off-site interviews: role and design
- Purpose of surveys:
  1. Extend the validity of the assessments of the case studies.
  2. Test possible hypotheses on learning questions.
- Off-site interviews rationale:
  - Many JSA-funded CD programs are multi-country with composite ratings (1–4) for entire programs in Assessment Reports.
  - To verify ratings when the evaluator visits only one country, additional interviews with project stakeholders in other countries will be attempted by telephone or Skype.
  - Conditions for off-site interviews: (a) telephone interviews feasible despite language barriers, (b) sufficient beneficiaries and experts agree and are available.
  - Interviews will be semi-structured and cover the same topics as in-country case studies.
- Survey design and timing:
  - An anonymous survey will be sent to experts and beneficiaries.
  - Survey sections:
    1. Information on the way projects are designed and managed.
    2. Learning questions on capacity development outputs and outcomes (individual and organisational level) and issues such as donor coordination and visibility of Japan.
  - Final questionnaire will be developed after the first two or three field missions to reflect insights and hypotheses from case studies.
  - Surveys planned directly following the field missions.
- Limitations of survey data:
  - Survey information from implementing parties (experts) is similar to reports from the evaluated organisation and is not validated (or triangulated) by independent evaluators.
  - Survey information from beneficiaries is likewise not independent; therefore survey information does not substitute validated case studies but can support confirmation or rejection of field-validated findings.

### Evaluation framework and triangulation
- Evaluation framework:
  - Assessments apply DAC criteria as described in the common evaluation framework of the IMF.
  - Assessments are made against reported ratings in existing IMF reports or derived from available (progress-) reporting and compared with independently established ratings where sufficient information is obtained in the field.
  - No assessment will be made where criteria are not evaluable in a given case, with explanations provided.
  - Extrapolation of findings will be done with care given different definitions for ratings applied during the review period.
- Triangulation approaches to validate and deepen understanding:
  - Data triangulation: involves time, space, and persons; seeks independent data and data sources to confirm or reject stated results.
  - Investigator triangulation: involves multiple researchers (two evaluators for each mission/case study).
  - Theory triangulation: involves using more than one theoretical scheme in interpreting observed change (reviewing factual theory of change compared to intended).
  - Methodological triangulation: involves using multiple data-gathering options (interviews, observations, questionnaires, documents), including interviews with IMF, beneficiaries, stakeholders, both directly for case studies and through surveys, document reviews, and external data assessment.

_The evaluation as described above is that of the IMF’s JSA-funded capacity development programs for the review period 1 May 2013–30 April 2017._

### 2.1 Effectiveness

### 2.1 Effectiveness

### Effectiveness — method for assessing attainment of objectives
- IMF/DAC criterion: The extent to which CD activities attained their objectives.
- Establishing effectiveness of the TA will be done as follows:
  - 1. Gathering rating information on all JSA-supported projects that are eligible in this review, and are completed (projects in progress can also be included, but would be a separate rating category)
  - 2. Extracting the ratings of the chosen sample
  - 3. Field visits (see below) and case study development, resulting in evaluators’ rating (validation)
  - 4. Computing (weighted averages) of the differences between the ratings of the evaluators and the IMF ratings (and ratings derived from survey between experts/beneficiaries and evaluators)
  - 5. Extrapolation of the weighted average differential over the total population, to arrive at an over assessment of effectiveness
- Rationale:
  - The method measures overall effectiveness by using the case studies to measure how accurate the reported values on effectiveness are, and extrapolates this, instead of directly extrapolating the results of 12 case studies over the total population, which would be likely too biased given possible selection biases.
  - This method is considered to offer a fairer and more accurate picture of total effectiveness.
- Qualitative assessment:
  - For success factors, evaluators will examine project design and implementation, including how well the TA intervention addressed beneficiary needs and the quality of CD delivered.
  - Evidence sources include IMF training satisfaction surveys and qualitative information from interviews with beneficiary staff.
  - The 5C assessment will be used to understand whether the CD delivered was actually put into practice and identify reasons and obstacles.

### Impact & attribution
- IMF/DAC Criterion: Measures the positive and negative changes brought about by CD activity, compared to the most likely counterfactual. The impacts can be direct or indirect, intended or unintended.
- Establishing impact:
  - First review documentation to establish the reported impact.
  - In case studies, crosscheck documented impact with beneficiaries and other stakeholders.
  - Where feasible, collect additional data to quantify impact (if not done in IMF documentation) and triangulate findings on impact level.
  - Exogenous factors that may have affected intervention results will be described.
- Attribution approach:
  - IMF’s common evaluation framework seeks to determine attribution by establishing the (most likely) counterfactual, but independent ex post verification of counterfactuals is not feasible within the review scope.
  - Evaluators will use contribution analysis to categorize likely contribution levels rather than produce quantitative attribution estimates.
  - Method: re-establish the impact pathway in each case study and assess likely contribution level (overview in Annex 6.5).
- Expected contribution categories:
  - (a) did not contribute to the observed changes (negative case)
  - (b) “making a difference”
  - (c) “being of critical importance”
  - (d) a “direct causal link”
- Limits and resources:
  - The available LoE per case study is about 3 person-days in the field; this precludes independent primary data collection and limits quantitative attribution.
  - Evaluators do not expect to arrive at a quantitative estimate of attribution for IMF’s CD program, but expect to determine contribution levels that inform how IMF interventions have contributed or are likely to contribute in future.

### Relevance
- IMF/DAC Criterion: The extent to which CD activities served important objectives of beneficiary countries.
- Assessment approach:
  - Determine the intended impacts (project objectives) and observed impacts (changes on goal level) in case studies.
  - Assess whether delivered TA was in line with the goals of the JSA and the beneficiary country.

### Efficiency
- IMF/DAC criterion: Measures the value of the outcomes or benefits of CD activities compared to the value of the inputs or costs incurred to achieve them.
- Comment and approach:
  - The definition used by IMF appears to refer to cost-effectiveness; the evaluation will distinguish between cost-efficiency and cost-effectiveness.
  - Cost-efficiency (input-output) is challenging due to lack of reliable benchmarks and comparability across organizations; evaluators will seek evidence to make reliable statements and will restrict to qualitative information if no reliable data exist.
  - Cost-effectiveness (input-outcome) is harder to measure because IMF reports do not include assessments of cost-efficiency that could be verified; valid observations will be included where possible.
- Case study practice:
  - Evaluators will (a) establish the full cost of the intervention reviewed, and (b) seek to estimate the monetary value of the results.
  - Monetary value is hard to determine exactly and needs adjustment for IMF attribution, which is rarely 100%.
  - Comparing cost with monetised result does not alone establish cost-effectiveness because that requires cost estimates of alternatives; such comparisons will be made only if they can be done with certainty.

### Sustainability
- IMF/DAC Criterion: Measures the extent to which the outcomes or benefits achieved by the CD activity are likely to continue or last.
- Assessment approach:
  - Sustainability will be assessed in case studies.
  - Two cases:
    - Completed projects: sustainability can be observed as fact.
    - Ongoing or recent projects: sustainability is a prediction based on indicators.
  - Use case-specific information and the 5C framework as indicators for predictions.
  - If the 5C analysis demonstrates strong capacity improvement, the likelihood of sustainable effects will generally be rated as high, and vice versa.

### Additionality & donor coordination
- Additionality assessment:
  - Considered on a case study basis only, as there is no reported indicator in the IMF M&E system.
  - Additionality focuses on whether a government could have self-funded and outsourced the TA, or whether other donors could have implemented the TA better.
  - If a government could have funded and sourced similar-quality TA itself, or another donor was doing the same (overlap), the intervention is not additional.
  - If the government could not have done so and no other donor provided similar support, the intervention is additional.
- Levels of coordination:
  - Coordination to avoid overlap represents one level.
  - Higher-level coordination is synergy, where two or more donor interventions jointly create a larger effect than independently; this constitutes the highest level of additionality for an IMF intervention.
- Practical limits:
  - Assessing whether a government could have funded and organized similar-quality TA is highly speculative and rarely possible to establish with certainty; other ratings are more likely to be supported by factual observations.
- Outputs:
  - Assessments will also inform the extent to which IMF JSA projects have been collaborating with other donors in the same or adjacent fields.
  - Any recommendations on enhancing donor coordination or collaboration will stem from these case-based assessments.

*Source: IMF.*

### 2.7 Specific questions

### 2.7 Specific questions

### 2.7.1 Assessment of the TAOLAM delivery model
- The ToR ask the evaluators to address additional, learning-oriented “satellite questions”, including:
  - To what extent the TAOLAM delivery model is efficient and effective, compared to other possible delivery models.
  - To what extent the TA delivers capacity development effects.
  - How the JSA-funded IMF TA is visible to the recipient countries.
- Institutional and operational facts about TAOLAM (IMF):
  - The IMF's Technical Assistance Office in Thailand (TAOLAM) provides TA and training mainly to Cambodia, Lao PDR, Myanmar, and Vietnam, consistent with the IMF’s capacity development strategy.
  - TAOLAM is part of the IMF’s Asia and Pacific Department (APD) and works closely with APD country teams to align capacity development with surveillance dialogue.
  - TAOLAM was established in Bangkok in September 2012; it initially covered only Lao PDR and Myanmar; in mid-2015 Vietnam and Cambodia were added.
  - A public financial management project for Southeast Asia, based in TAOLAM, also covers Indonesia, Malaysia, Philippines, and Thailand.
  - External donors: Japan and Thailand. The Bank of Thailand hosts the TA Office and provides in kind support; Japan supports TA and training activities through JSA in five areas: (1) public financial management, (2) monetary and foreign exchange operations, (3) government finance statistics, (4) external sector statistics, and (5) macroeconomic management.
  - Current staffing: 4 resident advisors in the TAOLAM office, 2 staff seconded from Bank of Thailand (an economist and a program officer), and 2 support staff (an office manager and an office driver).
  - Under current and future operations, direct administrative support for TAOLAM is likely to be provided mostly by IMF and Bank of Thailand resources.
  - Capacity development delivery: regional advisors based in TAOLAM supported by IMF functional departments, TA missions and training providers, long-term resident advisors (LTX) and short-term experts (STX); collaboration with regional development partners, IMF-Singapore Regional Training Institute (STI), and the Regional Office for Asia and the Pacific in Tokyo.
  - Note: It is likely that the Bank of Thailand will second an additional resident advisor to TAOLAM towards the end of 2017 to help prepare for training and workshops; currently only the program officer is covered by JSA resources (he receives a monthly salary top-up from the APD/ICD macro project).
- Reformulated research questions for comparison:
  - To what extent is the TAOLAM delivery model efficient compared to two alternative TA delivery models available (either an RTAC, or an HQ-driven delivery of TA)?
  - To what extent is the delivery of TA through TAOLAM effective and relevant compared to these alternatives?
  - To what extent is Japan visible as a donor in the TAOLAM delivery model, as compared to alternative delivery models (RTAC or HQ-driven)?
- Evaluation approach (three steps):
  - Compare results (relevance, effectiveness, impact, sustainability) of TAOLAM-delivered projects (case studies for at least two or possibly three TAOLAM countries) with projects implemented through other delivery mechanisms, using previous evaluations, desk research, interviews and surveys; produce qualitative “pros & cons” and an evaluators’ statement whether alternative models are likely to score higher, similar, or lower.
  - Conduct a document-based process analysis to establish facts on efficiency: cost-efficiency, cost-effectiveness, throughput times, flexibility in planning, and other emergent factors; compare with other delivery models and describe advantages/disadvantages to assess likely efficiency ratings relative to alternatives.
  - Pay specific attention to visibility of Japan as donor by comparing whether TAOLAM-delivered projects lead to higher (and better) visibility of Japan; use case study scores and qualitative findings.
- Comparative research matrix elements (criteria and indicative rating approaches):
  - Efficiency: evaluators’ rating likely to be: Higher, similar, lower; indicators include cost-efficiency, economies of scale, speed of decision making, cost effectiveness; description of facts and arguments (“Pro’s & Con’s”).
  - Effectiveness & sustainability: Score 1-4; evaluators’ rating likely to be: Higher, similar, lower; outcome achievement including sustainability; description of facts and arguments (“Pro’s & Con’s”).
  - Synergies/coordination with other IMF and Japan-funded programs: Description of facts and arguments (“Pro’s & Con’s”).
  - Synergies/coordination with other donor programs: Description of facts and arguments (“Pro’s & Con’s”).
  - Peer learning opportunities: Description of facts and arguments (“Pro’s & Con’s”).
  - Relevance & impact: Score 1-4; evaluators’ rating likely to be: Higher, similar, lower.
  - Consistency with own strategic priorities (countries, issues): Description of facts and arguments (“Pro’s & Con’s”).
  - Visibility of Japan: Visibility ratings likely to be: Higher, similar, lower; visibility scores of TAOLAM projects vs. other IMF projects; description of noticed visibility benefits and facts (“Pro’s & Con’s”).
- Evaluation constraint: The evaluators will not formulate an opinion on whether one alternative delivery model is definitively advantageous, because such a conclusion would require subjective weighting of criteria differing across stakeholders; the analysis is intended to add validated facts and arguments to inform decision-making.

### 2.7.2 Capacity development effects of JSA-funded TA program
- Objective: assess higher-level capacity development (CD) effects and sustainability — measure extent to which beneficiary organizations have improved capacity to deliver better economic decision-making and can sustain capacity without further TA.
- Analytical framing: distinguish “competence” (individual ability, e.g., from training) from “capacity” (collective organizational ability to deliver results).
- Method: apply the 5C method to establish how five core functions of an organization have changed over time; the end result will be a case-study based table describing changes in each of the 5 core competencies, colour-coded based on ratings.
- 5C adapted definitions and rating scale (Low/medium/high):
  - C1 - Commit and engage:
    - The organisation has a mandate to carry out its tasks.
    - There is leadership buy-in for the organisation.
    - Rating: Low/medium/high.
  - C2 - Carry out tasks:
    - Staff has the necessary skills and is carrying out the tasks required.
    - Rating: Low/medium/high.
  - C3 - Relate and attract resources:
    - Leadership has developed functional relationships with key stakeholders.
    - The organisation has sufficient means – in its context – to fund its operations.
    - Rating: Low/medium/high.
  - C4 - Adapt and self-renew:
    - Staff is able to adapt to new circumstances without outside (donor) assistance.
    - Rating: Low/medium/high.
  - C5 - Maintain coherence:
    - The organisation is able to ensure that its objectives are not counteracted by other organisations, but can establish (policy-) coherence that reinforces its objectives.
    - Rating: Low/medium/high.
- CD research matrix for each case study (fields and rating/changes preserved verbatim):
  - Core Capacity | Before intervention (Low/medium/high) | After intervention (Low/medium/high) | Change (Positive/similar/negative) | IMF’s contribution (Autonomous change, some contribution, high contribution)
  - Applies to: C1 - Commit and engage; C2 - Carry out tasks; C3 - Relate and attract resources; C4 - Adapt and self-renew; C5 - Maintain coherence.
- Aggregation: for aggregates, (weighted) averages will be used to show the CD effect detected in the case studies.

### 2.7.3 Visibility of Japan
- Objective: assess extent of visibility of Japan as donor for JSA-funded programs; evaluators assess extent but do not judge sufficiency (no concrete definition or target for visibility).
- Conceptual framework: a 4X4 matrix with four indicators (quality) and four audiences (depth).
  - Indicators (quality, V1–V4):
    - V1 - Awareness of JSA funding.
    - V2 - Seen as effective & adequate.
    - V3 - Added value of Japan as donor.
    - V4 - Positive image of Japan as donor.
  - Audiences (depth, D1–D4):
    - D1 Beneficiary, directly involved.
    - D2 Beneficiary, superior dep't/man't.
    - D3 Key stakeholders.
    - D4 Broader public.
- Rating options and special cases:
  - V1–V3: Yes, somewhat, no for D1–D3; D4 is marked n/a for V1–V3.
  - V4: Yes, somewhat, no for D1–D3; for D4 (broader public) V4 is Yes/No (appearances in media).
  - Visibility to the broader public is solely assessed by media presence; media publications are likely not about technical content but about V4 (positive image of Japan). Therefore V1–V3 are not scored if there is a media appearance. If media coverage is negative on Japan, it is counted as no positive visibility (“no” on V4).
- Scoring presentation: indicators will be rated in a stop light system (yes, no, or something in between); the aggregate assessment will describe (a) extent and quality of Japan’s visibility with JSA-funded programs, and (b) whether there is a difference between TAOLAM and non-TAOLAM delivered TA on visibility.
- Composite scoring option (conditional):
  - The rating system as presented does not generate a composite score by default because weighting quality vs. depth is subjective.
  - If IMF prefers a composite rating and specifies relative importance (quality vs. depth), a total score can be calculated; an illustrative composite approach (if quality and depth deemed equal) would start at 0 and add 1 point for each “yes” and 0.5 for each “somewhat” in the matrix; the maximum score in that illustrative method is then 13 (V1-4 X D1-3, plus one for media appearance).

### 2.7.4 Results Based Management (RBM)
- Context: during the review period a new result-based management (RBM) system was developed and introduced at the IMF.
- Relevance to evaluation: although assessing the new RBM system is not explicitly requested by the ToR, the evaluators will review key characteristics of the new RBM system and determine to what extent evaluation findings may justify recommendations for improvement or adaptation of the current RBM system.

---


_Source: https://www.imf.org/-/media/files/capacity-developement/jsa-annual-reports/jsa-2018.pdf_