## EXECUTIVE SUMMARY

## Source details

**Canonical URL:** [EXECUTIVE SUMMARY](https://www.imf.org/-/media/files/capacity-developement/jsa-annual-reports/jsa-eie-2024.pdf)

## Other formats

- [Markdown version](/-/media/files/capacity-developement/jsa-annual-reports/jsa-eie-2024.pdf.md)
- [Structured JSON version](/-/media/files/capacity-developement/jsa-annual-reports/jsa-eie-2024.pdf.json)

---

### Background
- The International Monetary Fund (IMF) appointed BDO LLP in January 2023 to carry out an independent mid-term evaluation of capacity development (CD) activities funded by the Government of Japan through the Japan Sub-Account for Selected Fund Activities (JSA).
- The evaluation was requested and sponsored by the Ministry of Finance, Japan (JMOF), and covered 32 JSA-funded programs delivered between 1 May 2017 and 30 April 2022.
- The evaluation assessed performance against the six OECD-DAC evaluation criteria: relevance, coherence, effectiveness, efficiency, impact, and sustainability.
- Fieldwork was carried out between June and October 2023 using: document review, semi-structured stakeholder interviews, e-surveys of recipient authorities and IMF staff, and detailed analysis of a sample of 15 JSA-funded programs; in-person field visits were conducted in Cambodia and Nigeria.
- During the evaluation period (FY2018 to FY2022), JSA contributed around US$174 million to support IMF CD activities, covering 30 percent of external financing for IMF CD (or about 20 percent of the Fund’s total expenditure on CD).
- Since 1990, Japan has contributed about $807 million to JSA; as at April 2023 some $661 million had been used to support IMF CD.

### Role, scope, and pandemic response
- JSA-funded programs are a key part of IMF CD and contribute substantially to capacity development and institution strengthening in Asia and some African countries, particularly fragile states.
- JSA-funded training is effective in building knowledge and policy-oriented skills across IMF membership in areas critical to macroeconomic and financial stability, fiscal and monetary frameworks, and emerging global challenges.
- The evaluation period was dominated by the COVID-19 pandemic; the IMF and JSA adapted by replacing in-person activities with virtual delivery of TA and training and expanding online learning programs.
- New delivery modalities contributed to integration of TA and training with surveillance.

### Relevance and coherence
- JSA-funded programs score highly for relevance and coherence: they address identified needs and complement other activities by development partners or recipient authorities.
- Programs were well aligned with priorities identified in IMF surveillance and lending programs during the evaluation period.
- BDO reviewers’ mean assessment scores (sample of 15 programs) — Relevance: 2.9; Coherence: 3.0.

### Effectiveness
- Most JSA-funded programs were relatively effective, largely or partially achieving milestones, outcomes, and objectives despite pandemic disruptions.
- Significant differences in effectiveness exist across sectors and across country types (fragile vs non-fragile, EMEs vs LICs); differing ambition levels of objectives mean effectiveness ratings do not differ widely between these groups.
- The Results-Based Management (RBM) framework remains limited as an assessment tool: progress reporting has improved but gaps and confusion persist between inputs, outputs, and outcomes.
- Departmental mean assessment scores for effectiveness (sample of 15 programs): FAD 2.3; ICD 3.3; MCM 2.5; STA 2.7; JSA overall 2.5.
- Training program outcomes and scale:
  - Total active learners in the online program rose by 90 percent (from 89K to 169K) between FY21 and FY23.
  - Completion rates, participant satisfaction, and learning gains were 68 percent, 95 percent, and 20 percentage points respectively.

### Multi-country program model
- Multi-country programs provide flexibility and potential efficiency gains at scale and facilitate peer learning and resource shifting across countries.
- Challenges include coordination within the Fund, transparency and accountability, deploying appropriate experts across heterogeneous countries, and evaluating mixed results across participant countries.
- Suggested mitigations:
  - Consider an upper limit to the number of recipient countries in full-fledged reform programs.
  - Consider restrictions for regional coverage (e.g., avoid combining APD and AFR).
  - Establish explicit criteria for reallocating resources between countries.
  - Set an indicative budget for each country at program outset.

### Efficiency and Value for Money (VFM)
- Qualitative evidence suggests VFM improved since the pandemic as remote delivery reduced costs and remained effective.
- Assessing VFM remains challenging due to lack of complete cost data and difficulty monetizing outcomes.
- Key constraints to assessing efficiency:
  - Cost and outcome heterogeneity across countries and activities.
  - Cost data historically collected at program level only; CDMAP launched late 2021 but not all functions yet operational.
  - Project budgets not disaggregated by country-project; limited information on actual budget use.
- Reallocating funds by early cancellation is impractical due to short intervention durations and limited interim information on likely ultimate achievements and costs; however, some multi-country program managers reallocated resources from stalled to better-performing country projects.

### Impact and sustainability
- Few wider impacts (higher-level effects beyond outputs and outcomes) were identified, largely due to short timeframes, attribution difficulties, and lack of routine post-completion data collection.
- Sustainability assessments relied on evidence available at or before completion (recipient authority buy-in; absorption of knowledge, tools, systems; availability of financial and human resources; staff retention and transmission).
- Based on these metrics, sustainability appears relatively modest, particularly in fragile states.
- BDO reviewers’ mean assessment scores for impact and sustainability (sample of 15 programs): Impact 2.1; Sustainability 2.4.

### Summary ratings of sample programs (sample of 15 programs; total of 58 individual projects)
- Overall average scores (four-point scale where 1 is ‘poor’ and 4 is ‘excellent’):
  - Relevance: 2.9
  - Coherence: 3.0
  - Effectiveness: 2.5
  - Efficiency: 2.9
  - Impact: 2.1
  - Sustainability: 2.4
  - Overall rating: 2.6
  - Number of projects in sample: 58
- By workstream (number of projects in sample in each workstream):
  - FAD: Relevance 2.9; Coherence 3.0; Effectiveness 2.3; Efficiency 2.8; Impact 2.2; Sustainability 2.2; Overall rating 2.5; Number of projects 31
  - ICD: Relevance 3.2; Coherence 3.3; Effectiveness 3.3; Efficiency 3.2; Impact 2.1; Sustainability 2.8; Overall rating 3.0; Number of projects 8
  - MCM: Relevance 2.8; Coherence 2.9; Effectiveness 2.5; Efficiency 2.7; Impact 1.9; Sustainability 2.5; Overall rating 2.5; Number of projects 13
  - STA: Relevance 2.9; Coherence 2.9; Effectiveness 2.7; Efficiency 3.1; Impact 2.3; Sustainability 3.0; Overall rating 2.8; Number of projects 6
  - Note: Efficiency figures are indicative only due to lack of robust and complete information on project expenditure.
- By region:
  - Asia-Pacific: Relevance 3.0; Coherence 3.1; Effectiveness 2.4; Efficiency 2.7; Impact 2.1; Sustainability 2.6; Overall rating 2.6; Number of projects 19
  - Africa: Relevance 2.9; Coherence 2.9; Effectiveness 2.5; Efficiency 2.9; Impact 2.2; Sustainability 2.3; Overall rating 2.6; Number of projects 33
  - Global / Mixed: Relevance 3.0; Coherence 3.0; Effectiveness 3.2; Efficiency 3.0; Impact 2.2; Sustainability 2.5; Overall rating 2.9; Number of projects 6
- By fragility:
  - Fragile States: Relevance 2.9; Coherence 3.0; Effectiveness 2.4; Efficiency 3.0; Impact 2.2; Sustainability 2.4; Overall rating 2.6; Number of projects 19
  - Non-fragile States: Relevance 2.9; Coherence 3.0; Effectiveness 2.5; Efficiency 2.8; Impact 2.1; Sustainability 2.5; Overall rating 2.6; Number of projects 33
  - Global / Mixed: Relevance 3.0; Coherence 3.0; Effectiveness 3.2; Efficiency 3.0; Impact 2.2; Sustainability 2.5; Overall rating 2.9; Number of projects 6

### Budgeted JSA expenditure FY18-22 by IMF CD department (budgeted expenditure during evaluation period)
- Fiscal Affairs Department (FAD): 44,663,853 (40% of total)
- Institute for Capacity Development (ICD): 24,991,795 (27% of total)
- Monetary and Capital Markets (MCM): 30,373,586 (22% of total)
- Statistics Department (STA): 9,574,744 (9% of total)
- Legal Department (LEG): 1,507,170 (1% of total)
- TOTAL: 111,111,148 (100% of total)
- Source: BDO analysis of IMF datasheet ‘Japan Projects and Execution FY18-FY22’. Note: Figures are for budgeted JSA expenditure during the evaluation period. They may not sum precisely due to rounding.

### Key recommendations (with stated priority)
- 1. The Fund should maintain or adopt important innovations and adaptations catalyzed by the pandemic – notably those relating to technology and remote delivery - across all relevant activities. (Priority: High)
- 2. Regarding training, the Fund should seek to maintain the progress made in using online and virtual courses rather than reverting fully to former modes of delivery. (Priority: High)
- 3. The Fund should seek more and earlier inputs from recipient authorities in deciding on a program, its design, and on the selection of an LTX. (Priority: High)
- 4. JSA should consider reallocating some of the funding from Asian emerging market economies to other regions, including African countries, which have greater needs for capacity building. (Priority: High)
- 5. The Fund and JSA should explore ways to structure multi-country programs that maintain their flexibility, while also supporting transparency and accountability, and reducing managerial challenges. (Priority: Medium)
- 6. The Fund should make greater efforts to make recipient authorities and IMF Mission Chiefs / Resident Representatives aware that JSA CD is funded by the Government of Japan, including increasing JSA ‘branding’ on project documents and requiring IMF project leads to inform local IMF staff and recipient authority counterparts as to the source of the funding. (Priority: Medium)
- 7. The Fund should consider ways to gather evidence routinely on the de facto sustainability and impacts achieved by JSA-funded programs after implementation. (Priority: Low)

*External Independent Evaluation of the Japan Sub-Account – Vol I | International Monetary Fund*

### EXECUTIVE SUMMARY ................................................................................... II

### EXECUTIVE SUMMARY

### Background
- The International Monetary Fund (IMF) appointed BDO LLP in January 2023 to carry out an independent mid-term evaluation of capacity development (CD) activities funded by the Government of Japan through the Japan Sub-Account for Selected Fund Activities (JSA).
- The evaluation was requested and sponsored by the Ministry of Finance, Japan (JMOF), and covered 32 JSA-funded programs delivered between 1 May 2017 and 30 April 2022.
- The evaluation assessed performance against the six OECD-DAC evaluation criteria: relevance, coherence, effectiveness, efficiency, impact, and sustainability.
- Fieldwork was carried out between June and October 2023 using: document review, semi-structured stakeholder interviews, e-surveys of recipient authorities and IMF staff, and detailed analysis of a sample of 15 JSA-funded programs; in-person field visits were conducted in Cambodia and Nigeria.
- During the evaluation period (FY2018 to FY2022), JSA contributed around US$174 million to support IMF CD activities, covering 30 percent of external financing for IMF CD (or about 20 percent of the Fund’s total expenditure on CD).
- Since 1990, Japan has contributed about $807 million to JSA; as at April 2023 some $661 million had been used to support IMF CD.

### Key findings and conclusions
- Role and scope
  - JSA-funded programs are a key part of IMF CD and contribute substantially to capacity development and institution strengthening in Asia and some African countries, particularly fragile states.
  - JSA-funded training is effective in building knowledge and policy-oriented skills across IMF membership in areas critical to macroeconomic and financial stability, fiscal and monetary frameworks, and emerging global challenges.
- Pandemic response and modality innovations
  - The evaluation period was dominated by the COVID-19 pandemic; the IMF and JSA adapted by replacing in-person activities with virtual delivery of TA and training and expanding online learning programs.
  - New delivery modalities contributed to integration of TA and training with surveillance.
- Relevance and coherence
  - JSA-funded programs score highly for relevance and coherence: they address identified needs and complement other activities by development partners or recipient authorities.
  - Programs were well aligned with priorities identified in IMF surveillance and lending programs during the evaluation period.
- Effectiveness
  - Most JSA-funded programs were relatively effective, largely or partially achieving milestones, outcomes, and objectives despite pandemic disruptions.
  - Significant differences in effectiveness exist across sectors and across country types (fragile vs non-fragile, EMEs vs LICs); however, differing ambition levels of objectives mean effectiveness ratings do not differ widely between these groups.
  - The Results-Based Management (RBM) framework remains limited as an assessment tool: progress reporting has improved but gaps and confusion persist between inputs, outputs, and outcomes.
- Multi-country program model
  - Multi-country programs provide flexibility and potential efficiency gains at scale and facilitate peer learning and resource shifting across countries.
  - Challenges include coordination within the Fund, transparency and accountability, deploying appropriate experts across heterogeneous countries, and evaluating mixed results across participant countries.
- Value for Money (VFM)
  - Qualitative evidence suggests VFM improved since the pandemic as remote delivery reduced costs and remained effective.
  - Assessing VFM remains challenging due to lack of complete cost data and difficulty monetizing outcomes.
  - Reallocating funds by early cancellation is impractical due to short intervention durations and limited interim information on likely ultimate achievements and costs; however, some multi-country program managers reallocated resources from stalled to better-performing country projects.
- Impact and sustainability
  - Few wider impacts (higher-level effects beyond outputs and outcomes) were identified, largely due to short timeframes, attribution difficulties, and lack of routine post-completion data collection.
  - Sustainability assessments relied on evidence available at or before completion (recipient authority buy-in; absorption of knowledge, tools, systems; availability of financial and human resources; staff retention and transmission).
  - Based on these metrics, sustainability appears relatively modest, particularly in fragile states.

### Summary ratings of sample programs (sample of 15 programs; total of 58 individual projects)
- Overall average scores (four-point scale where 1 is ‘poor’ and 4 is ‘excellent’):
  - Relevance: 2.9
  - Coherence: 3.0
  - Effectiveness: 2.5
  - Efficiency: 2.9
  - Impact: 2.1
  - Sustainability: 2.4
  - Overall rating: 2.6
  - Number of projects in sample: 58
- By workstream (number of projects in sample in each workstream):
  - FAD: Relevance 2.9; Coherence 3.0; Effectiveness 2.3; Efficiency 2.8; Impact 2.2; Sustainability 2.2; Overall rating 2.5; Number of projects 31
  - ICD: Relevance 3.2; Coherence 3.3; Effectiveness 3.3; Efficiency 3.2; Impact 2.1; Sustainability 2.8; Overall rating 3.0; Number of projects 8
  - MCM: Relevance 2.8; Coherence 2.9; Effectiveness 2.5; Efficiency 2.7; Impact 1.9; Sustainability 2.5; Overall rating 2.5; Number of projects 13
  - STA: Relevance 2.9; Coherence 2.9; Effectiveness 2.7; Efficiency 3.1; Impact 2.3; Sustainability 3.0; Overall rating 2.8; Number of projects 6
  - Note: Efficiency figures are indicative only due to lack of robust and complete information on project expenditure.
- By region:
  - Asia-Pacific: Relevance 3.0; Coherence 3.1; Effectiveness 2.4; Efficiency 2.7; Impact 2.1; Sustainability 2.6; Overall rating 2.6; Number of projects 19
  - Africa: Relevance 2.9; Coherence 2.9; Effectiveness 2.5; Efficiency 2.9; Impact 2.2; Sustainability 2.3; Overall rating 2.6; Number of projects 33
  - Global / Mixed: Relevance 3.0; Coherence 3.0; Effectiveness 3.2; Efficiency 3.0; Impact 2.2; Sustainability 2.5; Overall rating 2.9; Number of projects 6
- By fragility:
  - Fragile States: Relevance 2.9; Coherence 3.0; Effectiveness 2.4; Efficiency 3.0; Impact 2.2; Sustainability 2.4; Overall rating 2.6; Number of projects 19
  - Non-fragile States: Relevance 2.9; Coherence 3.0; Effectiveness 2.5; Efficiency 2.8; Impact 2.1; Sustainability 2.5; Overall rating 2.6; Number of projects 33
  - Global / Mixed: Relevance 3.0; Coherence 3.0; Effectiveness 3.2; Efficiency 3.0; Impact 2.2; Sustainability 2.5; Overall rating 2.9; Number of projects 6

### Budgeted JSA expenditure FY18-22 by IMF CD department (budgeted expenditure during evaluation period)
- Fiscal Affairs Department (FAD): 44,663,853 (40% of total)
- Institute for Capacity Development (ICD): 24,991,795 (27% of total)
- Monetary and Capital Markets (MCM): 30,373,586 (22% of total)
- Statistics Department (STA): 9,574,744 (9% of total)
- Legal Department (LEG): 1,507,170 (1% of total)
- TOTAL: 111,111,148 (100% of total)
- Source: BDO analysis of IMF datasheet ‘Japan Projects and Execution FY18-FY22’. Note: Figures are for budgeted JSA expenditure during the evaluation period. They may not sum precisely due to rounding.

### Recommendations (with stated priority)
1. The Fund should maintain or adopt important innovations and adaptations catalyzed by the pandemic – notably those relating to technology and remote delivery - across all relevant activities. (Priority: High)
2. Regarding training, the Fund should seek to maintain the progress made in using online and virtual courses rather than reverting fully to former modes of delivery. (Priority: High)
3. The Fund should seek more and earlier inputs from recipient authorities in deciding on a program, its design, and on the selection of an LTX. (Priority: High)
4. JSA should consider reallocating some of the funding from Asian emerging market economies to other regions, including African countries, which have greater needs for capacity building. (Priority: High)
5. The Fund and JSA should explore ways to structure multi-country programs that maintain their flexibility, while also supporting transparency and accountability, and reducing managerial challenges. (Priority: Medium)
6. The Fund should make greater efforts to make recipient authorities and IMF Mission Chiefs / Resident Representatives aware that JSA CD is funded by the Government of Japan, including increasing JSA ‘branding’ on project documents and requiring IMF project leads to inform local IMF staff and recipient authority counterparts as to the source of the funding. (Priority: Medium)
7. The Fund should consider ways to gather evidence routinely on the de facto sustainability and impacts achieved by JSA-funded programs after implementation. (Priority: Low)

*External Independent Evaluation of the Japan Sub-Account – Vol I | International Monetary Fund*

### 1. More ‘design thinking’, to better understand the political environment of recipient countries concerned

### 1. More ‘design thinking’, to better understand the political environment of recipient countries concerned

### Key recommendations for CD design and implementation
- Adopt more “design thinking” to better understand the political environment of recipient countries, rather than focusing just on transfer of technical knowledge.  
- Operationalize institutional development, potentially by hiring institutional development experts with knowledge of how to assess an institution, formulate realistic institutional development plans, and manage institutional change.  
- Apply “political economy thinking”, making use of “political economy analysis” to inform decision-making on whether to engage in an intervention, or to change its goals.

### Evaluation findings on JSA-funded capacity development (CD)
- CD outputs are of high technical quality, and the Fund has generally avoided duplication with other CD providers.  
- Improved use of Results-Based Management (RBM) and log frames has led to better defined milestones that help monitor implementation, and there has been progress in defining objectives, outcomes and outputs.  
- The strengthened strategic goal of aligning CD with surveillance and lending has de facto led to more political economy considerations in the design and implementation of programs.  
- The visibility of the Government of Japan as funding source for the JSA appears to have declined since the previous evaluation: while all IMF implementing staff are aware that their JSA project is funded by the Government of Japan, this is often not the case among recipient authorities and Area Department staff.

### Evaluation scope, objectives, and institutional context
- The independent evaluation was requested and sponsored by the Ministry of Finance, Japan (JMOF) in its capacity as agency responsible for the JSA.  
- The aim was to assess the degree to which JSA-funded projects achieved their respective objectives, to inform the IMF and Japan’s MOF regarding performance, to identify good practices for replication, and to provide clear and actionable recommendations to improve CD delivery and entity-level operations.  
- The evaluation paid attention to coordination in the field between JSA-funded IMF CD and other donor partners; integration of JSA-funded programs with IMF surveillance and lending activities; maximizing and accurately capturing outcomes achieved by JSA-funded programs; and increasing visibility for Japan on JSA-funded CD activities.  
- The evaluation covered CD programs taking place between 1 May 2017 and 30 April 2022, including JSA-funded online training and the STI, but excluding the IMF-Japan Tax seminars by agreement with the IMF and the JMOF.  
- IMF fiscal years run from 1 May to 30 April. The period covered by the evaluation is 1 May 2017 to 30 April 2022.

### Evaluation framework and criteria
- Performance was assessed against CD objectives and related outcomes defined in the IMF’s RBM framework, using the OECD-DAC criteria of relevance, efficiency, coherence, effectiveness, impact and sustainability.  
- The overarching project-level assessment framework was the IMF’s Updated Common Evaluation Framework (CEF) and Guidance Note.  
- The main basis for evaluating individual projects were the log frames in the IMF’s RBM Framework; log frames and monitoring data for JSA-funded programs were made available during the inception phase.  
- The OECD-DAC criteria were interpreted as:  
  - Relevance — extent to which objectives and design respond to beneficiaries, global, country and partner/institution needs, policies and priorities.  
  - Coherence — compatibility with other interventions in a country, sector or institution.  
  - Efficiency — extent to which results are delivered in an economic and timely way.  
  - Effectiveness — extent to which objectives were achieved or expected to be achieved.  
  - Impact — extent to which intervention has generated or is expected to generate significant higher-level effects.  
  - Sustainability — extent to which net benefits will continue.

### Methodology and sample
- The evaluation aimed to gather and analyse data on all 32 JSA-funded programs taking place in the period subject to evaluation.  
- A sample of selected programs was subject to in-depth analysis, including document review and scoring against the OECD-DAC criteria.  
- Document review included overall JSA annual reports, individual project and program documentation, and third-party reports and analysis.  
- Semi-structured interviews were conducted at central/global, regional and country levels, mostly remotely via MS Teams; in-person interviews were held with some IMF staff in Washington DC, and with staff and stakeholders in Cambodia and Nigeria (on-site field visits).  
- Three e-surveys using MS Forms were run between 22 September and 29 October 2023 covering: JSA recipient authorities (survey in English or French), IMF implementing staff (survey in English), and IMF Country Mission Chiefs (MCs) and Resident Representatives (RRs) (survey in English).  
  - The survey of recipient authorities received 39 responses (33 in English and six in French). This equated to 17% of possible individual respondents, covering at least one project from 64% of JSA-funded programs.  
  - The survey of IMF implementing staff received 54 responses (a response rate of 59%).  
  - The survey of MCs and RRs received 25 responses (a response rate of 20%).  
- An in-depth review was carried out of a sample of 15 JSA programs covering all individual countries/projects within the selected programs.  
- Selection of the 15 programs by lead department was based on the proportion of the JSA budget allocated to each department during the evaluation period. Budgeted JSA expenditure proportions and the corresponding sample profile for the 15 programs were:  
  - Fiscal Affairs Department (FAD): 40% — sample 6 projects (actual profile achieved in sample 6).  
  - Institute for Capacity Development (ICD): 27% — sample 4 projects (actual profile achieved in sample 4).  
  - Monetary and Capital Markets (MCM): 22% — sample 4 projects (actual profile achieved in sample 4).  
  - Statistics Department (STA): 9% — sample 1 project (actual profile achieved in sample 1).  
  - Legal Department (LEG): 1% — sample 0 projects (actual profile achieved in sample 0).  
  - TOTAL: 100% — sample 15 projects (actual profile achieved in sample 15).  
- Programs in the sample were assessed against OECD-DAC criteria using a 1–4 scale, where 1=poor, 2=modest, 3=good, and 4=excellent. For effectiveness the evaluation started with RBM ratings on milestones and outcomes and triangulated with surveys and interviews. Five programs included “deep dives”, with semi-structured interviews and fieldwork in Cambodia and Nigeria for some deep dives.

### Findings on relevance (sample results and survey evidence)
- BDO reviewers’ mean assessment scores for relevance (sample of 15 programs) by lead department: FAD 2.9, ICD 3.2, MCM 2.8, STA 2.9; JSA overall 2.9.  
- BDO reviewers’ mean assessment scores for relevance by region: African Countries 2.9, Asia Pacific Countries 3.0, Global / Mixed 3.0; JSA overall 2.9.  
- BDO reviewers’ mean assessment scores for relevance by fragile vs non-fragile classification: Fragile states 2.9, Non-fragile states 2.9, Global / Mixed 3.0; JSA overall 2.9.  
- Many projects showed appropriate linkages to government priorities and to findings/recommendations from IMF surveillance and strategy documents; many projects addressed needs expressed or acknowledged by recipient authorities, which correlated with higher effectiveness and sustainability scores.  
- Greater involvement of Area Departments and integration with surveillance and lending enhanced alignment of CD activities with country needs and willingness to address capacity gaps, though this did not substitute for full systematic “political economy analysis” and “design thinking” recommended by the previous JSA evaluation.  
- Some projects lacked recipient authority “buy-in” from an early stage, often in multi-country programs where anticipated buy-in did not materialize; lack of ownership led to poor effectiveness and sustainability in some cases and resources were sometimes reassigned.  
- Low absorption capacity at recipient authorities remains a problem, particularly in low-income and fragile states; Risk Assessment and Mitigation modules of project proposals often contained generic or incomplete mitigation measures, or were left pending until after project launch. Interim and final assessments tended to be more specific about risks and whether they materialized.  
- The evaluation suggests scope for proposal documentation to include more explicit evidence of authority commitment (e.g., an official request) that the intervention meets an identified need and that the authority will support delivery; such commitment is easier where authorities are already engaged with the IMF through surveillance or lending arrangements (e.g., countries accessing the IMF Extended Credit Facility while receiving TA).  
- E-survey results: 88% of recipient authorities responding agreed that their JSA-funded project was “correctly targeted at our country’s development needs”, and the same proportion agreed it was “correctly targeted to fill gaps in our national capacity in the relevant sector.” 77% agreed that their views “were taken on board in the design of the JSA-funded project.” (Margin of error for the 88% figure = ±9% at 95% confidence level.)  
- Recipient authorities reported that the impetus for initiating projects frequently came from the IMF rather than themselves: 41% (±14%) said IMF staff played the leading role in initiating the project; 33% (±13%) said it was an equal joint decision between IMF staff and national authorities; 26% (±13%) said the Ministry of Finance or national authorities played the leading role.  
- IMF implementing staff (Implementing Managers and Mission Chiefs / Resident Representatives) reported relevance scores similar to or higher than recipient authorities.

*Source: External Independent Evaluation of the Japan Sub-Account – Volume I, International Monetary Fund*

### 2.2 COHERENCE

### 2.2 COHERENCE

### 2.2.1 OVERVIEW
- Definition: Coherence is the compatibility of the intervention with other interventions in a country, sector or institution.
- Overall finding: Reviewers found that the coherence of most JSA-funded interventions was generally strong.
- Integration with IMF operations:
  - JSA-funded TA and training were almost always directed towards priorities identified in Article IV consultations and/or contributing to implementation of Fund-supported lending programs.
  - Integration went beyond listing nearby CD activities and enhanced the relevance, coherence, and likely effectiveness of CD.
- Notable program examples of coherent integration:
  - Revenue Administration COVID-19 (FAD_IMF_2021 09): supported two African countries in implementing their respective ECF-supported programs.
  - MCM’s CD support to Mongolia (MCM_MNG_2018_04): contributed to implementation of financial sector reforms included under the Fund-supported Extended Funding Facility (EFF).
  - Macroeconomic Frameworks (ICD_IMF_2021-05): assisted several central banks to improve forecasting capacities identified as priorities by IMF surveillance.
- Synergies with other donors:
  - Few examples of active synergies between JSA-funded projects and other donor-funded interventions, mainly due to lack of subject-matter overlap rather than missed opportunities.
  - Notable exception: a three-way collaboration on revenue administration in one African country where the JSA-funded IMF project explicitly built on work by the African Development Bank and Japan International Cooperation Agency.
- Recipient coordination and IMF–World Bank coherence:
  - Recipient authorities often played a key role coordinating TA interventions to avoid duplication.
  - Coherent approaches between the IMF and the World Bank were facilitated by good HQ-level relations in Washington DC and in-country contact between IMF experts and WB resident missions.

- BDO reviewers’ coherence scores (mean assessment score; 1 = poor and 4 = excellent; sample of 15 JSA-funded programs):
  - By lead department:
    - FAD: 3.0
    - ICD: 3.3
    - MCM: 2.9
    - STA: 2.9
    - JSA overall: 3.0
    - Source: BDO review of a sample of 15 JSA-funded programs (1 = poor and 4 = excellent)
  - By region:
    - African Countries: 2.9
    - Asia Pacific Countries: 3.1
    - Global / Mixed: 3.0
    - JSA overall: 3.0
    - Source: BDO review of a sample of 15 JSA-funded programs (1 = poor and 4 = excellent)
  - Fragile vs non-fragile:
    - Fragile states: 3.0
    - Non-fragile states: 3.0
    - Global / Mixed: 3.0
    - JSA overall: 3.0
    - Source: BDO review of a sample of 15 JSA-funded programs (1 = poor and 4 = excellent)

- Survey evidence on relevance and coherence:
  - Between 85% and 100% of IMF Implementing Managers agreed with the equivalent statements regarding relevance (Margin of error at 95% confidence level = ±6% and ±2% respectively). Source: BDO e-survey of IMF implementing staff (n=54).
  - Between 76% and 84% of IMF Mission Chiefs (MCs) and Resident Representatives (RRs) agreed with the equivalent statements regarding relevance (Margin of error at 95% confidence level = ±15% and ±13% respectively). Source: BDO e-survey of Mission Chiefs and Resident Representatives (n=25).
  - Recipient authorities’ views: Around 80% of recipient authorities agreed that the JSA-funded intervention complemented other relevant activities in their country (Margin of error at 95% confidence level = ±c.12%). Source: BDO e-survey of recipient authorities (n=39).

### 2.2.2 AWARENESS OF THE GOVERNMENT OF JAPAN AS THE SOURCE OF JSA FUNDING
- Evaluation focus: Extent to which recipient authorities and other stakeholders were aware that the source of funding for IMF JSA TA was the Government of Japan.
- Main finding: Awareness among recipient authorities was fairly good but with scope for improvement.
  - More than a third of recipient authorities responding to the survey were unaware that their JSA project was funded by the Government of Japan.
    - 36%. Margin of error at 95% confidence level = ±14%.
- How awareness was learned:
  - Almost two thirds of recipient authorities who were aware the funding was from the Government of Japan had learned this only through discussions with IMF staff rather than through ‘branding’ or other sources.
  - Implication: Scope to improve visibility by increasing JSA ‘branding’ on project documents and ensuring IMF staff make clear to counterparts who is funding the TA.
- Awareness among IMF staff:
  - IMF implementation staff are almost always aware that JSA is the source of funding.
  - More than half of IMF Mission Chiefs and Resident Representatives responding to the survey were not aware which projects and programs in their country were funded from JSA, limiting their ability to complete a survey specifically about JSA-funded TA.
  - Given donor importance of visibility, recommendation to improve branding and communication between TA managers and permanent in-country IMF staff.

*External Independent Evaluation of the Japan Sub-Account – VOL I | INTERNATIONAL MONETARY FUND*

### 2.3 EFFECTIVENESS

### 2.3 EFFECTIVENESS

### 2.3.1 OVERVIEW
- Effectiveness is defined as the extent to which the intervention achieved, or is expected to achieve, its objectives, and its results, including any differential results across groups.  
- Review methodology: reviewers assessed JSA-funded projects using RBM data / assessment reports, interviews, wider document review, and e-surveys.  
- Overall finding: JSA-funded TA was fairly effective, with some outstanding examples delivering all planned outputs and outcomes and receiving excellent feedback from recipient authorities.  
- Departmental mean assessment scores for effectiveness (1 = poor and 4 = excellent) from a BDO review of a sample of 15 JSA-funded programs:
  - FAD: 2.3
  - ICD: 3.3
  - MCM: 2.5
  - STA: 2.7
  - JSA overall: 2.5
- Interpretation: ICD training programs scored higher partly because many consisted solely of training—structured and repeatable knowledge-transfer processes with clear metrics (e.g., participant feedback ratings, ‘before and after’ test scores). ICD adapted to COVID-19 by scaling up online programs, accelerating MOOCs and short modular/blended courses.
- Training program outcomes and scale:
  - Total active learners in the online program rose by 90 percent (from 89K to 169K) between FY21 and FY23.
  - Completion rates, participant satisfaction, and learning gains were 68 percent, 95 percent, and 20 percentage points respectively.
- Key contextual factors reducing effectiveness:
  - Greater complexity of FAD and MCM projects that involve deep and lengthy institutional reforms.
  - COVID-19 disruptions that impeded implementation across much of the JSA portfolio; some programs adapted to online or hybrid modes but initial disruptions affected delivery of planned outputs, outcomes, and objectives.
  - Presence of outlier projects that failed to achieve most or all objectives, and a significant number that achieved objectives only partially.
- Regional and fragility comparisons: reviewers scored effectiveness against planned objectives (per RBM guidelines), not absolute progress. As projects in African and/or fragile countries sometimes set more modest objectives than in Asian countries, effectiveness scores were roughly similar across contexts.
  - By region (BDO sample of 15):
    - African Countries: 2.5
    - Asia Pacific Countries: 2.4
    - Global / Mixed: 3.2
    - JSA overall: 2.5
  - By fragility classification:
    - Fragile states: 2.4
    - Non-fragile states: 2.5
    - Global / Mixed: 3.2
    - JSA overall: 2.5

### Main drivers of effectiveness
- Country ownership and commitment to reform (proxied by relevance, who selected the project, and authorities’ contribution to project design).
- Initial conditions in the recipient country and agency (countries with better institutions and agencies with more resources used JSA-funded CD most effectively).
- Better internal coordination within the Fund (e.g., between two CD departments, or between a CD department and the country team), including integration of CD with surveillance and lending programs.
- Quality of experts, their relationship with participants, and delivery modality.
- Buy-in from authorities—poor buy-in frequently contributed to reduced effectiveness, sustainability, and impact.
- Program and project complexity reduced effectiveness by taxing authorities’ and staff’s implementation capacity.
- Multi-country programs sometimes complicated coordination within the Fund and created uncertainty about resource availability for each country-project.

### Illustrative examples (from evaluation sample)
- ICD_IMF_2021_05 (Macroeconomics Program):
  - One country trained authorities in Debt Dynamics Tools (DDTs), enabling development of their own tool used for public debt forecasts and scenarios; all outputs and outcomes were achieved; TA supported priorities under the country’s 2022 ECF.
  - Another country customized forecasting tools and developed sophisticated models for policymaking; all outputs and outcomes were achieved and authorities reported catalytic effects.
  - One country project in the same program made little or no progress due to COVID-19, national elections, authorities’ limited engagement, and intermittent implementation.
- FAD_IMF_2021_10 (‘Public Financial Management Covid-19’): successfully introduced new tools for risk identification and management in one African country but made limited progress on strengthening central fiscal oversight of sub-national government due to complicated political situation and institutionalization difficulties.
- MCM_AFR_2021_02 (‘Strengthening Debt Management Operational Frameworks in African Countries’): failed in several countries to achieve objectives such as integrating cash/debt management and implementing risk management frameworks due to low capacity relative to scale required.
- FAD_APD_2020_02: project objective achievement hampered by timing with country’s EFF difficulties, reluctance to engage with virtual TA, decentralization reform requiring realignment, and change in government reprioritizing MOF focus.

### 2.3.2 QUALITY OF REPORTING ON EFFECTIVENESS
- Comparison of RBM/Assessment Reports with other sources found few significant discrepancies between IMF Assessment Reports and interviews/documents/e-surveys; sometimes recipient authorities’ feedback was more positive than staff assessments (but authorities commented in general rather than on specific objectives).
- RBM implementation observations:
  - RBM was used in the design of all sampled programs; near-universal compliance in entering required information and ratings.
  - RBM contributed to better defined milestones, outputs, and outcomes, despite some confusion between inputs, outputs, and outcomes in a few cases.
  - The RBM database remains limited for assessing progress against milestones, outputs, outcomes, and objectives in detail.
- Example of conceptual confusion: one Macroframeworks project (ICD_IMF2021_05) used ‘percentage of staff trained’ as an outcome indicator, which at most is an output rather than an outcome.

### 2.3.3 EFFECTIVENESS OF DIFFERENT DELIVERY MODELS
- LTXs vs STXs:
  - A small number of cases where recipient authorities were unhappy with the quality of a JSA-funded LTX impacted buy-in and effectiveness due to the long-term nature of engagement.
  - Potential improvements: greater recipient authority input into drafting LTX TORs; consultation when deciding on LTX contract extension/termination at end of first year; possible recipient co-financing of LTX costs in middle-income countries to increase ownership.
  - Choice between LTX and STX can be driven by political context and recipient preference (e.g., some countries reject in-country LTXs for confidentiality concerns; others insist on permanently based in-country LTXs for continuity and integration).
- In-person vs virtual vs hybrid delivery:
  - Recipient authority survey (BDO e-survey, n=39; excludes pure ICD training programs) findings:
    - Delivery modality distribution: Online (virtual) only: 0%; In-person only: 31%; Hybrid (part online and part in-person): 69%.
    - In opinion on whether modality was best possible: Yes: 87%; No: 13% (Margin of error at 95% confidence level = ±10%).
    - Only 13% stated they would have preferred a different delivery method.
  - Many respondents emphasized the continuing importance of face-to-face approaches for some training activities, fact-finding missions, and meetings.
  - COVID-19 forced many projects to switch to virtual or hybrid delivery; ICD and JSA-funded investments in distance learning positioned the Fund well to substitute in-person training with online and virtual courses.
  - Online/virtual modalities proved effective and sustainable for certain activities, particularly training; combined modalities (online prerequisites, virtual follow-ups, blended delivery) often proved as effective and less costly than traditional one-off STX missions or regional LTX visits.
  - The Fund’s distance-learning experience accelerated integration of online training, synchronous virtual missions, and in-person deliveries by resident LTXs; some programs now require online training as a prerequisite for receiving TA.

Italic attribution line:
*External Independent Evaluation of the Japan Sub-Account – VOL I | INTERNATIONAL MONETARY FUND*

### 2.4 EFFICIENCY

### 2.4 EFFICIENCY

### 2.4.1 OVERVIEW
- Definition: Efficiency is the extent to which an intervention delivers, or is likely to deliver, results in an economic and timely way.
- Key constraints to assessing efficiency of JSA-funded TA:
  - It is not possible to ‘monetize’ activities, outputs and outcomes robustly and comparably across projects and programs, because:
    - The ‘value’ of outputs varies by recipient authority structure, country size, economic circumstances and development level, and by the nature and timing of resultant changes in resource use and financial flows.
    - Results timing varies across activities (immediate for forecasting models or databases; long-term for customs administration strengthening).
  - Until recently, cost data were collected at program level only and generally not by project; more complete information is now being collected via CDMAP (launched late 2021), but not all CDMAP functions are currently operational.
  - Project documentation for most evaluated programs does not disaggregate budgets between country projects, and there is limited information on actual budget use.
  - Budget figures are frequently adjusted or reiterated without a full ‘audit trail’, and are not consistently updated on a rolling basis.
  - Assessing timeliness is challenging because up-to-date information on delivery against milestones is not routinely available at individual project level; target dates for milestones can change as part of ongoing project management (COVID caused common slippage).
  - The COVID-19 pandemic significantly disrupted delivery, affecting both cost (breakdown of planned delivery models and non-availability of recipient authorities) and timeliness (extended lockdowns and slippage on milestones).
- Methodological response: In the absence of robust quantitative cost-effectiveness data, the evaluation relied on a detailed review of 15 programs and e-surveys to gather qualitative data on risks to and drivers of efficiency.

### 2.4.2 EFFICIENCY OF SINGLE VS MULTI-COUNTRY PROGRAMS
- Advantages of multi-country programs:
  - Flexible and generally efficient during the evaluation period.
  - Allow launching CD without the upfront time and resource investment required for single-country programs by avoiding the need to ascertain full commitment from each authority.
  - Resources can be shifted between recipient countries as priorities change.
  - A single LTX can be shared across multiple countries, making LTX employment more cost-effective than in single-country programs.
  - Peer learning opportunities arise from bringing diverse countries together.
- Disadvantages and risks:
  - Difficulty identifying and hiring appropriate experts when recipient countries are very heterogeneous.
  - Program management is more difficult; examples include country managers unable to respond due to uncertainty about resource availability despite overall program underspend.
  - Insufficient information on allocation of budgeted resources across country-projects complicates responsiveness and evaluation (expected to diminish once CDMAP is fully implemented).
  - Difficulty evaluating completed programs when outcomes differ across participating countries.
- Potential mitigations to maximize multi-country program benefits while managing risks:
  - Consider an upper limit to the number of recipient countries in full-fledged reform programs.
  - Consider restrictions for regional coverage (e.g., avoid combining APD and AFR).
  - Establish explicit criteria for reallocating resources between countries.
  - Set an indicative budget for each country at program outset.

### 2.4.3 REALLOCATING RESOURCES TO OPTIMISE EFFICIENCY
- Question addressed: Is there scope to identify ‘failing’ projects earlier to cancel them and reallocate funds to higher-probability-of-success activities?
- Findings:
  - Reviewers identified very few programs with a clear-cut case for cancelling work and transferring funding.
  - Practical and procedural constraints make early cancellation often impractical:
    - Most programs are relatively short; cancellation processes and documentation required by the Fund and JSA would be onerous.
    - Annual interim assessments are generally inadequate to justify cancellation; interim assessment reports often contain limited amount and quality of information.
  - At country project level within multi-country programs, there is some opportunity for responsive reallocation:
    - Several multi-country programs reduced activities to a minimum in countries with major implementation challenges and transferred resources elsewhere within the same program.
    - Retaining a largely notional project in a problematic country while transferring resources can maintain engagement at low cost and keep channels open for future more effective TA if conditions improve.

### 2.5 IMPACT (Summary of findings relevant to efficiency context)
- Definition: Impact is the extent to which an intervention generates significant higher-level effects (e.g., faster economic growth, lower fiscal deficits).
- General observation: Reviewers struggled to identify clear wider impacts from JSA-funded projects; scores for impact were generally lower than for effectiveness, reflecting lack of evidence of attributable impacts rather than evidence of failure.
- BDO reviewers’ mean assessment scores for impact (1 = poor; 4 = excellent):
  - By lead department: FAD 2.2; ICD 2.1; MCM 1.9; STA 2.3; JSA overall 2.1.
  - By region: African Countries 2.2; Asia Pacific Countries 2.1; Global / Mixed 2.2; JSA overall 2.1.
  - By classification: Fragile states 2.2; Non-fragile states 2.1; Global / Mixed 2.2; JSA overall 2.1.
- Identified attributable impacts (selected examples):
  - Continuing collection of customs revenue despite COVID-19 disruption due to improved organizational capacity from JSA-funded project.
  - Enhanced central bank technical and policy analysis capacity leading to timely, clearly communicated monetary policy actions and increased central bank credibility.
  - Increased tax revenue resulting from implementation of electronic systems and procedures.
  - Increased bank supervision capacity, improving risk management and financial stability.
  - Fiscal risk evaluations informing SOE portfolio review, leading to privatization of several companies and restructuring of public university financing.
- Challenge to evidencing impacts:
  - Short evaluation timeframes, lack of long-term follow-up, confusion between outputs/outcomes/impacts, and confounding external factors make attribution and robust evidence of impact difficult.
- Suggested improvement:
  - Commission or carry out periodic follow-up surveys of recipient authorities to collect longer-term impact and sustainability data.

### 2.6 SUSTAINABILITY
- Definition: Sustainability is the extent to which net benefits of an intervention will continue or are likely to continue.
- Overall assessment: Evidence indicates benefits from JSA-funded TA are fairly sustainable, but subject to risks; ratings are largely based on proxies (recipient buy-in, absorption of knowledge/tools/systems, staff continuity) rather than long-term evidence.
- BDO reviewers’ mean assessment scores for sustainability (1 = poor; 4 = excellent) by lead department:
  - FAD 2.2; ICD 2.8; MCM 2.5; STA 3.0; JSA overall 2.4.
- Observations by department and region:
  - STA programs scored higher on sustainability because statistics-related CD often establishes data collection/analysis/publication systems that persist with manuals and embedded processes.
  - Some FAD and MCM programs failed to establish structures/systems necessary for sustainability; significant institutional or sectoral reform was sometimes required but not achieved (e.g., due to political resistance or scale of restructuring).
  - Sustainability appears lower in African countries compared with Asian countries, mainly due to lack of resources, staff continuity issues, and greater risk of disruption from political upheaval—particularly in fragile states.
- BDO reviewers’ mean assessment scores for sustainability:
  - By region: African Countries 2.3; Asia Pacific Countries 2.6; Global / Mixed 2.5; JSA overall 2.4.
  - By classification: Fragile states 2.4; Non-fragile states 2.5; Global / Mixed 2.5; JSA overall 2.4.
- Survey results on perceived sustainability (% who state results will be ‘very sustainable’):
  - IMF Project Managers: In the short term (up to 1 year) 61%; medium term (1-5 years) 39%; longer term (5-10 years) 35%.
  - Recipient Authorities: In the short term (up to 1 year) 33%; medium term (1-5 years) 28%; longer term (5-10 years) 33%.
  - Survey sample sizes: IMF implementing staff (n=54); recipient authorities (n=39).
  - Reported margin of error examples: IMF managers survey margin of error at 95% confidence level = ±9%; recipient authorities longer-term figure 33% margin of error = ±13%.
- Main risks to sustainability reported by recipient authorities (ranked in survey responses):
  - Lack of sufficient funding once the JSA financing ends.
  - Lack of buy-in from government stakeholders.
  - Lack of buy-in from other stakeholders.
  - Lack of coordination with other development initiatives in the country.
  - Political barriers in country.
  - Economic barriers (including impact of COVID-19).
  - Legislative or regulatory barriers.
  - Difficulties retaining staff who received new knowledge and/or skills.
- Evidence limitations and suggested actions:
  - Reviewers’ scores and survey responses are primarily based on prerequisites for sustainability rather than retrospective evidence.
  - To improve evidence on sustainability and longer-term impacts, the IMF could commission follow-up research or require periodic post-completion reporting from recipient authorities.

*External Independent Evaluation of the Japan Sub-Account – Vol I | INTERNATIONAL MONETARY FUND*

### 2.7 LESSONS LEARNED

### 2.7 LESSONS LEARNED

### On improving relevance
- Sustained and intensive engagement by CD staff is particularly crucial in low-income, low-capacity countries to secure buy-in and ownership. The evaluation found an example where intensive engagement by the LTX turned initial weak ownership into stronger authority ownership as the project progressed.
- LTX modalities are the most obvious way to deliver intensive engagement; the same principle can apply to virtual or hybrid delivery. The evaluation cited a case where consistent and ongoing remote support by an IT expert helped a lower-middle-income African country establish a new IT system for tax administration despite capacity limitations at the recipient authority.
- IMF lending activities can be a powerful lever for achieving engagement and ownership by authorities because of complementary technical aspects and existing engagement mechanisms such as ECF. Conversely, loss of engagement on the lending side can have a knock-on effect on engagement with IMF CD. The evaluation advises teams to reflect these factors in assumptions and risk assessments about authorities’ readiness to engage with proposed TA.

### On improving coherence
- Administrative mechanisms and structures can enhance coherence and coordination where multiple TA interventions and providers operate simultaneously. The evaluation described a country where a PFM Reform Steering Committee chaired by the Deputy Prime Minister ensured high coordination between authorities, CD providers (including the IMF), and donor partners.
- In PFM, donor coordination was achieved by periodic consultative meetings; beneficiary–donor coordination was achieved by regular meetings between the steering-committee secretariat and donor representatives.

### On improving efficiency and effectiveness
- The optimal modality for delivering TA depends on practical issues (connectivity, staff capacity) and cultural aspects. During COVID some recipients rejected virtual TA on principle; in other cases virtual delivery proceeded but with negative impacts on efficiency and effectiveness, often due to difficulties building relationships remotely rather than technological issues.
- Within multi-country programs, small-scale TA projects—even with minimal expenditure or deliverables—can maintain relationships with authorities unwilling or unable to engage more substantially. The evaluation cited a country-project that was kept open despite lack of progress, allowing gradual resumption once authorities indicated readiness.

### Summary conclusions from the evaluation (main findings)
- JSA-funded TA and training are a key part of overall IMF CD and contribute greatly to developing capacity and strengthening institutions in Asia and some African countries, particularly fragile states. The evaluation noted JSA is the main source of funding for many CD activities including online training and support for JSA priority Southeast Asian countries: Cambodia, Lao PDR, Myanmar, and Vietnam.
- Through support for training, JSA contributed to developing online and blended (online, virtual, in-person) activities that have been highly effective and enabled continuity of delivery during the COVID-19 pandemic.
- The evaluation found JSA-funded CD was generally well aligned with IMF surveillance and lending, per the Fund’s 2018 strategy, while identifying scope to further enhance relevance, effectiveness and impact by tailoring activities to country needs and absorption capacity.
- The pandemic dominated the evaluation period; the Government of Japan accommodated necessary changes to JSA-funded programs. The Fund replaced in-person training with virtual and online courses and in-person TA with virtual missions; blended modalities reduced average costs while maintaining or improving participant satisfaction and learning outcomes.
- CD delivery recovered unevenly: training scaled up rapidly via ICD’s online program; TA recovered more gradually through virtual missions and resident LTXs, reaching pre-pandemic levels in many countries by end-2021. The evaluation observed TA recipients sometimes asked to take online courses before in-person TA missions, and in-person missions often followed by virtual consultations.
- Overall, JSA-funded programs scored highly for relevance and coherence. The evaluation reported the average for relevance and coherence overall was around 3.0 (on a scale from 1 to 4).
- Awareness of JSA funding among IMF project managers was very good, but more than a third of recipient authorities responding to the survey were not aware their JSA project was funded by the Government of Japan. Of those aware, two thirds learned this through discussions with IMF staff rather than branding or other sources. The evaluation suggests improving visibility through increased JSA ‘branding’ on project documents and clearer communication by IMF staff to counterparts.

### Effectiveness, RBM, and departmental differences
- Most JSA-funded programs were relatively effective, largely or partially achieving milestones, outcomes, and objectives, but with significant differences across sectors and countries.
- The evaluation reported average ratings by department: ICD 3.3, STA 2.7, FAD 2.3, and MCM 2.5.
- Drivers of effectiveness included:
  - country ownership and commitment to reform;
  - initial conditions in recipient country and agency (countries with better institutions and agencies with more resources used JSA-funded CD most effectively);
  - better internal coordination within the Fund between delivering CD departments and corresponding country teams.
- Program and project complexity generally reduced effectiveness by taxing authorities’ and staff’s implementation capacity.
- The evaluation cautioned against using the RBM framework to compare effectiveness across programs or countries without context: RBM proxies (objectives, outcomes, milestones) are set relative to expectations given country and agency capacity. The RBM database remains limited as a tool for assessing progress because milestones, outputs and outcomes are often adjusted during implementation.

### Multi-country programs
- Multi-country programs can be flexible and efficient, sometimes delivering returns to scale. Advantages cited:
  - launch CD country-level projects without prior full investment of time and resources required for single-country programs;
  - ability to shift resources across countries in response to changing priorities;
  - a single LTX can be shared by multiple countries, enhancing cost-effectiveness;
  - opportunities for peer learning across diverse countries.
- Challenges of multi-country programs:
  - complicate coordination within the Fund and can pose challenges to effectiveness, transparency and accountability;
  - difficulty identifying and hiring appropriate experts when recipient countries are heterogeneous;
  - program management difficulties, including uncertainty about availability of resources for country managers even when overall program budgets had underruns;
  - difficulty evaluating completed programs when some countries achieve results and others do not.
- The evaluation noted that improved use of CDMAP and its functionalities is expected to reduce information gaps on allocation of budgeted resources across country-projects.

### Efficiency, value-for-money (VFM), costs and modality changes
- Assessing efficiency and VFM of JSA-funded activities is difficult due to lack of complete cost data and difficulty monetizing outcomes. The evaluation reported systems for budgeting and monitoring costs were in transition to CDMAP during the period.
- Pandemic-driven budget underruns occurred in 2020 and 2021 as travel expenses collapsed. Blended delivery modalities reduced average costs per participant while maintaining or improving outcomes; surveys and interviews indicate staff and some recipients see benefits in blended modalities beyond cost reduction, due to more continuous interactions between recipients and Fund experts.
- Qualitative evidence suggests VFM improved since the pandemic as new delivery modalities are less costly and appear equally effective.

### Impact and sustainability
- Limited evidence was found that JSA-funded programs are generating wider impacts beyond initial outputs and outcomes. The evaluation attributed this largely to short timeframes, lack of routine post-completion data collection, and difficulties attributing effects to the JSA-funded intervention.
- Programs generally scored fairly low for impact: the evaluation reported an average across all projects for impact of 2.1.
- Sustainability of JSA-funded programs appears relatively modest overall, but better in Asian emerging economies than in low-income fragile states in Africa. The evaluation gave an average sustainability rating of 2.4 (out of 4) across all programs reviewed, with an average of 2.6 for Asia and the Pacific and an average of 2.3 for Africa. Higher sustainability ratings correlated with strong recipient authority buy-in, absorption of knowledge/tools/systems by organizations, and staff continuity allowing transmission of experience to colleagues.

*Source: External Independent Evaluation of the Japan Sub-Account – Vol I | International Monetary Fund*

### 11. We do not consider that it would be practical to try to optimize VFM by identifying programs for early

### 11. We do not consider that it would be practical to try to optimize VFM by identifying programs for early cancellation and reallocation of funds.

### Findings on early cancellation and interim assessments
- The vast majority of programs and country projects are relatively short, making early cancellation impractical given the processes and documentation required in both the Fund and JSA.
- In most cases, the Fund and JSA would need to prepare a thorough document to explain the cancelation to the recipient authority; the annual interim assessment currently being produced would not be adequate for that task.
- Reviewers found that the amount and quality of information provided in interim assessment reports was frequently limited, and would often not have supported a robust decision to cancel or reorientate the project at that stage.
- Multi-country programs allow projects that are not working to be effectively cancelled at the country project level without necessarily severing engagement with the country concerned.

### Recommendations (3.2 RECOMMENDATIONS)
- Overall observation:
  - JSA has supported the IMF in expanding the reach and impact of its CD, recognized as best in class on issues at the core of its mandate and expertise.
  - JSA-funded activities have been effective at building capacity in many countries, especially in Asia and parts of Africa.
  - There is scope to further improve the performance of the JSA and its programs; the following recommendations are made to the IMF and/or the JSA.

- 1. Maintain or adopt pandemic-catalysed innovations (Priority: High)
  - Preserve innovations and adaptations notably relating to technology and remote delivery across all relevant activities.
  - RCDCs played a heightened role enabling in-person and virtual TA support on a regular basis.
  - Except possibly for fragile states, JSA should consider prioritizing virtual and blended in-person/virtual activities to build capacity over one-off missions.
  - Brief missions are useful to prepare diagnostics; implementation generally requires regular follow-up for which virtual modalities are generally better suited because of costs and coordination.
  - The IMF should continue to favour in-person LTX engagement as a crucial delivery model for fragile states.

- 2. Training delivery: retain online and virtual formats (Priority: High)
  - Maintain progress made in using online and virtual courses rather than reverting fully to former modes of delivery.
  - Consider shifting to online formats (particularly MOOCs) for imparting basic general knowledge, including on policy issues such as financial programming.
  - Consider shifting to virtual mode of CD delivery for upper-middle-income countries, and for lower-middle-income countries with good connectivity.
  - Virtual and in-person training is more effective and more likely to lead to institutional strengthening when delivered to teams working on corresponding issues in a single country rather than off-site to audiences from many countries.
  - Such training facilitates integration with TA and makes for a less strict distinction between training and TA.
  - Regional training permits peer learning and helps to build networks.

- 3. Seek more and earlier inputs from recipient authorities (Priority: High)
  - Before approving a project, JSA could require proposal documentation to include a brief explicit official request by recipient authorities to confirm buy-in.
  - Documentation should include confirmation from IMF staff that other donors and implementing organizations have been consulted to identify or rule out opportunities for synergies or joint working.
  - Selection of LTXs should consider not only technical expertise but also the rapport they can build with recipient authorities; authorities could have a greater say in drafting the LTX’s TOR.
  - JSA could require that recipient authorities from middle-income countries requesting LTX support contribute to its funding (e.g., housing and other local costs) to indicate commitment and defray costs.

- 4. Consider reallocation from Asian emerging market economies to other regions (Priority: High)
  - Many recipient countries in Asia have attained middle and even upper-middle-income status; human capital and state capacity have improved, diminishing the value added of CD and creating an opportunity to reassign resources to countries in greater need.
  - Reallocation could be done without sacrificing the scale of activities in Asian countries if the Fund delivers more CD virtually in countries with good connectivity.
  - Absorption capacity in low-income and fragile states remains an issue, but reallocating some resources currently spent in Asia to African fragile states may be a risk worth taking.
  - The Fund and JSA could consider creating a joint IMF-JSA facility similar to the Infrastructure Governance Facility (IGF) to allow dedicated resources for CD in fragile states.

- 5. Structure multi-country programs to balance flexibility, transparency, and manageability (Priority: Medium)
  - Consider limiting the number of country projects within a single program and ensure recipient countries are relatively similar in profile and need.
  - Provide some indication of how resources are allocated across different country-projects at program inception and increase accountability for allocation over the program life.
  - Consider restrictions on coverage (e.g., not combining projects across regions or countries with very different initial capacity).
  - Set an indicative budget for each country at the outset and establish explicit criteria for when to reallocate resources from one country to another.

- 6. Raise awareness that JSA funding is from the Government of Japan (Priority: Medium)
  - IMF implementation staff could be required to highlight the source of the funding when engaging with recipient authorities and in-country IMF colleagues.
  - Enhance JSA ‘branding’ presence not just on final project/program reports, but also on proposals and planning documentation.
  - Consider holding specific workshops or sessions during Annual Meetings showcasing JSA work to Fund staff and recipient countries.

- 7. Gather routine evidence on de facto sustainability and impacts post-implementation (Priority: Low)
  - Carry out periodic follow-up surveys with recipient authorities to gather information on post-completion results (for example through an e-survey similar to the one employed by the current evaluation).
  - A more modest approach would require authorities to track and provide a statement on the results of each project periodically (for at least a year) after their completion.
  - Ensure program documentation includes up-to-date contact information for the counterpart person or unit in each country, both during and after implementation.

### Implementation considerations and operational notes
- Virtual modes of delivery are emphasized for cost, coordination, and follow-up, except where in-person LTX engagement is critical (fragile states).
- Reallocation of resources should weigh improved capacity in many Asian countries against unmet needs in African fragile and low-income states; operational approaches include joint facilities and targeted virtual delivery.
- Strengthening documentation at project approval (explicit recipient requests, consultations with other donors, indicative country budgets) is recommended to improve accountability and enable better reallocation decisions within multi-country programs.

---


_Source: https://www.imf.org/-/media/files/capacity-developement/jsa-annual-reports/jsa-eie-2024.pdf_