## D4D mid-term evaluation — staff response and action plan ## Source details **Canonical URL:** [D4D mid-term evaluation — staff response and action plan](https://www.imf.org/-/media/files/capacity-developement/d4d/mid-year-check-in-meeting/d4d-mid-term-evaluation-staff-response-and-action-plan.pdf) ## Other formats - [Markdown version](/-/media/files/capacity-developement/d4d/mid-year-check-in-meeting/d4d-mid-term-evaluation-staff-response-and-action-plan.pdf.md) - [Structured JSON version](/-/media/files/capacity-developement/d4d/mid-year-check-in-meeting/d4d-mid-term-evaluation-staff-response-and-action-plan.pdf.json) --- ### Executive summary — purpose, scope, and context - D4D Fund: a multi-donor initiative financing capacity development (CD) to improve data-driven macroeconomic policy decisions, primarily in low-income and lower-middle-income countries (LICs and LMICs). - Evaluation coverage: Phase 1 of D4D Fund operations (June 2018 – April 2021), assessed using OECD-DAC criteria and the IMF’s Updated Common Evaluation Framework (CEF). - Budget commitments as of June 2021: nine partners committed to the US$33.1 million five-year budget (including a 7-percent IMF Trust Fund Management Fee). - D4D modules and focus: - M1 – Addressing Data Needs and Quality Concerns (submodules: RSS, ESS, PSDS/GFS) - M2 – Financial Access Survey (FAS) - M3 – Online Learning (eight online fundamental statistics courses) - M4 – Statistical Information Management (SIM practices and environments) - Target countries: mainly LICs and LMICs worldwide, including fragile states. ### Key findings by OECD‑DAC criterion - Relevance - D4D Fund found to be highly relevant; IMF-wide processes generally ensure CD aligns with recipient countries’ needs and IMF priorities. - More than 90 percent of surveyed stakeholders agreed D4D-funded CD corresponded to authorities’ needs and priorities. - Evidence that country authorities were only partially involved in CD design in some cases, hindering ownership. - Effectiveness - Largely effective in data compilation and dissemination and knowledge transfer. - "80 percent of the rated milestones were largely or fully achieved during the evaluation period." - Main obstacles: external circumstances and recipient-side constraints (high workload, lack of senior management buy-in); COVID-19 hindered effectiveness and required extra project-team effort. - M1 case studies suggested longer-term planning and closer post-mission follow-up could improve effectiveness. - Impact and Sustainability - The Fund needs greater effort to target and track results at the level of data usage (not just compilation/dissemination) to achieve the goal of improving data-driven macroeconomic policy decisions. - CD impact most evident where data outputs translated into actual data usage and where strong external or internal demand existed. - Risks to sustainability: lack of senior management involvement, limited absorption capacity, insufficient financial and human resources, and staff turnover. - Coherence - Coordination with other IMF activities and development partners generally good but with room for improvement. - Opportunities to strengthen synergies (e.g., greater use of FAS in IMF technical assistance; better M3–M1 sequencing). - Efficiency - Despite limited budgetary data and low budget execution, D4D Fund considered efficient with high value for money in CD delivery. - Cost-effectiveness assessment impeded by former project management system (CD-PORT) and budget allocation practices (by workstream rather than country/mission). - Low budget execution partially due to COVID-19; no major timeliness issues in CD delivery. ### Quantitative assessment (evaluation team scores; scoring scale: 1 = Not Achieved; 2 = Partially Achieved; 3 = Largely Achieved; 4 = Fully Achieved) - Relevance: - D4D-wide assessment 3 - Average case study rating 3.5 - Overall rating 3.25 - Effectiveness: - D4D-wide assessment 3 - Average case study rating 2.5 - Overall rating 2.75 - Impact: - D4D-wide assessment 2 - Average case study rating 2.5 - Overall rating 2.25 - Sustainability: - D4D-wide assessment 2 - Average case study rating 2.5 - Overall rating 2.25 - Coherence: - D4D-wide assessment 3 - Average case study rating 2.5 - Overall rating 2.75 - Efficiency: - D4D-wide assessment 3 - Average case study rating 3.5 - Overall rating 3.25 ### Prioritized recommendations (seven; potential impact and effort/cost preserved) - Priority 1 (Impact and sustainability) - Move from a “CD delivery” to a “change management” approach to increase likelihood that short-term CD translates into longer-term organizational change and policy improvements (especially for M1 and M3). - Potential impact: High; Effort/cost: Medium. - Priority 2 (Relevance/internal coherence/effectiveness/impact/sustainability) - Require an explicit needs assessment for data use during CD planning and design for: CD recipients (M1 and M3); area departments (M1 and M3); third-party data users (all modules). - Potential impact: High; Effort/cost: Medium. - Priority 3 (Relevance/effectiveness/impact/sustainability) - Require more systematic assessment of ownership and absorption capacity, particularly for M1. - Potential impact: Medium; Effort/cost: Low. - Priority 4 (Impact) - Improve impact monitoring and evaluation at the level of data users (include user-level indicators in strategic log frame; encourage systematic reporting; increase post-mission follow-up monitoring; specify monitoring responsibilities; allocate share of D4D budget for monitoring). - Potential impact: High; Effort/cost: High. - Priority 5 (External coherence) - Enhance coordination and strengthen synergies with other development partners. - Potential impact: Medium; Effort/cost: Medium. - Priority 6 (Efficiency and effectiveness) - Further tailor the RBM framework to the D4D Fund. - Potential impact: Medium; Effort/cost: Medium. - Priority 7 (Efficiency and effectiveness) - Continue developing and institutionalizing the blended (online and in-person) CD delivery model. - Potential impact: Low; Effort/cost: Low. ### Methodology, case studies, and limitations - Evaluation design: - D4D-wide assessment; bottom-up assessment of five M1 country projects (Uzbekistan ESS (6 missions); Zimbabwe PSDS/GFS (2 missions); Ghana PSDS/GFS (2 missions); Vietnam RSS (7 missions); Sri Lanka RSS (2 missions)); M2 and M3 assessed via strategic log frames. - Triangulation of RBM portfolio analysis, interviews, stakeholder survey, desk research, mission report analysis, in-depth interviews, course evaluation reports, and course enrollment data. - Survey response rates: - General M1 survey 9.7 percent; M3 survey 8.3 percent; partial responses raised M3 to 12.6 percent. - Limitations: - RBM data coverage limited; former project management and budget systems (CD-PORT) impeded detailed cost analyses; COVID-19 affected budget execution and implementation modalities. - Evaluation avoided conclusions based on single information sources. ### Budget, expenditures, and budget execution (exact figures preserved) - Total planned budget over five years: 33 mln - Expenses as of end-April 2021: 9,465,600 - Summary by (sub)module (originally planned budget; Budget FY19-FY21Q1; Expenses FY19-FY21; Total budget use (in %); Budget use FY19-FY21 (in %)): - Module 1 – Addressing Data Needs and Quality Concerns: 19-21 mln; 5,033,929; 4,243,012; 21%; 84% - Real Sector Statistics: 1,738,629; 1,596,263; 92% - External Sector Statistics: 917,481; 940,973; 103% - Fiscal and Debt Reporting: 2,377,819; 1,705,776; 72% - Ad-hoc Advisory Services: -; 39,804; - - Module 2 – Financial Access Survey: 4-5 mln; 1,492,170; 1,334,343; 29%; 89% - Module 3 - Online Learning: 5-7 mln; 2,188,579; 2,473,785; 42%; 113% - Module 4 – Statistical Information Management: 1-1.5 mln; 212,279; 114,124; 9%; 54% - RM Project Manager: 485,762; 681,091; 140% - Subtotal: 9,412,719; 8,846,355; 94% - Trust Fund Management Fee (7%): 658,890; 619,245; 94% - Total: 33 mln; 10,071,610; 9,465,600; 29%; 94% - Observations: - Budget execution was low (budget use at the end of the evaluation period was below one third of the total budget). - Module 3 and Module 2 showed higher budget utilization (42 percent and 29 percent, respectively) versus Module 1 (21 percent). - Resources were transferred to Module 3 from other modules (e.g., US$0.5 million from M4). ### RBM framework assessment and monitoring findings - RBM system under CD-PORT: - Project managers rate outcomes and milestones on a scale of 1–4. - Under CD-PORT the D4D Fund had four “projects” corresponding to submodules/workstreams; objectives and outcomes were often formulated at a high, abstract level. - Result: many project managers did not use RBM actively; only 2 out of 65 country-specific projects had rated outcomes as of April 2021 (roughly 3 percent of the portfolio). - Milestone ratings were available for 35 projects (48 percent of the portfolio), with distribution (N = 118): - 1 = not achieved: 2% - 2 = partially achieved: 18% - 3 = largely achieved: 18% - 4 = fully achieved: 62% - "80 percent of the rated milestones were largely or fully achieved during the evaluation period." - Transition and improvements: - IMF switched from CD-PORT to CD-MAP in May 2021; STA RBM Catalogue expanded (Catalogue 2 (May 2020) → Catalogue 3 (July 2021)): - Number of objectives: 9 → 25 - Number of outcomes: 120 → 176 - Number of indicators: 521 → 840 - Expectation: CD-MAP and a revised RBM Catalogue allow more granular country-level outcomes and indicators; recommendation to further tailor RBM Catalogue to D4D needs. ### Impact, sustainability, and lessons learned - Impact - Surveys and IMF staff largely positive about potential impact given better data availability; however, D4D did not systematically monitor data use. - Examples of impact during project lifetime: - First-time publication of external sector statistics increased transparency and contributed to Eurobond issuance. - New public sector debt statistics allowed IMF country team to update Debt Sustainability Analysis (DSA) before formal dissemination. - Monitoring gap: RBM and strategic log frame tracked compilation/dissemination but not data use by policy makers, IMF country teams, or third parties. - Sustainability determinants - Senior management involvement, resource availability (financial and human), political commitment, continuity of compiling teams, continuity of IMF CD teams, coordination with area departments, and frequent/timely follow-up. - Survey evidence: almost 20 percent of recipients indicated insufficient political support; more than 40 percent noted insufficient financial resources for staff development. - Lessons learned (factors contributing to impact and sustainability) - Stable compiling teams with low staff turnover. - Continuity of IMF CD teams. - Strong coordination with IMF area department teams. - Senior management support and allocation of sufficient national resources. - Frequent and timely follow-up interactions. ### Coherence — internal and external coordination - Internal coherence - D4D-funded CD generally coherent with IMF surveillance and lending; around 85 percent of provider-respondents indicated CD was closely linked to IMF program or surveillance work. - Opportunities: strengthen M3–M1 sequencing (e.g., more cases where M3 course participation precedes M1 TA). - External coherence - IMF comparative advantage in macroeconomic statistics recognized; complementarity with World Bank, UN, OECD acknowledged. - Survey evidence: 80 percent of recipients and 90 percent of providers believed D4D-funded CD and CD delivered by other DPs reinforced each other. - Recommendations include systematic overviews of other DPs’ CD activities, increased sharing of TA reports, and institutionalized coordination with the World Bank. - Planned/immediate actions: - Creation of a Financial Access Survey Advisory Group. - Stronger collaboration with the World Bank and PARIS21. - Beginning in 2024, D4D Fund Annual Reports to feature more systematic reporting on complementarity with other DPs. ### IMF staff response and priority actions (selected) - Implementation context - Actions to be undertaken primarily by STA in close collaboration with ICD and other Capacity Development Departments and Area Departments. - Most actions to be implemented during remaining cycle of D4D Fund Phase I and proposed D4D Fund Phase II to start in July 2025. - Selected planned actions by recommendation: - Recommendation 1 (Change management) - CDMAP shifted planning to three-year MTW plans; STA to discuss roadmaps earlier with authorities and ADs; STA to encourage documentation of compilation processes; develop Compilers’ Hub; integrate practical compilation exercises into workshops and online learning. - Recommendation 2 (Needs assessment) - STA to consult extensively with authorities at design phase; form FAS Advisory Group under M2; encourage national user consultations for key CD projects. - Recommendation 3 (Ownership and absorption) - STA to systematically engage with authorities on pressing data gaps, resource needs and absorptive capacity; include use case analysis in diagnostic missions. - Recommendation 4 (Impact monitoring) - STA to explore inclusion of qualitative information on data use in progress reports; outreach to policymakers and IMF mission chiefs to collect examples of data use; STA to continue monitoring internal usage of FAS data in staff products. - Recommendation 5 (Coordination with DPs) - Leverage Clearinghouse for Financing Development Data; implement Staff Operational Guidance on Dissemination of Capacity Development Information (2022); institutionalize mechanisms with World Bank, strengthening design-stage engagement. - Recommendation 6 (RBM tailoring) - Design Phase II strategic log frame in consultation with ICD; draw monitoring inputs from CDMAP; implement RBM data completeness monitor. - Recommendation 7 (Blended delivery) - Blended learning established; roll out blended courses piloted during FY23; consider M3 course certification as precondition for follow-up TA in some cases; expand blended modality to scheduled TA activities. ### Action planning and monitoring - STA will propose to the D4D Fund Steering Committee a share of the Phase II budget for systematic monitoring, including post-project surveys; proposal to be articulated in the Program document for endorsement. - Under CDMAP, STA project managers are responsible for active project monitoring during the project life cycle; CDMAP has shifted planning toward medium-term work plans and three-year roadmaps. *Source: D4D Fund mid-term evaluation — Staff Response and Action Plan (IMF / SEO Amsterdam Economics).* ### Part I ............................................................................................ page 5 ### MID-TERM EVALUATION OF THE DATA FOR DECISIONS FUND (D4D) — FINAL REPORT ### Executive Summary — purpose, scope, and context - The Data for Decisions (D4D) Fund is a multi-donor initiative financing capacity development (CD) to improve data-driven macroeconomic policy decisions, primarily in low-income and lower-middle-income countries (LICs and LMICs). - The evaluation assesses Phase 1 of D4D Fund operations (June 2018 – April 2021) using OECD-DAC criteria and the IMF’s Updated Common Evaluation Framework (CEF). - The evaluation was commissioned by the International Monetary Fund and authored by SEO Amsterdam Economics (authors listed). Publication header shows "AMSTERDAM, FEBRUARY 23". - As of June 2021, nine partners committed to the US$33.1 million five-year budget (including a 7-percent IMF Trust Fund Management Fee). - D4D Fund modules and focus: - M1 – Addressing Data Needs and Quality Concerns (submodules: RSS, ESS, PSDS/GFS) - M2 – Financial Access Survey (FAS) - M3 – Online Learning (eight online fundamental statistics courses) - M4 – Statistical Information Management (SIM practices and environments) - Target countries: mainly LICs and LMICs worldwide, including fragile states. ### Key findings by OECD-DAC criterion - Relevance - D4D Fund found to be highly relevant; IMF-wide processes ensure CD aligns with recipient countries’ needs and IMF priorities. - Stakeholders widely agreed D4D-funded CD corresponded to national authorities’ priorities and was sensitive to country contexts. - Some evidence (survey respondents and M1 case studies) that country authorities were only partially involved in CD design, hindering ownership in some cases. - Effectiveness - Generally positive findings from interviews, surveys, and case studies. - Largely effective in data compilation and dissemination, and knowledge transfer to recipients. - Projects generally appeared on track to achieve stated objectives; available RBM data suggest most milestones were largely or fully achieved (limited RBM coverage cautioned). - Projects’ objectives and outcomes in the IMF-wide RBM framework were too high-level for D4D project management (improvement likely since RBM Catalogue expansion). - Main obstacles to outcomes: external circumstances and constraints on authorities’ side (high recipient workload, lack of senior management buy-in); COVID-19 hindered effectiveness and required additional project-team effort. - M1 case studies indicated longer-term planning and closer post-mission follow-up could improve effectiveness. - Impact and Sustainability - Greater effort needed to target and track results at the level of data usage (in addition to compilation and dissemination) to achieve the Fund’s aim of improving data-driven macroeconomic policy decisions. - CD impact most evident where data outputs translated into actual data usage and where strong external or internal demand existed. - Risks to sustainability include lack of senior management involvement and limited absorption capacity; when senior management prioritized CD objectives, resources and progress tracking improved. - Coherence - Coordination with other IMF activities and development partners (DPs) was generally good but with room for improvement in some case studies. - D4D-funded CD was generally coherent with IMF surveillance and lending and with other IMF CD activities; opportunities to strengthen communication and synergies (e.g., greater use of FAS in IMF technical assistance; better M3–M1 synergies). - Complementarity with other CD providers generally good; IMF’s comparative advantage in macroeconomic statistics recognized by DPs; managers actively sought DP coordination but further enhancement possible. - Efficiency - Despite limited budgetary data, evaluation found D4D Fund to be efficient. - Cost-effectiveness assessment difficult due to former project management system (CD-PORT) and budget allocation practices (by workstream rather than country/mission). - Case studies provided multiple examples of cost reduction measures; quality of CD delivery and knowledge transfer suggested high value for money. - Low budget execution partially due to COVID-19; no major timeliness issues in CD delivery. ### Quantitative assessment (evaluation team scores) - Table S.1 scores (Source: SEO Amsterdam Economics; scoring scale: 1 = Not Achieved; 2 = Partially Achieved; 3 = Largely Achieved; 4 = Fully Achieved): - Relevance: D4D-wide assessment 3; Average case study rating 3.5; Overall rating 3.25 - Effectiveness: D4D-wide assessment 3; Average case study rating 2.5; Overall rating 2.75 - Impact: D4D-wide assessment 2; Average case study rating 2.5; Overall rating 2.25 - Sustainability: D4D-wide assessment 2; Average case study rating 2.5; Overall rating 2.25 - Coherence: D4D-wide assessment 3; Average case study rating 2.5; Overall rating 2.75 - Efficiency: D4D-wide assessment 3; Average case study rating 3.5; Overall rating 3.25 ### Recommendations — prioritized, actionable, and cost-considered - Seven prioritized recommendations with potential impact and effort/cost assessments (summarized highlights): 1. (Priority 1) Impact and sustainability — Move from a “CD delivery” to a “change management” approach to increase likelihood that short-term CD translates into longer-term organizational change and policy improvements (especially for M1 and M3). Potential impact: High; Effort/cost: Medium. 2. (Priority 2) Relevance/internal coherence/effectiveness/impact/sustainability — Require an explicit needs assessment for data use during CD planning and design for: CD recipients (M1 and M3); area departments (M1 and M3); third-party data users (all modules). Potential impact: High; Effort/cost: Medium. 3. (Priority 3) Relevance/effectiveness/impact/sustainability — Require more systematic assessment of ownership and absorption capacity, particularly for M1. Potential impact: Medium; Effort/cost: Low. 4. (Priority 4) Impact — Improve impact monitoring and evaluation at the level of data users (include user-level indicators in strategic log frame; encourage systematic reporting in progress reports; increase post-mission follow-up monitoring, especially for Module 1; specify monitoring responsibilities; allocate share of D4D budget for monitoring). Potential impact: High; Effort/cost: High. 5. (Priority 5) External coherence — Enhance coordination and strengthen synergies with other development partners. Potential impact: Medium; Effort/cost: Medium. 6. (Priority 6) Efficiency and effectiveness — Further tailor the RBM framework to the D4D Fund. Potential impact: Medium; Effort/cost: Medium. 7. (Priority 7) Efficiency and effectiveness — Continue developing and institutionalizing the blended (online and in-person) CD delivery model. Potential impact: Low; Effort/cost: Low. - Note: Recommendation 7’s potential impact is scored as low because D4D has already made good progress with blended CD delivery; further work is institutionalization. ### Methodology, case studies, and supplementary notes - Evaluation design: - D4D-wide assessment, bottom-up assessment of M1 country projects (five M1 case studies: Uzbekistan ESS (6 missions); Zimbabwe PSDS/GFS (2 missions); Ghana PSDS/GFS (2 missions); Vietnam RSS (7 missions); Sri Lanka RSS (2 missions)), and assessments of M2 and M3 guided by strategic log frames. - Triangulation of data sources: RBM portfolio analysis, interviews with stakeholders, large stakeholder survey (noting low response rates: general M1 survey 9.7 percent; M3 survey 8.3 percent; partial responses raised M3 to 12.6 percent), desk research, mission report analysis, in-depth interviews, course evaluation reports, and first-hand course enrollment for M3 assessment. - Limitations: - RBM data coverage limited; former project management and budget systems (CD-PORT) impeded detailed cost analyses; COVID-19 affected budget execution and implementation modalities. - Survey response rates and RBM coverage limitations led the evaluation team to avoid conclusions based on single information sources. *Final Evaluation Report by SEO Amsterdam Economics; commissioned by the International Monetary Fund.* ### 2.1 Relevance ### 2.1 Relevance ### Overview of processes and demand - There are appropriate IMF-wide processes in place to ensure that CD is relevant to recipient countries' needs. - A key first step in the project selection and approval process is that country authorities submit a formal CD request to the D4D Fund team. - Requests can emerge from prior STA missions or from IMF area departments in the context of surveillance or lending activities; some projects therefore may not be strictly “demand driven.” - The Program Document requires selection of D4D projects to be based on a needs assessment and expects work plans to be formulated in coordination with country authorities. - The needs assessment is supposed to consider: (i) demand from countries, (ii) policy relevance, (iii) identified data gaps and data quality concerns, (iv) complementarity with CD provided by RCDCs and others, and (v) the need for flexibility to react to a country’s capacity and readiness to accept CD. - For M1, the Program Document states that “country-owned work plans to achieve the intended outcomes [will] be established at the first TA mission and/or workshop and agreed with the country authorities to ensure ownership.” ### Survey and stakeholder perceptions - Survey respondents largely saw D4D-funded CD as relevant: - More than 90 percent agreed or strongly agreed that D4D-funded CD corresponded to the needs and priorities of the authorities. - A small majority of surveyed providers indicated that the design of the provided CD took the country’s absorption capacity and ownership into account; around 15 percent deemed that this was done insufficiently. - A small majority of both recipients and providers indicated that recipient institutions had sufficient opportunity to influence the design of the CD project; only 10 percent thought otherwise. - Survey sample and figure notes: - Source: SEO Amsterdam Economics, based on a survey among 69 stakeholders (recipients, providers and funding partners). - Chart sample sizes shown: N=38, N=38, N=39 for respective questionnaire items. ### Case study findings by module - M1 (national statistics and mission-based CD) - Most M1 case studies showed indications that D4D-funded CD was relevant for local authorities, although ownership by country authorities could be improved. - In two case studies, improvements of national statistics compilation were driven by national priority explicitly expressed by national authorities. - In three out of five M1 case study countries, interviewees were not aware of whether and when a next mission was planned; compilers sometimes were not fully aware of RBM frameworks and strategic log frames used. - CD design was generally informed by experts’ initiative and STA information on data gaps, discussions with IMF area departments, and in some cases by direct recipient involvement validating expert proposals. - M2 (FAS — Financial Access Survey) - The FAS was relevant for global monitoring of financial inclusion and provided a global public good, aligning with G-20 and SDG priorities. - The FAS responded to a call from the United Nations Advisors Group on Inclusive Financial Sectors in 2008. - Before D4D coverage began in 2018, the FAS had been conducted annually since 2009 with financial support from the Ministry of Foreign Affairs of the Netherlands and the Bill & Melinda Gates Foundation (2013-14). - When the D4D Fund took over the FAS in 2018, it did not carry out a further needs assessment among individual national authorities; a further assessment might be useful for strategic direction. - M3 (online learning) - Users praised online courses for meeting needs and learning goals and valued that courses were opened to the broader public. - The broader public (e.g., local journalists and local researchers) may lack sufficient background knowledge to fully benefit; data showed these participants were less successful and tended to drop out more quickly. - Online training was deemed most relevant for participants from national authorities. - M4 and resource allocation - The Fund reallocated resources from M1 and M4 to online learning (M3) during the COVID-19 pandemic, accelerating online course development. ### Relevance and ownership issues - Country authority involvement and ownership - Although formal CD requests and acceptance of work plans are required, terms of reference and work plans are typically written by the IMF team itself; this can leave authorities only “made aware” rather than actively owning objectives. - Case studies suggest active involvement of authorities in design increases ownership, resource devotion, and contribution to implementation. - End-user needs assessment gaps - The D4D Fund did not yet systematically assess the needs of modules’ end-users of national statistics. - Project proposals typically did not contain a clear assessment of how data might be used by (i) the authorities for policy purposes, (ii) IMF country teams, and (iii) relevant third-party users (e.g., rating agencies, investment banks, researchers, development partners, NGOs). - Recommendation (referenced): require project managers to explicitly include an end-user needs assessment for these three groups in project proposals (See Recommendation 2 for further details). ### Flexibility, COVID-19 response, and delivery modalities - Flexibility and agility - The Fund’s modular structure and funding allocation by module (funds not pre-allocated by country) enhanced flexibility. - The Steering Committee (SC) approval process for changes to the work plan was short: five business days at most, on a lapse-of-time basis. - This agility allowed prompt responses to urgent operational CD requests and a focus on activities deliverable online. - COVID-19 delivery - During the COVID-19 pandemic, the Fund continued to offer support through online methods; this was generally suitable given constraints on in-country delivery. - Most case study recipients and providers considered remote CD generally less effective than in-country CD: online activities were more cumbersome, more time consuming, and hampered by technical shortcomings. - The possibility of “blended CD,” combining remote CD with in-country CD, was seen as more promising. - Outcome note - Despite increased need for basic capabilities following the pandemic, the Annual Report FY22 noted that the share of statistics projects in the total number of IMF projects had continued to stagnate in FY22. ### Lessons learned for relevance (Box 2.1) - Actively involving country authorities at the design stage improves the likelihood of CD being effective and sustainable - When country authorities were involved in actively setting project objectives, they were more likely to “own” them, devote resources to them, and contribute to their implementation. - Information and recommendations tailored to the country were appreciated by country authorities - Authorities particularly appreciated recommendations that were actionable and could help address challenges. - An expert from the region who understood the political economy enabled better tailored TA in one country. - Data compilation approaches tailored to the country’s context were appreciated, especially when they took into account specific constraints and needs. *D4D Fund mid-term evaluation — Section 2.1 Relevance.* ### 2.2 Effectiveness ### 2.2 Effectiveness ### Overall assessment - Evaluation findings based on the RBM framework, progress reports, interviews, surveys and case studies were generally positive about the D4D Fund's effectiveness. - The D4D Fund’s was largely effective in terms of data compilation and dissemination, as well as knowledge transfer to recipients, although constraints on the authorities’ side hindered effectiveness in some of the M1 case studies. - Respondents to the D4D stakeholder survey were generally positive about the D4D Fund’s overall effectiveness in terms of data compilation and dissemination, offering higher added value compared to other development partners (DPs) in the same field. - Nearly all recipients and providers were positive about the D4D Fund’s overall contribution to compiling and disseminating economic statistics. - No real differences between the individual level (i.e., strengthened knowledge, skills and practices of staff members) and the institutional level (i.e., institutional ability to compile and disseminate statistics) were observable, suggesting that the gained knowledge is in fact put to practice and institutionalized. - Nearly all recipients and donors also indicated that the D4D-funded CD met (or exceeded) their expectations when it comes to overall effectiveness. ### Stakeholder survey and participation - Survey conducted among 69 stakeholders (recipients, providers and funding partners). - Figure details: N=24 and N=40 (presented in the evaluation materials) for provider/recipient response groupings in the assessment of statements such as: - "Strengthened the knowledge/skills of the local staff" - "Has improved the way local staff compile or disseminate..." - "Resulted in better alignment of national methodologies..." - "Has strengthened the institution’s ability to compile ..." - "Has strengthened the institution’s ability to disseminate ..." - Most recipients perceived the added value of the D4D Fund to be higher than that of other DPs in the same field. ### RBM data, milestone and outcome ratings - The available RBM data are limited and should be interpreted with caution. - The D4D Fund was expected to use the IMF-wide RBM system for reporting and monitoring outcomes (and milestones linked to outcomes) of CD projects, where IMF project managers rate outcomes and milestones on a scale of 1-4. - Milestone ratings were available for only half of the projects, while outcome ratings were available for a much smaller share of the projects. - Migration to a new monitoring system (CD-MAP) partly contributed to limited availability of ratings. - "80 percent of the rated milestones were largely or fully achieved during the evaluation period." - Progress reports documented progress for each (sub-)module but did not explicitly summarize performance against RBM or Strategic Log Frame outcome targets. ### Case study findings and variations by module - Generally, more than half of the analysed case studies showed that the projects were on track to meet their objectives. - Several M1 case studies: CD contributed to improvements in staff capacity and in the compilation of data, resulting in largely or fully achieved outcomes. - Reported staff capacity improvements included: developing source data, applying new compilation methods, and understanding limitations of source data. - Reported data compilation improvements included: improved consistency among different data sets and implementation of new legal frameworks or regulation to accommodate data collection and compiling needs. - M2: two of the objectives were fully achieved (namely, the continuous production of the FAS and the provision of the monitoring basis for the UN’s SDG indicator on financial inclusion), while the other two were partly and largely achieved, respectively. - M3: objectives and outcomes were largely achieved, although constraints were present, such as IT complications in LICs. - In three other M1 case studies, external circumstances were the main obstacles to achieving outcomes; in three M1 case studies, outcomes were only partly achieved on average. - Constraining factors included: high workload for recipient staff (mainly compilers), lack of active buy-in from senior management, noncompliance by reporters, other priorities at the authority, and ongoing staff rotation at the recipient authorities’ side. ### COVID-19 pandemic impact and CD modality adjustments - The COVID-19 pandemic hindered effectiveness in several respects: - "Lockdowns turned data compilation much more difficult and costly, especially in LLMICs. The availability and quality of data sources suffered as surveys were interrupted and administrative processes came to a stop." - In 2020 there was a shift of resources towards pandemic control activities in many LLMICs. - The D4D Fund made extra efforts to ensure continuation of CD delivery; Module 2 required extra effort to ensure continuation of the FAS amid reporting issues due to lockdowns. - A large share of D4D-funded CD had to be transformed to an online modality; virtual missions/training courses were appreciated but lost out to the in-person equivalent according to recipients and providers. - Remote CD was the most effective temporary option during the pandemic to ensure continued TA, but respondents preferred face-to-face methods for knowledge transfer (confirmed in M2 assessment and M3 online-course participant survey). ### CD modalities and effectiveness - The diversity of CD modalities improved effectiveness: - Modalities used included workshops, seminars, in-country and remote TA and training courses. - Remote CD: flexible, easily spread out in time, and useful to ensure periodic engagement between missions; disadvantages include screen fatigue, lack of attention, and network issues. - In-country CD: allowed familiarization with experts, engagement with key decision-makers, and in-country workshops. - Workshops and seminars: enable meeting peer compilers and CD providers. - Peer study visits (currently not part of the IMF approach) were identified by recipients as allowing learning from regional peer compilers. - STA experimented with “blended” or synchronized CD, combining remote and offline activities. - CD recipients and providers in three M1 case studies experienced good levels of communication and understanding, supporting tailored guidance. ### Role of IMF technical expertise - The strong technical expertise of IMF providers was often seen as an important contributing factor to overall effectiveness. - CD was effective in covering legal, operational, technical, methodological, analytical and dissemination aspects needed to compile topical statistics per relevant statistical guidelines. - TA recipients lauded the expertise, knowhow and ability to find bespoke solutions to country-specific issues; some recipients indicated they would not have achieved as much without the experts’ help. ### Lessons learned for effectiveness (Box 2.2) - Blending different complementary CD modalities can help improve effectiveness: - Short-term missions provide opportunities for focused and tailored knowledge transfer. - Regional workshops and peer learning through country visits offer opportunities to develop and exchange knowledge and learn from other countries’ experiences. - Remote CD works well when in-person CD is difficult or impossible (e.g., during the COVID-19 pandemic); it helps ensure periodic engagement with authorities between missions and follow up on recommendations, but common disadvantages include screen fatigue, lack of attention of participants, and network issues. Stakeholders widely agreed that remote CD should only be used as a complement, and never a full substitute, for in-person CD. - Factors that stood out in case studies as contributing to the effectiveness of CD: - Robust collection of source data (legally underpinned); - Engagement with stakeholders and knowledge of user needs; - Minimum level of compiler skills (as one or two key compilers often act as main drivers/implementers); - Mission schedule known a certain number of months in advance, and next mission announced before the end of current mission; - In-country CD, supported by other modalities; - Longer missions (e.g., two weeks rather than one week); - Higher frequency of missions; - Timely (translated) reports. *Source: D4D Mid-term Evaluation — 2.2 Effectiveness.* ### 2.3 Impact and Sustainability ### 2.3 Impact and Sustainability ### Impact - Both survey respondents and the IMF were largely positive about the potential impact of CD projects, since they successfully provided policy makers with better data. - Survey evidence: survey among 69 stakeholders (recipients, providers and funding partners) showed a lower percentage of respondents reported that CD projects had made some impact thus far, but most respondents predicted that the projects were likely to have an impact within the next few years. - Key stakeholder perspectives and assertions: - An STA representative: “based on many years of experience in interacting with country authorities through economic surveillance and policy making, STA remains profoundly convinced that this assumption [that disseminating better quality, more timely, higher frequency data in key areas of economic statistics has a positive impact on economic analysis and decision making], which is reflected in the RBM framework and the D4D Fund’s strategic log frame, remains valid.” - A key IMF stakeholder: “the D4D Fund should not exist if it produces data that nobody uses.” - D4D Fund stated objective: "The D4D Fund puts more and better data in the hands of decision-makers, thereby enhancing evidence-based macroeconomic policies and supporting the achievement of the SDGs in low-income and lower-middle-income countries." - Monitoring gap: - At the time of the mid-term evaluation, the D4D Fund did not monitor outcomes related to the use of data; the RBM Framework and D4D Fund strategic log frame only tracked projects’ contribution to data compilation and dissemination, and did not monitor if or to what extent the improved data were used by policy makers, IMF country teams or third parties. - A D4D stakeholder at the IMF: “we have to know whether data are being used in the long run,” but added that “this is not the day-to-day work of the D4D Fund.” - Another IMF representative noted that monitoring long-term impact was not meant to be part of the RBM Framework, which is “set purposefully at the ‘effectiveness’ level” as a project management tool, and that impact is largely outside the control of project managers. - The evaluation team recommends that more attention and resources should be devoted to monitoring the use of data, potentially during the second phase of the D4D Fund. - Case-study examples where impact (data use) occurred within project lifetime: - In one country, external sector statistics were published for the first time, increasing transparency of the country’s external position and contributing to investor confidence, which helped the country to issue Eurobonds for the first time (still during the lifetime of the CD project). - In another country, production of new concept data on public sector debt statistics allowed the IMF country team to update its Debt Sustainability Analysis (DSA) for the country, even before the data was formally disseminated. - In another country, despite improved internal capacity and identification of data gaps, there was no tangible impact on better evidence-based policy formulation because authorities and the IMF country team continued to rely on the official data produced under a previous standard. - Demand-driven success: - Case studies indicated CD was more likely to be impactful when motivated by strong external (user) demand from authorities or the IMF itself (example: strong pre-mission demand to develop Residential Property Price Indices (RPPI) led to improved compilation and dissemination). - Relation to IMF strategic priorities: - D4D-supported CD directly influenced debt sustainability work (e.g., fiscal CD being frontloaded, prioritization of fiscal topics in online learning). - In one M1 case study, improved compilation and dissemination of ESS likely contributed to the country's first issuance of Eurobonds and thus to lower borrowing costs. - Indirect or limited impact on other priorities (climate change, inequality) because D4D projects did not always directly and immediately affect those priorities. - Indirect development effects reported: proceeds of Eurobonds were reported to have been used for development projects supporting decent working conditions, economic growth, gender equality and increasing infrastructure sustainability. - Better PSDS data were expected to allow authorities to better assess required fiscal space for energy transition and other SDG-related investments. - M2 potentially contributes to the IMF’s work on financial inclusion. ### Sustainability - Determinants of sustainability identified by the evaluation team: - Senior management involvement: - Some key compilers developed significant knowledge over the TA, but lagging output generation and lack of engaged users could forestall progress. - Key policy decisions and senior management engagement can potentially make the difference for sustainability. - Resource availability: - Financial resources: Insufficient financial resources allocated by country authorities to data production and dissemination endanger sustainability. - Human resources: Staff turnover and overall staff availability are risks to sustainability; staff rotation hinders institutional knowledge retention; in some cases government officials were overwhelmed by multiple tasks and could not fully commit (noted particularly in GFS where compilation is one of many tasks). - Statistics development as a policy decision: - Commitment to statistics development depends on political will; political risks and lack of ownership can endanger sustainability. - Continued engagement with country authorities: - Lack of post-project support is a risk; follow-up work could address this (see Recommendation 1 referenced by the evaluation). - Survey evidence on concerns: - Almost 20 percent of surveyed recipients indicated that recipient institutions had insufficient political support to achieve change. - More than 40 percent of surveyed recipients noted that their institutions had insufficient financial resources for staff development. - Recipients were largely positive about CD being part of the country’s long-term strategy and about measures taken by the D4D Fund to ensure sustainability; respondents were generally positive about sustainability of improvements in statistics and technical capacity. - Survey base: N=69 stakeholders; specific question bases shown as N=37, N=38, N=37, N=36 for items in Figure 2.4. - Case study corroboration: - Lack of sufficient political and senior management support commonly constrained sustainability across multiple case studies. - In the three countries with visible support and ownership from authorities, CD impact was more likely to be sustainable as countries continued producing and disseminating data. - Where CD objectives were a priority for senior management, increased resources were allocated to compiling units and authorities tracked progress, improving sustainability. - Underlying resource constraints and high staff turnover undermined sustainability in several countries. ### Lessons learned (factors contributing to impact and sustainability) - Stable compiling teams with low staff turnover. - Continuity of IMF CD teams (same TA provider and experts for an extended period of time). - Strong coordination with the IMF area department team, which often includes key data users. - Senior management support for CD activities and follow-up implementation. - Allocation of sufficient national resources for CD implementation. - Frequent and timely follow-up interactions (in between and post-missions), which can activate technical staff and senior management; in one country, having (remote) access to the STA economist empowered authorities to ask for follow-up advice. *Source: D4D Fund mid-term evaluation (staff response and action plan).* ### 2.4 Coherence ### 2.4 Coherence ### Internal coherence - D4D-funded CD is generally coherent with wider IMF surveillance and lending; coherence with other IMF CD activities was also good although some case studies indicated that communication and synergies could be further improved. - IMF processes and guidance supporting internal coherence: - IMF-wide Operational Guidelines for CD Integration with Surveillance and Lending (the “Structured exercise”), issued in 2021. - STA guidance issued in 2019 to STA staff on aspects related to the structured exercise, indicating coordination with area departments is valued within STA. - Complementarity and coordination across IMF CD modalities: - D4D-funded CD under Module 1 (M1) should “complement the provision of CD on ‘bread-and-butter’ statistics, including those provided through RTACs.” - RCDCs provide "internationally accepted statistical methodologies" and their continuous presence supports in-person CD that is delivered remotely under the D4D Fund, which focuses on "operational data for policy making." - Operational country teams in IMF area departments, including Resident Representatives, should be properly taken into account in CD design and delivery. - Survey and document evidence on internal coordination: - Around 85 percent of provider-respondents to the M1 survey indicated that CD was generally closely linked to IMF program work in the country or to IMF surveillance work. - More than 85 percent of surveyed recipients indicated that D4D-funded CD created synergies and interlinkages with IMF training activities and other IMF TA. - Around 70 percent of surveyed CD providers saw the connection with other IMF CD activities. - Case study observations: - In most M1 case studies, coordination with area departments and other functional departments took place; in at least two M1 case studies CD work plans explicitly accounted for IMF country team surveillance needs. - STA engaged with other CD-providing IMF departments to coordinate and field joint missions (e.g., with FAD); in one case, D4D-funded mission results were directly used by the IMF-FAD fiscal economist. - Internal coherence was more limited between M2 and other statistics TA, and between M3 and other statistics TA (including M1): - M2 case study suggested some use of the FAS data in IMF surveillance but very little use in IMF broader technical assistance. - M3 case study suggested room to increase synergies between M3 and other statistics TA; some evidence that absorption of M1 mission information was higher after participating in an M3 training course. - The evaluation team found few instances where authorities who previously participated in an M3 online training course subsequently received M1 TA, suggesting missed sequencing opportunities. - Suggested operational option: - Consider Module 3 course certification by essential authorities' staff as a requirement for follow-up Module 1 TA (or for other TA in the same area offered by STA or other trust funds) to maximize synergies between training (M3) and TA (M1). ### External coherence - IMF comparative advantage and role: - IMF has an intrinsic comparative advantage in macroeconomic statistics and plays an important role in setting global standards for macroeconomic statistics (e.g., Balance of Payments and International Investment Position manual, Government Finance and Statistics manual), sometimes jointly with other international institutions (World Bank, OECD, EC, UN). - Article VIII.5 of the Articles of Agreement of the IMF permits the Fund to require members to furnish national data on matters that include foreign exchange, total exports and imports of merchandise, international balance of payments, international investment position, national income and price indices; and members undertake “to furnish the desired information in as detailed and accurate a manner as is practicable and, so far as possible, to avoid mere estimates.” - Complementarity with other development partners (DPs): - Other DPs acknowledged the D4D Fund’s specificity in focusing on macroeconomic statistics; other DPs have comparative advantages in integrated government data systems, legal and policy environment, data management and data sharing. - World Bank had prior trust funds similar to D4D (SRF-CF and TF-SCB) with a broader focus (e.g., National Statistics Development Strategies) and can be seen as complementary. - Coordination and information-sharing developments: - During Phase I, D4D Fund managers actively sought coordination with other DPs; annual reports mentioned coordination efforts (e.g., with the World Bank). - IMF STA has been a long-standing partner of the World Bank Development Data Group and is part of PARIS21; STA and D4D Fund provided timely and detailed data to PARIS21 initiatives such as the Clearinghouse for Financing Development Data. - D4D Fund country-level data were lacking in the Clearinghouse because the Clearinghouse had not requested that level of detail from the D4D Fund. - Since around 2021, coordination and overview of WB projects with statistics components improved due to the WB’s new data governance architecture, enabling easier identification of projects with statistics components. - Survey evidence on external coordination: - 80 percent of recipients believed that D4D-funded CD and CD delivered by other DPs reinforced each other and added value. - 90 percent of providers believed that D4D-funded CD and CD delivered by other DPs reinforced each other and added value. - Case study and interview observations highlighting coordination gaps: - Half of the case studies were rated partially coherent, half largely coherent. - Module 3 topics sometimes overlapped with other DPs’ offerings; joint efforts with the UN to use UN experts in IMF courses did not materialize, while coordination with the World Bank was more successful in some instances (e.g., CPI course with a WB expert). - Coordination on the FAS (M2) with the World Bank’s Global Findex database occurs to some extent, but could be further improved—for example, by formalized discussions of analyses/implications between IMF and World Bank staff. - Despite extensive external engagement (including with the Women’s Financial Inclusion Data partnership), some external coherence issues remain. ### Lessons learned and recommendations for improving coherence - Using a matrix overview of different CD initiatives in a country can help improve monitoring of coherence; one country successfully produced such a matrix within a sector working group where multiple partners collaborated. - Coordinating with relevant development partners in advance of missions can improve the coherence and effectiveness of CD. - Accounting for surveillance needs when designing CD projects can improve internal coherence and impact by increasing the likelihood that data will be used by area departments for surveillance purposes. *D4D Fund mid-term evaluation, IMF.* ### 2.5 Efficiency ### 2.5 Efficiency ### Findings on efficiency and value for money - The evaluation found that, despite limited budgetary data and low budget execution, the D4D Fund was efficient and D4D-funded missions generally provided high value for money, especially considering the knowledge transferred to recipients. - Limitations in the project management system (CD-PORT) impeded cost-effectiveness assessment because costs were allocated by workstream (e.g., STX, LTX, HQ staff) rather than by country or mission; raw hours data by country/activity were not accessible. - IMF stakeholders generally considered budgeting and cost tracking by activity “unmanageable”. - Case studies showed multiple examples of cost reduction measures and instances where building internal capacity (rather than outsourcing to external consultants) improved efficiency and value for money. - Online training consumed a large part of the D4D Fund budget, which appeared appropriate during COVID-19; online courses used more visuals and fewer “talking heads” to save translation costs. - Some online courses were made freely available to private sector participants from high-income countries who may potentially have been willing and able to pay. - Recipients reported generally positive views of CD providers, who were seen as friendly, highly skilled, and able to efficiently improve procedures and generate new data output that conformed to methodology. ### Budget, expenditures, and budget execution (exact figures preserved) - The D4D Fund had an estimated budget of US$33 million over a five-year period; expenses as of end-April 2021 stood at around US$9.5 million. - Budget and expenses by (sub)module (originally planned budget, Budget FY19-FY21Q1, Expenses FY19-FY21, Total budget use (in %), Budget use FY19-FY21 (in %)): - Module 1 – Addressing Data Needs and Quality Concerns: 19-21 mln; 5,033,929; 4,243,012; 21%; 84% - Real Sector Statistics: 1,738,629; 1,596,263; 92% - External Sector Statistics: 917,481; 940,973; 103% - Fiscal and Debt Reporting: 2,377,819; 1,705,776; 72% - Ad-hoc Advisory Services: -; 39,804; - - Module 2 – Financial Access Survey: 4-5 mln; 1,492,170; 1,334,343; 29%; 89% - Module 3 - Online Learning: 5-7 mln; 2,188,579; 2,473,785; 42%; 113% - Module 4 – Statistical Information Management: 1-1.5 mln; 212,279; 114,124; 9%; 54% - RM Project Manager: 485,762; 681,091; 140% - Subtotal: 9,412,719; 8,846,355; 94% - Trust Fund Management Fee (7%): 658,890; 619,245; 94% - Total: 33 mln; 10,071,610; 9,465,600; 29%; 94% - Budget execution was low (budget use at the end of the evaluation period was below one third of the total budget). Module 3 and Module 2 showed higher budget utilization (42 percent and 29 percent, respectively) versus Module 1 (21 percent). - Resources were transferred to Module 3 from other modules (e.g., US$0.5 million was taken from M4). - The FY20 Progress Report showed a planned budget until FY21Q1 of about US$10 million and stated an additional US$4.3 million was planned to be spent by the end of FY21, indicating total planned budget until the end of FY21 was around US$14 million; this suggests some reduction in activity may have been due to operational delays as well as COVID-19. ### Timeliness of delivery and COVID-19 impact - Despite low budget execution, survey and case studies indicated a high level of satisfaction regarding the timeliness of CD delivery. - In a stakeholder survey among 69 stakeholders, the vast majority of both providers and recipients expressed satisfaction with number, duration and timing of D4D-funded CD projects; all recipients were positive about the timeliness of the IMF’s approval of the CD request. - The COVID-19 pandemic affected implementation: many offline (in-person) CD activities were delayed, cancelled or replaced with online activities; some scheduled inputs (e.g., a planned regional workshop) never took place. - Case study interviews highlighted that providers achieved a lot in a short time during missions (e.g., improving compilation procedures). ### Lessons learned for efficiency (Box 2.5) - Having data available before initial missions improves efficiency: - Since data compilation is data dependent, having prerequisite data available before missions was seen as more efficient and more effective. - In-country missions can be efficient if they lead to more sustainable knowledge transfer: - An in-person mission was seen as creating more extensive and more sustainable knowledge sharing within the country with a similar use of resources compared with sending one person to an HQ course. - Using more visuals increases the cost-effectiveness of video tutorials: - When video tutorials mainly used visuals rather than showing a speaker, it was more cost-effective to create multiple language versions using voice-overs than having to re-shoot the video for each language. ### Recommendations relevant to efficiency and effectiveness (from the evaluation’s prioritized set) - Table of prioritized recommendations (priority ranking and brief description preserved): - 1 (Impact and sustainability): Take steps to move from a “CD delivery” to a “change management” approach — Potential impact: High; Effort/cost: Medium - 2 (Relevance, internal coherence, effectiveness, impact, sustainability): During intervention planning and design, include an explicit requirement to provide a clear needs assessment for data use — Potential impact: High; Effort/cost: Medium - 3 (Relevance, effectiveness, impact, sustainability): Require a more systematic assessment of ownership and absorption capacity — Potential impact: Medium; Effort/cost: Low - 4 (Impact): Improve the monitoring and evaluation of impact at the level of data users — Potential impact: High; Effort/cost: High - 5 (External coherence): Enhance coordination and strengthen synergies with other development partners — Potential impact: Medium; Effort/cost: Medium - 6 (Efficiency and effectiveness): Further tailor the RBM framework to the D4D Fund — Potential impact: Medium; Effort/cost: Medium - 7 (Efficiency and effectiveness): Continue developing and institutionalizing the blended (online and in-person) CD delivery model — Potential impact: Low; Effort/cost: Low - Specific actions to increase efficiency and long-term impact (excerpts preserved): - Move from “CD delivery” to a “change management” approach to increase the likelihood that short-term CD effects translate into longer-term organizational change and policy improvements (especially for M1 and M3). - Require country authorities (preferably senior management or senior technical staff) to review CD proposals and include their suggestions into CD design; ask authorities to confirm explicitly that staff and resources are available for implementation. - Encourage recipients to share newly acquired knowledge within institutions and embed it in organizational systems and processes to mitigate staff turnover risks. - Prior to missions or during the first mission, ask authorities to formally agree to dedicate sufficient national resources to CD implementation and indicate how they envisage data to be used for national policy purposes and by whom. - During missions, devote more attention to how newly compiled or improved data can be used for policy purposes and include such recommendations in training courses and TA reports. - Ensure regular follow-up with authorities after projects have ended to discuss adoption/implementation of CD recommendations and bottlenecks preventing outcomes or impact; coordinate these follow-ups with area departments and consider remote sessions two or three times a year. - Further tailor the RBM framework to D4D Fund needs and continue institutionalizing the blended (online and in-person) CD delivery model to maintain efficiency gains achieved during the pandemic. *The evaluation findings, figures, and recommendations are from the D4D mid-term evaluation conducted for the D4D Fund (IMF / SEO Amsterdam Economics).* ### 2. During intervention planning and design, include an explicit requirement to provide a clear needs ### During intervention planning and design, include an explicit requirement to provide a clear needs assessment for data use, actively involving area departments, CD recipients and data users ### Needs assessment for country authorities (mostly for M1 and M3) - The evaluation team recommends project proposals require an assessment of: - how the intervention would fill recipients’ data gaps (e.g., where they lag behind international standards); - any indications of local interest in using the improved data for macroeconomic analysis and policy decisions; - the likelihood that the data would be used by national authorities (and by which agencies, for what purposes). - Recipients should be involved more actively in jointly designing terms of reference of missions, including selection of realistic and relevant project outcomes/milestones (recipients often were not aware of milestones). - More frequent (online) discussions during the planning stage are recommended to assess needs and increase ownership by country authorities (see Recommendations 1 and 3). - For M3, include more country-specific examples in training modules. - The evaluation team notes that some assessments are already embedded in D4D Fund processes (authorities formally request CD; area departments are actively involved in identification and prioritization), but recommends deepening these assessments and requiring them to be explicitly and systematically included in project proposals, ideally by adjusting proposal templates accordingly. ### Needs assessment for area departments (mostly for M1 and M3) - Include the IMF country team's assessment of: - the data gaps addressed by the project; - the relevance of these data for specific surveillance or program monitoring purposes; - a score that reflects the likelihood that the data will be used by area departments. - (Note: If this likelihood is rated at 100 percent, this could be seen as a commitment from the area department to use the data in e.g., Article IV reports.) ### Needs assessment for third party data users (for all modules) - Assessments could be qualitative. For example, for the FAS, consider conducting a survey among third party data users on their data needs and feed results into project design. ### Assessment of why recommendations are or are not adopted - Finding out why certain recommendations are or are not adopted/implemented is important for learning and can feed into design of follow-up CD activities in the same country and similar CD activities in other countries. --- ### Require a more systematic assessment of ownership and absorption capacity during CD planning and design - Particularly for M1, require an explicit assessment of: - factors that affect ownership for data-related reforms, to ensure D4D-supported CD projects address areas where there is political willingness to reform; - absorption capacity, i.e., capacity gaps among recipient organizations. - Distinguish between assessing ownership of national authorities (this recommendation) and enhancing ownership by national authorities (Recommendation 3). --- ### Improve monitoring and evaluation of impact at the level of data users - Include user-level indicators in the strategic log frame and/or the next version of the RBM Catalogue because impact on policy change depends on data use rather than data compilation and dissemination per se. - The RBM Catalogue did not yet (at the time the evaluation started) include (sufficient) outcomes that measure progress at the user level, despite the Fund name “Data for Decisions”. - Case studies showed that use of improved data sometimes appears during CD implementation, not only years after completion. - Encourage project teams to report on user-level indicators more systematically in progress reports or annual reports. More frequent post-mission follow-up should collect information on (examples of) use of newly compiled or improved data. - Progress or annual reports should include qualitative, country-specific proxies or examples of successful use of improved data for policy analysis and decisions. Ideally collect this information at three levels: - IMF area departments (examples of data use for Article IV reports, DSAs, REOs, WEO, etc.); - National authorities (examples of data use by relevant government or central bank agencies in their reports/publications and on their websites); - Relevant third parties (examples of data use by rating agencies, investment banks, researchers, development partners, NGOs, etc.). - Clarify responsibility for follow-up monitoring. Consider handing over follow-up monitoring to STA economist, STA project manager, Resident Representative offices (if present), or the IMF country team, especially for Module 1, since external expert contracts often end and relationships can fade. - The evaluation team acknowledges this monitoring can be costly; to enable it allocate a share of the D4D Fund budget to monitoring user-level outcomes by short-term experts or STA staff. Monitoring could sometimes be conducted via ex-post surveys (especially for training courses). - Example: consider including questions in the FAS itself on the extent to which FAS data are used by national authorities or third parties. --- ### Enhance coordination and strengthen synergies with other development partners - Include a systematic overview in the D4D Fund’s annual reports of CD activities of other relevant development partners (DPs) in macroeconomic statistics, potentially using information obtained through Paris21. - Consider disseminating D4D Fund TA reports more widely: - Given new operational guidelines, consider ways to more actively share TA reports with other DPs offering CD in macroeconomic statistics, and consider publishing summaries of TA reports on the IMF website. - Communicate more clearly in project proposals, TA reports and annual reports how D4D Fund-supported CD is complementary to others’ work, and which improved data are expected to become available. - Continue recent progress coordinating with the World Bank: - Institutionalize agreements to regularly and systematically obtain information from the World Bank on relevant statistics-related CD projects that have passed the design phase. - Increase coordination on potential complementary projects at the design stage (e.g., invite each other to share proposals at the design stage). - Improve mapping of projects, e.g., by aligning field codes of projects with those from the World Bank to identify overlaps and complementarities. - The evaluation team was pleased that the World Bank had recently started sharing lists of relevant projects with STA, filtered by specific field codes. - Strengthen external coordination at the project level: - Renew efforts to involve UN Statistics staff in online M3 courses such as NASx. - Enhance monitoring of potential overlaps and complementarities with the World Bank between the FAS (M2) and the World Bank Global Findex databases, and discuss possible measures to increase complementarities. --- ### Further improve (the use of) the RBM Catalogue and Strategic log frame - Discuss with ICD whether the RBM framework used by the D4D Fund can be better adapted to, or merged with, the D4D Fund Strategic Log Frame, since PMs find working with two sets of objectives and outcomes confusing and burdensome. - Given that the RBM framework was seen by D4D project managers as too generic to be useful as a project management tool, consider further expanding the granularity of the new RBM Catalogue. In particular: - Include more outcomes and indicators that better fit the country context (while still being general enough to be part of the Catalogue). - Include outcomes and outcome indicators that can be used (for relevant projects) as proxies for data use, as proposed in Recommendation 4. - Require a more direct link between project or country-specific milestones and (general) outcomes, especially for Module 1, because linkages between milestones and outcomes are sometimes not clear or lacking. - To account for exogenous country circumstances, combine country-specific outcomes and milestones with global or regional targets (e.g., target that a certain share of countries should have outcome ratings of 3 or 4 for a certain outcome or objective), and report both country targets and global targets in annual progress reports. - Ensure STA project managers of D4D-supported projects rate outcomes and milestones in the new CD-MAP system on a timely basis, and provide more guidance to ensure consistent interpretation of RBM ratings. - Note: an outcome rating should be interpreted as a change, rather than a level assessment, to indicate what the country has achieved during the course of the project (relative to a baseline assessment). --- ### Further develop and institutionalize a more blended CD delivery profile - Leverage COVID experience with remote CD delivery to enhance synergies between remote CD (including online courses) and in-person CD, treating them as complements. - Prior to missions, use remote tools more systematically to conduct pre-CD assessments (assess need for M3 online courses or online component). Based on assessment, offer online training prior to CD missions (e.g., one month in advance) to improve absorption capacity and level the playing field. - Consider making M3 course certification by senior technical staff a requirement for receiving follow-up TA from M1 (or other TA in the same area offered by STA or other IMF trust funds). - This would increase both the relevance of online training and the effectiveness of in-person CD, since M3 can improve absorption capacity for in-person CD (funded by M1 or other sources). - During missions, encourage IMF staff and experts to recommend participants sign up for online courses if technical capacity gaps are identified. - Following missions, use remote tools more systematically to follow up and keep momentum (e.g., via regular meetings recommended in Recommendation 1). - Maintain remote CD as a complement rather than a substitute for in-person CD given IT capacity constraints in fragile and low-income countries, lack of focus when recipients can be called away, and more limited opportunities for hands-on support and dialogue. --- ### Appendix A — Methodology (evaluation scoring and data sources) - The evaluation scored each OECD-DAC criterion (Relevance, Coherence, Efficiency, Effectiveness, Impact, Sustainability) at two levels: - the individual CD project level (case studies); - the overall D4D Fund portfolio level (all projects). - At the individual CD project level, scores for each OECD-DAC criterion were on a scale of 1–4, based on triangulation of: 1. Project documents (e.g., project proposals, CD mission briefs, TA reports, end-of-project reports, IMF country reports, strategy and planning documents). 2. Existing information on project outcomes, outcome indicators, and milestones from the RBM database, Strategic log frame or project documents. 3. Interviews with key stakeholders (STA staff, short-term experts, other IMF staff, recipients, other relevant stakeholders). 4. Additional relevant project-specific documents and data on data use, if available. - At the D4D portfolio level, scores for each OECD-DAC criterion (scale 1–4) were based on triangulation of: 1. D4D Fund program documents (e.g., annual reports and workplans, progress reports, program budget documents). 2. Portfolio-wide RBM ratings for milestones and outcomes (where available). 3. Portfolio-wide stakeholder survey among all key stakeholders, using the IMF’s CVent tool. 4. Interviews with key D4D Fund stakeholders, including D4D Fund program managers, donors, STA staff, ICD staff, and other IMF staff where relevant. 5. Additional relevant project-specific documents and data where available (e.g., coordination with development partners, statistics on data use, numbers of downloads). --- ### Appendix B — Case study selection (criteria and final sample) - The evaluation team selected a diverse and representative sample of country projects as case studies, in line with the ToR. Selection criteria agreed with IMF STA and ICD staff included: 1. Reflect distribution of D4D activities across Modules M1, M2, M3; select five country projects for Module 1 (covering GFS, ESS, RSS) and assess M2 and M3 as a whole (one specific M3 online course). 2. Cover most important D4D Fund TA objectives (thematic, corresponding to Module/submodule). 3. Limit M1 project selection to country projects with at least 2 missions (more than 70% of country projects in the shared portfolio consisted of only 1 mission). 4. Include country projects with complementary CD effort by a relevant IMF Regional CD Center (RCDC) where present. 5. Reflect regional distribution of CD missions. 6. Reflect portfolio share of Fragile and Conflict-affected States (FCS). 7. Reflect portfolio distribution across LICs, LMICs and UMICs. - Final selection included: - 2 case studies covering Modules 2 and 3, respectively. - 5 country project case studies for Module 1 (distribution across submodules and regions). - Table highlights from the selected M1 sample (as approved in the inception phase): - Country project examples and missions: - Uzbekistan ESS MCD Non-FCS LMIC 4 (2018/19/20/21). - Vietnam RSS (RPPI) APD Non-FCS LMIC 7 (2 on RPPI). - Indonesia RSS (RPPI) APD Non-FCS LMIC 3 (1 on RPPI). - Mongolia RSS (RPPI) APD Non-FCS LMIC 5 (2 on RPPI). - Cambodia RSS (RPPI) APD Non-FCS LMIC 3 (2 on RPPI). - Sri Lanka RSS (RPPI) APD SARTTAC Non-FCS LMIC 2 on RPPI. - Zimbabwe GFS (PSDS) AFR AFS FCS LMIC 2 (2019/21). - Ghana GFS AFR AFW2 Non-FCS LIC 2 (2019/21). - The selected M1 sample covered key D4D CD objectives for Module 1 across regions (ESS in MCD; RSS in APD; GFS in AFR) and largely met required criteria: - Portfolio distribution by submodule: RSS 46%, GFS 29%, ESS 24%. Case study sample: 2 in RSS (40%), 2 in GFS (40%), 1 in ESS (20%). - Representative spread across regions: AFR 45%, APD 27%, MCD 19%, WHD 4%, EUR 4%. Case study sample: 2 in AFR (40%), 2 in APD (40%), 1 in MCD (20%). - Representative share of FCS: FCS 15%, Non-FCS 85%. Case study sample: 1 FCS (20%). - Representative spread across income groups: LMIC 73%, LIC 19%, UMIC 9%. Case study sample: 4 LMIC (80%), 1 LIC (20%). - Country has RCDC activity in statistics: 3 case studies (60%). - Source for selection and tables: SEO Amsterdam Economics, based on D4D Fund portfolio data for FY2018-FY2021. *Source: Evaluation team, D4D Fund mid-term evaluation staff response and action plan.* ### Appendix C Evaluation questions ### d4d mid-term evaluation — Appendix C: Evaluation questions; Appendices D–E: RBM assessment and ratings ### Evaluation questions — Relevance - 1.1 To what extent were the CD objectives derived from capacity gaps identified by the authorities, IMF surveillance/program, and other partners/institutions? - 1.2 To what extent do the national authorities consider the CD objectives among the priorities of the country and/or agency? - 1.3 To what extent was the design sensitive to the context (e.g. economic, political economy, technical capacity) in which it took place? - 1.4 How well has the D4D Fund adapted and/or flexibly responded to emerging issues and changing external circumstances, e.g., addressing issues such as COVID-19, financial inclusion, gender, etc.? - 1.5 To what extent are diagnostic tools used to define and inform reform priorities and CD design in the D4D Fund beneficiary countries? ### Evaluation questions — Effectiveness - 3.1 To what extent were the CD outcomes and objective, as defined by the RBM framework, achieved or are they likely to be achieved? - 3.2 In the process, validate the RBM ratings for outcomes and objective. - 3.3 Assess to what extent the observed direct results were attributed to/ happened as a result of the CD provided. - 3.4 To what extent are the different CD delivery modalities, including long-term experts and CD training, being used to effectively address CD demands? What other modalities should the D4D Fund consider? ### Evaluation questions — Impact & Sustainability - 5.1 What is the contribution of the program to better evidence-based policy formulation? - 5.2 Assess any consequences of the CD project for relevant development and the Fund’s strategic priorities, such as climate change and inequality (economic, gender and financial inclusion). - 6.1 To what extent is any transfer of knowledge likely to be retained and/or further disseminated? - 6.2 To what degree do beneficiary countries have systems and institutional arrangements that facilitate organizational memory of D4D Fund contributions? Is there capacity and intent in recipient countries to continue the program activities on their own? ### Evaluation questions — Coherence - 2.1 To what extent does the CD project support or undermine other interventions (particularly policies), and vice versa? - 2.1.1 Internal coherence: To what extent has D4D CD been well coordinated and complementary within the IMF, including surveillance missions, the RCDCs and the IMF regional training centers? - 2.1.2 External coherence: To what extent has D4D CD been complementary to interventions by other development partners? ### Evaluation questions — Efficiency - 4.1 Were the CD inputs converted to outputs, outcomes and impact in the most cost-effective way possible, compared to feasible alternatives? - 4.2 Assess the operational efficiency of CD delivery, including the quality of output and backstopping and timeliness of delivery. - 4.3 Considering the level of CD effectiveness and the efficiencies of delivery including cost, to what extent does CD provide value for money?** - **Note:** “This question is discussed under Relevance.” ### Assessment of the RBM framework (key findings) - The D4D Fund is expected to use the IMF-wide Results-Based Management (RBM) system; each CD project should have an objective, one or more outcomes, and one or more outcome indicators per outcome drawn from the IMF's RBM Catalogue. - Under the RBM system project managers rate outcomes and milestones on a scale of 1–4 (outcomes: 1 = not achieved; 4 = fully achieved); a maximum of four milestones per outcome is expected, with due dates and rating. - In May 2021 the IMF switched from CD-PORT to CD-MAP; the evaluation analyzed only RBM data until April 2021 produced under CD-PORT, and did not assess CD-MAP. - Under CD-PORT the D4D Fund contained four “projects” (three for M1 and one for M4) corresponding to submodules and workstreams: - STA_IMF_2019_03 D4D M1 RSS Submodule — Real Sector – National Accounts: Objective — Strengthen compilation and dissemination of real sector statistics - national accounts - D4D M1 RSS Submodule — Real Sector – Prices: Objective — Strengthen compilation and dissemination of real sector statistics - prices - STA_IMF_2019_04 D4D M1 GFS Submodule — Government Finance: Objective — Strengthen compilation and dissemination of fiscal statistics - STA_IMF_2019_05 D4D M1 ESS Submodule — External Sector: Objective — Strengthen compilation and dissemination of external sector statistics - STA_IMF_2019_06 D4D M4 SI Submodule — Financial institutions: Objective — Strengthen compilation and dissemination of data on macroeconomic and financial statistics for decision making according to the relevant internationally accepted statistical standard, including developing/improving statistical infrastructure, source data, serviceability and/or metadata. - Issue: objectives and outcomes were formulated at a high and abstract level (submodule/workstream), reducing granularity and limiting the usefulness of RBM for project steering; example objective phrasing: “Strengthen compilation and dissemination of real sector statistics - national accounts.” - Resulting behaviors and consequences: - Many project managers did not use RBM actively as a project management tool under CD-PORT. - Only 2 out of 65 country-specific projects had rated outcomes as of April 2021 (roughly 3 percent of the portfolio). - Milestone ratings were used more often: 48 percent of the country projects had rated milestones as of April 2021. - Project managers often reported progress using “action points” in TA reports (conceptually akin to milestones) rather than using RBM milestones/outcomes in CD-PORT. - Providers reported the RBM system was not easy to use and did not help to “steer D4D-funded CD delivery.” - The RBM logical framework and the D4D Fund strategic log frame focused mostly on data compilation and dissemination rather than data usage; for M3, monitored “outcomes” were technically outputs fully under IMF control. - Transition and expected improvements: - D4D teams used strategic log frames for M2 and M3 because of lack of suitable objectives in the old RBM Catalogue; the introduction of CD-MAP and a revised RBM Catalogue was expected to allow more granular country-level outcomes and indicators. - D4D management acknowledged RBM had not worked well as a project management tool under CD-PORT but expected change following transition to CD-MAP; the manager noted: “At the moment we are not firm about having to rate the outcomes.” and “They know that if they don’t complete these outcomes, they get emails.” - Changes in the STA RBM Catalogue: - Catalogue 2 (May 2020) → Catalogue 3 (July 2021) - Number of objectives: 9 → 25 - Number of outcomes: 120 → 176 - Number of indicators: 521 → 840 - These updates were after April 2021 and linked to CD-MAP; the evaluation could not analyze associated ratings. ### Summary of RBM ratings (Appendix E key statistics and findings) - Available RBM data are limited and should be treated with caution. - Milestone ratings: - 80 percent of the rated milestones were largely or fully achieved during the evaluation period. - Milestone ratings were available for 35 projects, which is 48 percent of all projects that fell within the evaluation period. - Distribution of milestone ratings (N = 118): - 1 = not achieved: 2% - 2 = partially achieved: 18% - 3 = largely achieved: 18% - 4 = fully achieved: 62% - Outcome ratings: - Outcome ratings were available for only two country-specific projects, roughly 3 percent of the portfolio, making interpretation not meaningful. - Outcome ratings (N = 7) distribution shown but noted as too limited to interpret. - Portfolio definition and coverage: - Defining “projects” as country-workstream combinations, the portfolio consisted of 72 D4D-funded projects. - Milestone ratings suggest good progress across M1 submodules; more than 70 percent of milestones were largely or fully achieved across submodules, with Real Sector Statistics (RSS) at 90 percent. - RSS projects had the largest share of fully achieved milestones; Government Finance Statistics (GFS) had the lowest share. - Virtually all milestones in the “Real Sector – Prices” workstream were largely or fully achieved. - Caveat: milestone ratings were available for 35 projects (48 percent of projects during the evaluation period); outcome ratings were available for 2 country-specific projects (roughly 3 percent of the portfolio). *Source: SEO Amsterdam Economics, based on IMF’s CD-PORT data and D4D staff information.* ### Appendix F Case Study Ratings ### Appendix F Case Study Ratings ### Key evaluation findings - Relevance - Virtually all case studies were rated as either largely or fully relevant. - Case study projects were mostly fully relevant, though in at least two cases recipients were only partially involved in CD design. - Effectiveness - Effectiveness of projects was mixed. - In the three case studies where outcomes were only partially achieved, main issues were high workload, staff rotation at the recipient side, and lack of active buy-in from senior management. - Impact - Impact ratings varied across case studies and depended on whether CD was motivated by strong external (user) demand for the data. - Case study projects had mixed scores on their impact. - Sustainability - Sustainability was only partially achieved in half of the case studies. - A lack of sufficient political and senior management support was a common constraint. - In the three countries where there was support and ownership of country authorities, sustainability was largely achieved. - Coherence - Coherence was largely achieved for almost all M1 case studies, whereas it was only partially achieved for M2 and M3. - The evaluation team rated case study projects as either partially or largely coherent. - Efficiency - The majority of the case studies was found to be largely efficient. - Most case studies largely achieved efficiency. ### Quantitative summary (as presented in figures) - Case Study Ratings — Relevance: number of case studies distributed across Not Relevant, Partially Relevant, Largely Relevant, Fully Relevant (virtually all largely or fully relevant). - Case Study Ratings — Effectiveness: distribution across Not Effective, Partially Effective, Largely Effective, Fully Effective (mixed results). - Case Study Ratings — Impact: distribution across Not Impactful, Partially Impactful, Largely Impactful, Fully Impactful (varied widely). - Case Study Ratings — Sustainability: distribution across Not Sustainable, Partially Sustainable, Largely Sustainable, Fully Sustainable (most only partially achieved). - Case Study Ratings — Coherence: distribution across Not Coherent, Partially Coherent, Largely Coherent, Fully Coherent (M1 largely coherent; M2 and M3 only partially coherent). - Case Study Ratings — Efficiency: distribution across Not Efficient, Partially Efficient, Largely Efficient, Fully Efficient (most largely efficient). ### Constraints identified - Insufficient political and senior management support undermined sustainability in many cases. - High workload and staff rotation at recipient agencies reduced likelihood of achieving intended outcomes. - Recipient engagement and ownership are not under the IMF’s direct control; underfunding of statistical programs hampers agility and resiliency of statistics-producing agencies. ### IMF staff assessment and overarching response - Overall results: D4D Fund provides CD that is relevant, effective, coherent, and efficient; more could be done to ensure longer-term sustainable impact. - Key evaluator recommendations highlighted: - Move from a CD-delivery-centered approach toward a change management approach, with emphasis on organizational change. - Increase focus on use of data for policy decisions; integrate into CD delivery, ex-ante assessments, and monitoring of results. - Actions already in progress or planned: - Creation of a Financial Access Survey Advisory Group. - Stronger collaboration with the World Bank. - Deeper synergies for Fragile and Conflict-Affected States (FCS) under the IMF’s new FCS Strategy. - Implementation of the Capacity Development Management and Administration Program (CDMAP) shifting CD planning from a one- to a three-year roadmap orientation through medium term work (MTW) plans. - An IMF Staff Operational Guidance on the Dissemination of Capacity Development Information (issued in 2022) for wider, more active, and timelier sharing of CD information. ### Priority actions and planned implementation (selected, by recommendation) - Overall implementation context - Actions proposed will be undertaken primarily by the Statistics Department (STA) in close collaboration with the Institute of Capacity Development (ICD), and in cooperation with other Capacity Development Departments (CDDs) and Area Departments (ADs). - Most actions to be implemented during the remaining cycle of D4D Fund Phase I and the proposed D4D Fund Phase II to start in July 2025. - Priority levels noted as High Medium Low. - Recommendation 1: Move from “CD delivery” to a “change management” approach - Evaluator proposals (selected): - Require or strongly recommend that country authorities (preferably senior management or senior technical staff) review CD proposals and include their suggestions into CD design. - Ask authorities to confirm explicitly that staff and resources are available for implementation. - Encourage recipients to share newly acquired knowledge within the institution and embed it in improved organizational systems and processes. - Include more country-specific recommendations and devote more attention to actual data compilation during training. - Prior to missions, ask authorities to indicate how they envisage the data to be used for national policy purposes. - Follow up regularly after projects end to assess adoption of CD recommendations and data usage for policy purposes; coordinate follow-up with area departments. - IMF planned and ongoing actions (selected): - CDMAP has shifted planning to three-year MTW plans that are periodically reviewed and updated. - STA will discuss roadmap proposals earlier with authorities and in conjunction with ADs to strengthen links to policy relevance and resource implications. - STA will proactively discuss resource needs and CD impact with high-level country authorities during IMF Annual and Spring Meetings. - STA will work with AD teams and Resident Representatives to identify and seek action on underfunding of statistical programs. - STA will explore more continuous country engagement via regular informal check-ins between CD missions. - STA will repurpose regional managers and country coordinators to ensure more regular dialogue with ADs on country data needs, MTW progress, and data usage for policy decision making. - STA will encourage documentation of country data compilation processes and promote online learning and How-to Notes; develop a Compilers’ Hub. - Integrate practical data compilation exercises into D4D-funded in-country/regional workshops and D4D-funded online learning curriculum. - Recommendation 2: Include explicit needs assessment for data use during planning and design - Evaluator proposals (selected): - Assess how interventions fill recipients’ data gaps and indications of local interest in using improved data for policy; likelihood of usage by national authorities. - Involve recipients in jointly designing terms of reference, including realistic outcomes/milestones. - For area departments, include the IMF country team's assessment of data gaps and relevance for surveillance/program monitoring and a score reflecting likelihood of use by area departments. - For third party data users, conduct qualitative assessments (e.g., surveys for the FAS). - IMF planned and ongoing actions (selected): - STA will consult extensively with country authorities at design phase to assess data use needs. - D4D-funded regional workshops will cover how CD interventions address data gaps and readiness of authorities. - IMF Country Engagement Strategies will inform CD planning and design for FCS; Article IV Reports include data adequacy assessments. - A FAS Advisory Group is being formed under M2 consisting of policymakers, reporting countries, stakeholder organizations, and academia to provide feedback on FAS development. - STA encourages countries to undertake national user consultations for key CD projects such as a GDP rebasing. - Recommendation 3: Require systematic assessment of ownership and absorption capacity - Evaluator proposals: - Require explicit assessment of factors that affect ownership and of absorption capacity (capacity gaps among recipient organizations). - IMF planned and ongoing actions (selected): - STA will systematically engage with country authorities before designing projects on their views of pressing data gaps and required resources. - Diagnostic missions will include use case analysis where feasible and address resource availability and absorptive capacity with key stakeholders, including compiling and source data-providing agencies. - Regional workshops on new statistical products will communicate importance of national agencies conducting user outreach prior to D4D-funded engagement. - Recommendation 4: Improve impact monitoring and evaluation at the level of data users - Evaluator proposals (selected): - Include user-level indicators in the strategic log frame and/or RBM Catalogue. - Encourage project teams to report user-level indicators more systematically in progress/annual reports. - Conduct more frequent post-mission follow-up monitoring (especially for Module 1) and clarify who is responsible for such monitoring. - Allocate a share of the D4D budget to ensure monitoring can take place; use short-term experts, STA staff, or ex-post surveys. - Collect information on data use at three levels: IMF area departments; national authorities; relevant third parties. - IMF planned and ongoing actions (selected): - STA will explore feasibility of including qualitative information on use of improved data for policy analysis and decisions in D4D Fund progress reports, drawing on government policy publications. - Outreach to national policymakers and IMF mission chiefs will be undertaken; examples of successful data use will be included in D4D Fund Annual Reports. - STA will continue to monitor internal usage of FAS data in Staff Reports, Policy Notes, Article IVs, and other documents. - STA's Data Product Evaluation will continue to estimate FAS user statistics and monitor usage of IMF statistical products (with caveats related to third-party data dissemination). *Source: SEO Amsterdam Economics; IMF Staff Response to the Independent Midterm Evaluation of the Data for Decisions (D4D) Fund and Action Plan.* ### references of FAS data and definitions in IMF ### D4D mid-term evaluation: staff response and action plan — references of FAS data and definitions in IMF ### Monitoring, post-project follow-up, and data quality - Under CDMAP processes, active project monitoring (during the project life cycle) is the responsibility of the STA project manager. - IMF contract arrangements with short-term experts limit their role on country engagement to mission duration. - IMF will assess feasibility of integrating post-project follow-up into existing processes. - Reversal of project gains (decline in data quality, for example) are typically identified: - by IMF staff in the context of surveillance or lending operations or through gaps in data reporting to STA; and - by national authorities as a basis for CD requests. - STA will propose to the D4D Fund Steering Committee a share of the Phase II budget for implementing systematic monitoring, including post-project surveys; the proposal will be articulated in the Program document for endorsement. ### Coordination and synergies with other development partners (Recommendation 5) - Evaluators proposed: enhance coordination with other development partners (e.g., the World Bank, UN Statistics Division, or via PARIS21), including: - A. Include a systematic overview in the D4D Fund’s annual reports of the CD activities of other relevant DPs in macroeconomic statistics, potentially based on information obtained through PARIS21. - B. Consider disseminating the D4D Fund’s TA reports more widely: - B.1. More actively share TA reports with other DPs offering CD in macroeconomic statistics, and consider publishing summaries of TA reports on the IMF website. - B.2. Communicate more clearly in project proposals, TA reports and annual reports how D4D Fund-supported CD is complementary to what others are doing, and which improved data are expected to become available. - C. Continue progress with coordination with the World Bank: - C.1. Institutionalize agreements to regularly and systematically obtain information from the World Bank on relevant statistics-related CD projects that have passed the design phase. - C.2. Increase coordination on potential complementary projects at the design stage (e.g., invite each other to share proposals at the design stage). - C.3. Improve mapping of projects, e.g., by aligning field codes of projects with those from the WB to identify overlaps and complementarities. - D. Strengthen external coordination at the project level: - D.1. Renew efforts to involve UN Statistics Division staff in online M3 courses such as NASx. - D.2. Enhance monitoring of potential overlaps and complementarities between the FAS (M2) and the World Bank Global Findex databases; discuss measures to increase complementarities. - IMF/STA responses and planned actions: - (A) STA will leverage granular information in the Clearinghouse for Financing Development Data to enhance coordination mechanisms. - Beginning in 2024, D4D Fund Annual Reports will feature more systematic reporting on how D4D Fund-supported CD is complementary to those of other development partners. - (B) STA will advance sharing of CD outputs in line with the IMF Staff Operational Guidance on the Dissemination of Capacity Development Information issued in 2022: - Steps to disseminate high-level summaries (HLS), and increase the number of technical assistance reports (TARs) on the IMF website. - Progress to be reported in D4D Fund Annual Reports. - TARs accessible to the D4D Fund donor group may be shared with other DPs on request and subject to the CD-recipient country’s agreement. - (C) STA has established mechanisms with the World Bank and will further strengthen collaboration (e.g., public sector debt statistics, national accounts, prices, climate). - Since 2022, the World Bank has been engaging STA at the design phase of their statistics CD projects to enable upstream coordination. - (D) Avenues for coordination with other DPs will be pursued for each new project/activity under the D4D Fund, including tapping expertise of other international organizations for online courses. - Under M2, coordination with the World Bank's Findex has helped enhance financial inclusion data collection. The FAS and Findex are discussing complementarities, particularly in the context of the FAS pilot data collection and Data Gaps Initiative 3 Recommendation 12: Fintech-enabled financial inclusion. - The FAS is also harnessing complementarities with other databases such as the FSB's Fintech Credit survey, World Bank’s Global Payment System Survey, and OECD's financial literacy surveys. ### Results-based management (RBM) framework tailoring (Recommendation 6) - Evaluators proposed (partially agree): - A. Discuss with ICD how the RBM framework used by the D4D Fund can be better adapted to, or merged with, the D4D Fund Strategic Log Frame to avoid duplication and reduce burden on project managers. - B. Expand the granularity of the RBM Catalogue: - B.1. Include more outcomes and indicators that better fit country context (while remaining general enough for the Catalogue). - B.2. Include outcomes and outcome indicators that can be used as proxies for data use, where relevant. - B.3. Require a more direct link between project- or country-specific milestones and general outcomes, especially for Module 1. - C. Combine country-specific outcomes and milestones with global or regional targets to account for exogenous country circumstances; report both country targets and global targets in annual progress reports. - D. Ensure STA project managers rate outcomes and milestones in CD-MAP on a timely basis and provide guidance for consistent RBM rating interpretation. - IMF/STA responses and planned actions: - (A) To be undertaken in designing the strategic log frame for Phase II of the D4D Fund in close consultation with ICD. - (B) STA has already expanded the number of objectives available in the RBM Catalog (example: real sector statistics has 4 objectives for national accounts instead of 1 in the old catalog); expansion will continue (e.g., for climate statistics CD delivery). - (C) Global or regional targets will build on country-specific outcomes in the design of the strategic log frame of Phase II of the D4D Fund. - (D) Inputs for monitoring and reporting of the strategic log frame will be directly drawn from CDMAP, ensuring ratings of outcomes and milestones are properly and timely recorded in CDMAP. - Annual project assessments are part of CDMAP deliverables. - STA has implemented an RBM data completeness monitor to improve the quality of these assessments. ### Blended (online and in-person) capacity development delivery (Recommendation 7) - Evaluators (fully agree) noted opportunities to enhance synergies between remote CD (including online courses) and in-person CD, treating them as complements rather than substitutes: - Prior to missions, use remote tools more systematically to conduct pre-CD assessments and offer online training in advance (e.g., one month) to improve absorption capacity and level the playing field. - Consider making M3 course certification by senior technical staff a requirement for receiving follow-up TA from M1 (or other TA in the same area offered by STA or other IMF trust funds). - During missions, encourage IMF staff and experts to recommend participants sign up for online courses when gaps are identified. - Following missions, use remote tools to follow up and keep momentum (e.g., via regular meetings). - Remote CD should remain a complement rather than substitute for in-person CD, given IT constraints in fragile and low-income countries and limitations of remote delivery for hands-on support. - IMF/STA actions and status: - Blended learning is now an established CD delivery modality; STA will continue to roll out blended learning courses piloted during FY23 to all work streams, countries and regions. - Integration of Modules 1 and 3 through blended learning has started; one example (HFIEA/QNA) is cited in the 2023 D4D Fund Annual Report. - STA will request authorities to encourage staff registration for available online courses; in some instances, completion of online (foundational) courses will be a precondition for attendance at in-person workshops and trainings. - STA will expand the blended modality to scheduled TA activities with embedded training components, leveraging the online learning curriculum developed under M3 (either full courses or selected videos). - STA will continue leveraging hybrid CD delivery to widen audience reach and tap expertise of other organizations (e.g., World Bank to discuss use of Quarterly External Debt Statistics during STA training on external debt statistics). --- _Source: https://www.imf.org/-/media/files/capacity-developement/d4d/mid-year-check-in-meeting/d4d-mid-term-evaluation-staff-response-and-action-plan.pdf_