## Part I ## Source details **Canonical URL:** [Part I](https://www.imf.org/-/media/files/capacity-developement/mnrw/mte2015i.pdf) ## Other formats - [Markdown version](/-/media/files/capacity-developement/mnrw/mte2015i.pdf.md) - [Structured JSON version](/-/media/files/capacity-developement/mnrw/mte2015i.pdf.json) --- ### Purpose, scope, and timeframe - Evaluation assessed extent to which the Managing Natural Resource Wealth Topical Trust Fund (MNRW TTF) is delivering planned objectives and outcomes at activity and trust fund level, identified lessons learned, and made recommendations for improvements. - Activities evaluated: technical assistance (TA), research projects, and workshops. - Coverage period of activities evaluated: April 2011 to April 2014. - The MNRW-TTF was launched in March 2011 and was funded with contributions totaling US$25.3 million from six countries and the EU. ### Evaluation methodology and team - Bottom-up approach: each project assessed using questions grouped by the OECD DAC criteria—relevance, effectiveness, efficiency, impact, and sustainability. - Information sources: desk reviews of documents and data, interviews, surveys, and country visits to Kenya, Mongolia, and Tanzania. - Evaluation team: Ms. Ana María Jul (Team Lead, macro-fiscal expert), Mr. Rolando Ossowski (public finance expert), and Mr. Reuben Hermoso (monitoring and evaluation expert). ### Summary of project performance and key metrics - Overall average rating of projects: Good (rating scale: Excellent, Very Good, Good, Modest, Poor). - Overall rating: G with an average score of 5.2 and a standard deviation (SD) of 0.42. - Distribution highlights: - More than a third of projects rated Very Good, including three projects rated close to Excellent. - Over 50 percent of projects rated Good. - Average ratings by DAC criterion: - Relevance: Very Good (average score 5.9, SD 0.82). - Effectiveness (outcomes and objectives): Good (average score 4.5; SD 1.39). - Efficiency, Impact, Sustainability: Good (scores clustered between 5.1 and 5.4; specific values reported below). - Average scores by criterion reported: - Relevance: 5.9 (VG) - Effectiveness of outcomes and objectives: 4.5 (G) - Inputs/outputs/risk/processes average: 5.2 (G, SD 0.71) - Efficiency of TA delivery: 5.2 (G, SD 0.71) - Efficiency of reporting: 5.4 (G, SD 0.61) - Impact: 5.1 (G, SD 0.99) - Sustainability: 5.3 (G, borderline VG, SD 0.72) ### Factors associated with successful projects - Pre-project proposal missions or extensive discussions with authorities to design TA. - Focused, realistic, and well-defined objectives and outcomes aligned with existing and potential capacity. - Joint preparation of action plans and project sequencing with timelines and milestones. - Continuity of engagement, including intensive use of Short-term Experts (STXs). - Strong emphasis on capacity building and hands-on training. ### Main program-level findings - The overall performance of TTF projects is solid and relatively even across DAC criteria; relevance rated highest. - Effectiveness of objectives and outcomes shows greater dispersion and is influenced by factors beyond IMF control (e.g., elections, legislation approval, Ebola). - Research and workshops produced important products but could better target issues faced by low-income and lower-middle-income resource-rich countries. - Governance structures (Steering Committee (SC), Secretariat, IMF reporting) functioned and provided strategic guidance, but IMF reporting could be further standardized and made more efficient. - Evidence of overestimation of project costs and dispersion in budget execution across projects; several projects had budgets cut in December 2014. ### Geographic distribution, project counts, and module composition (through April 2014) - Projects endorsed: 34 TA projects in 17 countries and one regional project. - By number of countries receiving TTF TA: 50 percent to Sub-Saharan Africa (SSA), 28 percent to East Asia and Pacific (EAP), 11 percent to Middle East and North Africa (MENA), and 11 percent to Latin America and Caribbean (LAC). - By number of projects: 62 percent to SSA, 24 percent to EAP, 9 percent to MENA, 6 percent to LAC. - By endorsed budget: 62 percent to SSA, 28 percent to EAP, 5 percent to MENA, 5 percent to LAC. - By actual expenses (through April 2014): 57 percent to SSA, 29 percent to EAP, 4 percent to MENA, 10 percent to LAC. - Module composition: - Module 1—Extractive Industries Fiscal Regime (largest): accounted for 39 percent of budgeted expenses and close to 50 percent of actual expenses. - Modules 2 and 3 each accounted for about a quarter of the TTF’s TA. - FAD’s modules accounted for 96 percent of total TTF TA spending. ### Project time frames, extensions, and drivers of delays - Average initial time frames by module: - Modules 1, 3, and 4: 24 months - Modules 2 and 5: 30 months - Of 32 projects that started, 24 were extended. - Average extensions by module: - Module 3: 14 months - Module 5: 12 months - Modules 1 and 2: 11 months - Module 4: 9 months - Project time-frame statistics (projects endorsed through April 30, 2014): - Module 1: Number of projects 14; Project Proposal 25; Average Time Frame 36; Extension 11 - Module 2: Number of projects 6; Project Proposal 30; Average Time Frame 41; Extension 11 - Module 3: Number of projects 8; Project Proposal 24; Average Time Frame 38; Extension 14 - Module 4: Number of projects 2; Project Proposal 24; Average Time Frame 33; Extension 9 (Excludes Iraq (4) and Uganda (4), which did not start.) - Module 5: Number of projects 2; Project Proposal 30; Average Time Frame 42; Extension 12 - Drivers of extensions included ambitious initial time frames, acute capacity constraints (including at ministries of finance), political issues and elections, inter-agency coordination difficulties, and variable commitment to reform. ### Budget execution (as of April 2014) - Total endorsed budget: US$19.2 million - Overall execution: 42 percent of the endorsed budget had been executed. - Execution by module (thousands of dollars): - Module 1: Endorsed Budget 7527; Actual Expenses 3893; Percent of budget 52 - Module 2: Endorsed Budget 4453; Actual Expenses 1973; Percent of budget 44 - Module 3: Endorsed Budget 4444; Actual Expenses 1807; Percent of budget 41 - Module 4: Endorsed Budget 2122; Actual Expenses 310; Percent of budget 15 - Module 5: Endorsed Budget 644; Actual Expenses 19; Percent of budget 3 - Total: Number of projects 34; Endorsed Budget 19190; Actual Expenses 8002; Percent of budget 42 - Execution by vintage (average length in April 2014 and percent executed): - FY2012: 14 projects still active, average length 30 months, executed on average 59 percent of budget - FY2013: 10 projects, average length 18 months, executed on average 33 percent of budget - FY2014: 5 projects, average length 8 months, executed on average 21 percent of budget - Evaluation noted dispersion in budget execution and evidence of overestimation of project costs in many projects; several projects had budgets cut in December 2014. ### Role and contributions of RTACs - RTACs participated in specific projects and areas (e.g., TSA work, training) providing implementation support and continuity. - RTAC examples: - AFRITAC Central assisted DRC DGI with e-filing implementation and participated in HQ-led missions. - AFRITAC West provided support on cash management and accounting systems in Niger outside the TTF project. - AFRITAC East provided resident revenue administration advisor participation in Tanzania and PFM experts in module 1 mission on natural gas development. - PFTAC provided assistance under module 1 in Timor Leste. - RTACs highlighted as an efficient alternative to LTXs by exploiting economies of scale and reducing the risk of taking over tasks that should be done by government officials. ### Relevance and coordination findings - Relevance rated VG overall (average score 5.9). - Strong sub-categories: consistency with TTF selection criteria and addressing country needs and priorities. - Weakest sub-category: capacity to implement TA advice (aligned with project extensions). - Coordination with other TA providers rated VG on average; 60 percent of government officials thought IMF projects were reasonably coordinated with other TA providers (40 percent had no opinion). - Five cases identified of conflicting or uncoordinated advice from other TA providers with outcomes ranging from significant delays to less significant disruption. ### Effectiveness findings (outcomes and TA delivery) - Effectiveness split into: - Effectiveness of outcomes and objectives (results delivered): average score 4.5 (G), large dispersion (SD 1.39). - Effectiveness of TA delivery (design, inputs, outputs, risk management, processes): average score 5.2 (G, SD 0.71). - Lack of well-defined timelines and milestones in project documents complicated evaluation; many outcomes had only end-dates. - Module performance on effectiveness of outcomes and objectives: - Module 2: four out of five projects assessed were rated M. - Module 3: rated G overall; three-fifths of projects were rated VG to G. - Module 1: large dispersion — almost half of projects rated VG and 40 percent rated M. - Risk management was the weakest sub-category across TA delivery; risks often identified pro-forma and mitigating measures not explicitly incorporated into project design (notable exception: Iraq). ### Efficiency, impact, and sustainability - Efficiency: - Efficiency of TA delivery: G with average score 5.2 (SD 0.71); distribution: one project E; 40 percent VG; 40 percent G. - Efficiency of reporting: G with average score 5.4 (SD 0.61); more than half of projects rated VG. - Areas for improvement: timeliness and standardization of project summaries, proposals and assessments. - Impact: - Rating: G with average score 5.1 (SD 0.99). - Observed positive impacts: improved outreach to government and civil society, improved practices to manage natural resources, improvements in resource investment environment and transparency. - Dispersion in impact reflects factors outside IMF control (e.g., parliamentary approval). - Sustainability: - Rating: G with average score 5.3 (borderline VG) and SD 0.72. - Strongest aspect: integration of project into countries’ reform agendas (rated VG). - Weakest aspect: capacity building — need for sustained hands-on TA and long-term effort. - Multi-year/modular approach facilitates integration into reform agendas but sustainable impact depends on capacity strengthening. ### Comparison of DAC criteria across respondent types (selected scores preserved) - Relevance: Project Managers E (6.6); Experts VG (6.2); Government Officials VG (5.9) - Effectiveness Outcomes and Objectives: PM G (4.9); Experts VG (5.6); GOs VG (5.5); Evaluation Team G (4.5) - Efficiency of TA Delivery: PM VG (5.6); Experts VG (6.1); GOs VG (6.0); ET G (5.2) - Impact: PM VG (5.9); Experts VG (5.7); GOs VG (5.5); ET G (5.1) - Sustainability: PM M (4.3); Experts VG (5.5); GOs VG (5.6); ET G (5.3) - Overall: PM G (5.4); Experts VG (5.7); GOs VG (5.7); ET G (5.2) - Note: Government Officials participation rate was 23 percent (about half that of PMs and Experts), possibly biasing GO scores upward. ### Country- and activity-level findings (selected examples) - Kenya (Module 1): Overall VG (6.3) — draft new income tax provisions for mining and petroleum completed; revised Income Tax legislation published in late 2014; Mining Bill reviewed and enacted in 2014; FARI model transferred and operating at NOCK; timelines for milestones were formally defined; strong stakeholder consultations contributed to fast-track approval of legislation. - Mozambique: - Module 1: project extended by 15 months to September 2015; by October 2014 execution of the budget amounted to 87 percent. - Module 2: natural resources unit created at MRA headquarters; project extended by 13 months to April 2016; by October 2014 execution of the budget amounted to 35 percent. - Module 3: progress slowed by inability to implement the Macro Fiscal Unit (MFU); project extended by 24 months to April 2016. - Module 5: STA mission in February 2015 identified major gaps; project extended to April 2016; by October 2014 execution of the budget amounted to 17 percent (module 5 overall low execution). - Guinea: VG (6.0) — contributed to review and redrafting of fiscal provisions in the Mining Code; project extended by two years to April 2015; by October 2014 budget execution was 50 percent; momentum affected by company data non-provision and Ebola outbreak. - Sierra Leone: VG (5.5) — comprehensive new legislation drafted by May 2012; project extended to April 2016 with varying execution rates across modules (e.g., Module 3 80 percent by October 2014). - Niger: Module 1 terminated in June 2014 with actual expenditure 21 percent of budget. - Solomon Islands: VG (5.8) — new fiscal regime for mining enacted April 2014 largely follows TA recommendations; project extended by 15 months to April 2015; by October 2014 budget executed by 20 percent. - Timor Leste: M rating (4.2) — ESI component active and approaching handover; macro-fiscal and some components terminated or scaled back; project extended by 32 months to December 2015 in parts. ### Evaluation of RBM (Results-Based Management) and reporting - Present top-down RBM limitations: - Aggregation obscures project-level patterns; 0–4 rating applied without timeline adjustment (outcomes get same rating regardless of project life-stage). - Top-down approach lacks disaggregation needed to interpret progress over time and to separate maturation from new-project effects. - Proposed bottom-up approach features: - Project -> objectives -> targeted outcomes with SMART indicators. - Weighting: targeted outcomes weighted within a project module; weights across projects proposed to be based on project budgets. - Two proposed bottom-up scores: composite score (weighted average of targeted outcomes) and on-track score (performance relative to timeline/expectations). - Concerns raised: - Redundancy and potential interpretability problems of using two separate scores (composite and on-track). - Distortions from weighting by project budgets; uncertainty whether to use budgets or actual expenses; cost-efficiency implications (e.g., Solomon Islands used 21 percent of budget). - Empirical cost-per-point of effectiveness (evaluation team exercise): - Module 1: cost ranged US$10,000 to US$151,000; average US$79,000; median US$80,000. - Module 2: cost ranged US$59,000 to US$155,000; average US$92,000; median US$84,000. - Module 3: cost ranged US$41,000 to US$86,000; average US$60,000; median US$83,000. - Recommendations on RBM: - Disaggregate outcomes and objectives in the log frame used in the top-down approach. - Use one set of scores for the bottom-up approach that incorporates timeline/milestone features (avoid separate composite and on-track scores). - Establish well-defined timelines and milestones so outcome assessments can reflect milestone achievement. - Complement RBM scores with qualitative statements and contextual information. ### Governance and Steering Committee (SC) findings - SC provided strategic guidance and monitored results in line with objectives defined in the MNRW TTF Program Document. - SC composition: donor representatives and IMF staff from ICD and TA departments; SC decides by consensus and endorses an indicative annual work plan. - SC priorities: shaping strategic direction of the TTF and monitoring results of TTF projects. - IMF reporting to the SC: timely and appropriate; improvements observed but further standardization and efficiency gains recommended. - Project submission and approval rules: - Proposed projects submitted via Project Summary (PS); after endorsement a Project Proposal (PP) is made available to the SC through the Donor Gateway. - Approval of the PP by ICD activates the project's budget. - SC endorsement required for budget changes in excess of 10 percent, project extensions longer than six months, changes to objectives or countries; three-week lapse-of-time waiting period for approval. - Areas for improvement: - Standardize PPs and PAs; clarify objectives vs. outcomes; ensure SMART indicators and milestone dates; consistent use of weights and rating interpretations. - Improve Donor Gateway usability and timeliness of uploads; improve mission reporting systems. ### Research projects and workshops — performance and lessons - Endorsed through April 2014: five research projects and four workshops; total initial budget US$718,500. - Four research projects evaluated (one launched after April 2014 not evaluated). - Findings: - Research Project 1 (Savings and Consumption Guidelines): rated G; delays in finalizing papers; draft materials used in IMF policy work and Workshop 1. - Research Project 2 (Mining Tax Administration): contributed to handbook Administering Fiscal Regimes for Extractive Industries (IMF–World Bank); extended by 1.5 years. - Research Project 3 (Resource Revenue Database for Government Revenue Statistics): draft template circulated; field-tested in six countries; contributed to an EITI Summary Data Template; completed June 2014 (slightly late). - Research Project 4 (Sovereign Asset-Liability Management handbook): published June 2014; relevance limited for low-capacity TTF countries; started a year and a half after placement on the Work Plan. - Workshops: - Workshop 1 (Fiscal Management of Oil and Natural Gas in East Africa): rated VG; held January 2014 in Arusha; high participant satisfaction; strong country presentations and peer learning. - Workshop 2 (Cross-Border Issues in Resource Taxation): held May 2012; insufficient information beyond relevance to fully rate; mixed feedback on practical relevance. - Workshop 3 (Central Asia): not launched and removed from Work Plan. - Workshop 4 (Sub-Saharan Africa): held March 2012 in Kinshasa; limited country presentations reduced relevance relative to later regional workshops. ### Recommendations — TTF-level (strategic and operational) - Recommendation No.1: Appoint a full-time supra-project manager in FAD financed by the TTF to: - Enhance standardization. - Strengthen the RBM system and conduct top-down RBM exercise. - Distill lessons across projects and review project documents for consistency. - Decide on transfer/termination of inactive projects in coordination with project managers. - Recommendation No.2: Consider providing MCM and STA-type support through IMF non-TTF TA for efficiency. - Recommendation No.3: Improve synergies and coordination across modules and with LEG: - Require Module 1 TA where inadequate fiscal regimes would hamper Module 2. - Ensure better synchronization when models transfer from Module 1 to Module 3. - Improve LEG planning through earlier FAD signals on needs. - Recommendation No.4: Improve project budgeting by analyzing projects with evidence of cost overestimation and applying lessons to future budgeting. - Recommendation No.5: Deal proactively with inactive projects: - Create an inactive project reserve list; transfer unused budget funds back to the TTF or retain around 20 percent of unused funds for reactivation within a year. - Consider terminating projects never launched within a year of endorsement and projects in the inactive list after a year. ### Recommendations — Activity-level (project design and delivery) - Recommendation No.6: Strengthen project design, planning, and TA delivery: - Use scoping missions prior to PS submission where IMF TA in the area has been limited. - Set focused and realistic objectives and outcomes; share draft proposals with authorities. - Prepare timelines and road maps with milestones for inputs, outputs, outcomes, and objectives. - Conduct more in-depth risk analysis and incorporate mitigation measures into project design. - Deliver more hands-on TA and aim at continuity of engagement and capacity building (greater use of STXs). - Recommendation No.7: Increase use of RTACs and regional advisors to provide implementation follow-up, continuity, and hands-on training; coordinate multi-year plans with RTAC participation where relevant. - Recommendation No.8: Better align research projects and workshops with operational TTF needs: - Focus research more on issues faced by low-income and lower-middle-income producers. - Consider using STXs for independent inputs and limit outsourcing of TTF research papers where IMF clearance requirements slow completion. - Design workshops to maximize peer-to-peer learning and include ample government official presentations and breakout sessions on technical issues. ### RBM and reporting — specific enhancements - Recommendation No.9 (Enhancing the RBM): - Disaggregate objectives and outcomes in the top-down log frame and define intermediate milestones. - Use a single bottom-up score that incorporates timeline/milestone features rather than separate composite and on-track scores. - Weighting approach to aggregation should be reconsidered (budgets versus actual expenses; simple averages as an alternative). - Complement quantitative scores with qualitative narrative on implementation context, authorities’ effort, main factors affecting progress, and risks. - Recommendation No.10 (Enhancing reporting efficiency): - Standardize PPs and PAs: SMART indicators, completion dates, weights for outcomes, uniform interpretation of achievement ratings. - Require revised PS and PP submission and SC endorsement when scope or budget change materially (e.g., >10 percent). - Improve Donor Gateway usability (search by country/module; clearer upload/version indicators) and timeliness of mission and project documentation. ### Implementation characteristics - Most recommendations require corrective actions rather than new instruments and can be implemented in the normal course of TTF work; sequencing may be needed but no specific time frame prescribed. - Strengthening risk assessment at project onset and making scoping missions a routine element where warranted are emphasized to improve design realism and implementation success. *Mid-Term External Evaluation of the MNRW TTF (DevTech Systems, Inc.), May 2015.* ### Part I ### Part I ### Purpose, scope, and timeframe - The evaluation assessed the extent to which the Managing Natural Resource Wealth Topical Trust Fund (MNRW TTF) is delivering its planned objectives and outcomes at both activity and trust fund level, identified lessons learned, and made recommendations for improvements. - Activities evaluated: technical assistance (TA), research projects, and workshops. - Coverage period of activities evaluated: April 2011 to April 2014. - The MNRW-TTF was launched in March 2011 and was funded with contributions totaling US$25.3 million from six countries and the EU. ### Evaluation methodology - Bottom-up approach: each project assessed using questions grouped by the OECD DAC criteria—relevance, effectiveness, efficiency, impact, and sustainability. - Information sources: desk reviews of documents and data, interviews, surveys, and country visits to Kenya, Mongolia, and Tanzania. - Evaluation team: Ms. Ana María Jul (Team Lead, macro-fiscal expert), Mr. Rolando Ossowski (public finance expert), and Mr. Reuben Hermoso (monitoring and evaluation expert). ### Summary of project performance - Overall average rating of projects: Good (rating scale: Excellent, Very Good, Good, Modest, Poor). - Distribution highlights: - More than a third of projects rated Very Good, including three projects rated close to Excellent. - Over 50 percent of projects rated Good. - Average ratings by DAC criterion: - Relevance: Very Good. - Effectiveness, Efficiency, Impact, Sustainability: Good. - Effectiveness of objectives and outcomes: on average somewhat lower than other criteria and exhibited large dispersion. - Factors contributing to successful TA projects: - Pre-project proposal missions or extensive discussions with authorities to design TA. - Focused, realistic, and well-defined objectives and outcomes aligned with existing and potential capacity. - Joint preparation of action plans and project sequencing with timelines and milestones. - Continuity of engagement, including intensive use of Short-term Experts (STXs). - Strong emphasis on capacity building and hands-on training. ### Comparison with project managers’ assessments - Of 24 projects compared using 2014 project assessments: - 11 projects rated in the same category as project managers. - 9 projects rated one category higher than project managers. - 3 projects rated two categories higher than project managers (these showed substantial progress in the second half of 2014, a period not considered in managers’ May 2014 assessments). - 1 project rated one category lower than project managers. ### Evaluation of research projects and workshops - Delivered important products that contributed to dissemination of findings and practical guidance on resource management for TTF-eligible countries. - In some cases, greater focus on issues faced by low-income and lower-middle-income resource-rich countries would have enhanced relevance to beneficiary countries and the TTF’s work. ### Evaluation of Results-Based Management (RBM) - The report evaluated the present top-down RBM approach and a proposed bottom-up approach. - Raises issues with the present top-down approach and questions the use of two sets of scores in the proposed bottom-up approach instead of a single set as used in this evaluation. - Recommendation to enhance RBM as a monitoring tool by disaggregating outcomes and objectives in the log frame used in the top-down approach and using one set of scores for the bottom-up approach. ### Evaluation of governance and Steering Committee (SC) reporting - The SC provided strategic guidance and monitored results in line with its objectives defined in the MNRW TTF Program Document. - SC members placed priority on: - Shaping the strategic direction of the TTF. - Monitoring results of TTF projects. - IMF reporting to the SC delivers timely and appropriate information and has improved over time; could be further strengthened through greater standardization. - Proposal to improve efficiency of reporting to the SC. ### Main findings - The overall performance of TTF projects is solid and relatively even across evaluation criteria. - Effectiveness of objectives and outcomes shows greater dispersion and is influenced by factors beyond IMF control. - Successful implementation is associated with careful project design, realistic scope, continuity, and hands-on capacity building. - Research and workshops have value but could better target low-income and lower-middle-income resource-rich contexts. - Governance structures function but can be strengthened—particularly IMF reporting standardization and SC monitoring efficiency. ### Recommendations — TTF-level (strategic and operational) - Appoint a full-time supra-manager in FAD financed by the TTF to: - Enhance standardization. - Strengthen the RBM system. - Address technical aspects of projects. - Distill lessons across projects. - Consider providing TA support on Monetary and Capital Markets (MCM) and Statistics Department (STA) modules under non-TTF IMF TA. - Improve synergies and coordination among FAD modules and with LEG. - Improve project budgeting to reduce the over-costing of projects. - Develop and implement a proposal to deal with inactive projects. ### Recommendations — Activity-level (project design and delivery) - Strengthen project design, planning, and TA delivery through: - Scoping missions. - More focused and realistic objectives, outcomes, and time frames. - Defining milestones and preparing road maps. - More in-depth risk analysis. - Delivering more hands-on TA. - Aiming at continuity of engagement and capacity building. - Increase participation of Regional Technical Assistance Centers (RTACs) and regional advisors. - Enhance the use of research projects and workshops to achieve TTF objectives. ### Other recommendations and implementation characteristics - Enhance RBM by disaggregating outcomes and objectives in the log frame and using one set of scores in bottom-up assessments. - Improve efficiency and standardization of IMF reporting to the SC. - Most recommendations require corrective actions rather than new instruments; sequencing may be needed but no specific time frame for implementation is prescribed. _Excerpted and summarized from the Mid-Term External Evaluation of the MNRW TTF (DevTech Systems, Inc.), May 2015._ ### 3. The technical assistance (TA) is demand driven and results based. Therefore, demonstrated ### mte2015i - 3. The technical assistance (TA) is demand driven and results based. Therefore, demonstrated ### TA design and objectives - TA financed by the TTF utilizes the IMF’s specialized expertise and ability to integrate policy, administrative and legislative dimensions. - The TA seeks to deliver concrete outcomes over a multi-year horizon through a programmatic approach covering the chain that transforms natural resource wealth into development. - The chain is covered through five modules, interconnected but administrable separately: - Module 1—Extractive Industries Fiscal Regime - Module 2—Extractive Industries Revenue Administration - Module 3—Extractive Industries Macro-Fiscal Policies, Public Financial Management, and Expenditure Policy - Module 4—Asset and Liability Management - Module 5—Statistics for Natural Resources - TTF activities include dissemination through conferences and workshops, and conducting research and analytical work to identify best practices and distill lessons. - TTF complements the Extractive Industries Transparency Initiative (EITI) and coordinates with initiatives such as the EITI Multi-Donor Trust Fund, the World Bank’s EITI++ Initiative, and the Natural Resource Charter. - Work is guided by a Steering Committee (SC) composed of representatives of the seven donors and IMF staff from ICD and TA departments; the SC decides by consensus and endorses an indicative annual work plan. ### Geographic distribution and project counts - Initial intended five-year geographic distribution: 44 percent to Sub-Saharan Africa (SSA); 28 percent to East Asia and Pacific (EAP); 15 percent to Middle East and North Africa (MENA); 8 percent to Latin America and Caribbean (LAC); and 5 percent to Europe and Central Asia (ECA). - Through April 2014: - 34 TA projects endorsed in 17 countries and one regional project. - By number of countries receiving TTF TA: 50 percent to SSA, 28 percent to EAP, 11 percent to MENA, and 11 percent to LAC. - By number of projects: 62 percent to SSA, 24 percent to EAP, 9 percent to MENA, 6 percent to LAC. - By endorsed budget: 62 percent to SSA, 28 percent to EAP, 5 percent to MENA, 5 percent to LAC. - By actual expenses (through April 2014): 57 percent to SSA, 29 percent to EAP, 4 percent to MENA, 10 percent to LAC. ### Module composition and spending - Module 1 was the largest module over the three-year period: - Accounted for 39 percent of budgeted expenses and close to 50 percent of actual expenses. - Modules 2 and 3 each accounted for about a quarter of the TTF’s TA. - FAD’s modules accounted for 96 percent of total TTF TA spending. ### Country coverage by modules - Projects endorsed through April 2014 were distributed across countries and modules (selected examples): - DRC, Guinea, Kenya, Iraq, Lao PDR, Mali, Mauritania, Mongolia, Mozambique, Niger, Papua New Guinea, Peru, Sierra Leone, Solomon Islands, Tanzania, Timor Leste, Uganda. - Note: One project (Mali) launched after April 2014. ### Project time frames and extensions - Average initial time frames by module: - Modules 1, 3, and 4: 24 months - Modules 2 and 5: 30 months - Out of 32 projects that started, 24 were extended. - Average extensions by module: - Module 3: 14 months - Module 5: 12 months - Modules 1 and 2: 11 months - Module 4: 9 months - Table of project time-frame statistics (projects endorsed through April 30, 2014): - Module 1: Number of projects 14; Project Proposal 25; Average Time Frame 36; Extension 11 - Module 2: Number of projects 6; Project Proposal 30; Average Time Frame 41; Extension 11 - Module 3: Number of projects 8; Project Proposal 24; Average Time Frame 38; Extension 14 - Module 4: Number of projects 2; Project Proposal 24; Average Time Frame 33; Extension 9 (Excludes Iraq (4) and Uganda (4), which did not start.) - Module 5: Number of projects 2; Project Proposal 30; Average Time Frame 42; Extension 12 - Drivers of extensions included ambitious initial time frames, acute capacity constraints (including at ministries of finance), political issues and elections, inter-agency coordination difficulties, and variable commitment to reform. ### Budget execution (as of April 2014) - Total endorsed budget: US$19.2 million - Overall execution: 42 percent of the endorsed budget had been executed. - Execution by module: - Module 1: Endorsed Budget 7527 (thousands of dollars); Actual Expenses 3893; Percent of budget 52 - Module 2: Endorsed Budget 4453; Actual Expenses 1973; Percent of budget 44 - Module 3: Endorsed Budget 4444; Actual Expenses 1807; Percent of budget 41 - Module 4: Endorsed Budget 2122; Actual Expenses 310; Percent of budget 15 - Module 5: Endorsed Budget 644; Actual Expenses 19; Percent of budget 3 - Total: Number of projects 34; Endorsed Budget 19190; Actual Expenses 8002; Percent of budget 42 - Execution by vintage (average length in April 2014 and percent executed): - FY2012: 14 projects still active, average length 30 months, executed on average 59 percent of budget - FY2013: 10 projects, average length 18 months, executed on average 33 percent of budget - FY2014: 5 projects, average length 8 months, executed on average 21 percent of budget - The evaluation noted dispersion in budget execution and evidence of overestimation of project costs in many projects; several projects had budgets cut in December 2014. ### RTAC participation - Regional Technical Assistance Centers (RTACs) participated in specific projects and areas such as treasury single account (TSA) work and training. - RTAC contributions and examples: - AFRITAC Central assisted DRC DGI with e-filing implementation and participated in HQ-led missions; assisted in public finance law implementation and capacity building outside the TTF project. - AFRITAC West provided support on cash management and accounting systems in Niger outside the TTF project. - AFRITAC East provided resident revenue administration advisor participation in Tanzania and PFM experts in module 1 mission on natural gas development. - PFTAC provided assistance under module 1 in Timor Leste. - AFRITAC East provided technical advice and implementation support to expand the TSA in Uganda under module 3. - RTACs were highlighted as an efficient alternative to LTXs, exploiting economies of scale and reducing the risk of taking over tasks that should be done by government officials. ### Evaluation approach and DAC criteria - Evaluation used a bottom-up approach: individual projects were assessed and rated; results averaged to obtain scores and ratings at project, country and TTF level across evaluation criteria. - Rating scale: 1 to 7, grouped into five categories: - Excellent (E): 6.5 to 7—outcomes and objectives were achieved - Very Good (VG): 5.5 to 6.4—outcomes and objectives were substantially achieved - Good (G): 4.5 to 5.4—outcomes and objectives were largely achieved - Modest (M): 2.5 to 4.4—outcomes and objectives were partly achieved - Poor (P): 1 to 2.4—little or no progress - ND used when insufficient information; NA used when question did not apply. - DAC criteria covered relevance; quality of TA and reporting; impact size; and likelihood of sustained reforms, among others. ### Overall performance and key metric scores - Overall rating: G with an average score of 5.2 and a standard deviation (SD) of 0.42. - Average scores by criterion ranged from: - 5.9 (VG) for relevance - 4.5 (G) for effectiveness of outcomes and objectives - Other criteria clustered between 5.1 and 5.4. ### Relevance findings - Relevance had the highest rating (VG) with average score 5.9 and SD 0.82. - Strongest sub-categories under relevance: - Consistency with TTF selection criteria - Extent to which projects address country needs and recipient governments’ priorities - Weakest sub-category: capacity to implement TA advice (existing capacity and potential capacity to be developed through TTF), aligned with findings on project extensions. - Coordination and complementarity with other TA providers, sequencing, and assessment of commitment were rated between G and VG. - Mechanisms to assess commitment included scoping and diagnostic missions; prior TA; information from area departments and other TA providers; and discussions with authorities at request time. - FAD modules rated VG with average scores higher than 6. - MCM projects rated M overall (issues in sub-categories other than compliance with selection criteria and addressing country needs). - STA projects rated VG, though progress slow for some (e.g., Mozambique active only from February 2015). ### Coordination with other TA providers - Projects rated VG on average for coordination and complementarity with other multilateral and bilateral donors. - Dimensions considered: use of prior TA; whether gaps in TA hindered implementation; conflicting policy advice; coordination of delivery and related activities. - In most projects IMF coordinated with other TA providers at inception and in-field through meetings and donor coordination groups. - Five cases were identified of conflicting or uncoordinated advice from other TA providers; outcomes ranged from significant delays to less significant disruption. - Survey responses: - 60 percent of government officials thought IMF projects were reasonably coordinated with other TA providers (40 percent had no opinion); all respondents reported they had rarely received conflicting advice. - 71 percent of managers and 85 percent of experts thought projects were well or reasonably well coordinated. - 20 percent of managers felt more could have been done to improve coordination. - 93 percent of managers and 80 percent of experts took steps to ensure TA complemented other providers and avoided duplication. - 75 percent of managers and 70 percent of experts took steps to bring in other TA providers to help implement recommendations. ### Effectiveness findings - Effectiveness assessed in two categories rated separately: - Effectiveness of outcomes and objectives (results delivered) — substantially affected by external factors. - Effectiveness of TA delivery (project design, inputs, outputs, risk management, processes) — mostly under IMF control. - Lack of well-defined timelines and milestones in project documents complicated evaluation; many outcomes had only end-dates and no intermediate milestones or prioritized components. - Evaluators identified timelines via documentation and interviews and adjusted ratings to differentiate on-track versus off-track performance (no adjustments for shocks such as Ebola or elections). - Effectiveness of outcomes and objectives is the criterion where IMF has least control due to external events and existing country conditions (e.g., competition for resource rents, political support, multiple stakeholders). - Twenty-five projects evaluated for effectiveness in achieving outcomes and objectives: - Composition: 23 FAD projects, 1 MCM project, 1 STA project. - Average score: 4.5 (G, borderline with M); FAD projects average 4.6. - SD across projects for this criterion: 1.39 (largest dispersion across criteria). - No project assessed as E; two projects had average scores of 6.3. - Distribution: close to half of projects VG to G; slightly less than 50 percent rated M; less than 10 percent rated P. - Implementation of several projects affected by elections and changes in government; 20 percent of projects affected by the Ebola epidemic. *Mid-Term External Evaluation of the MNRW TTF, DevTech Systems, Inc.* ### 29. Module 2 was the weakest performer on effectiveness on outcomes and objectives among ### Module 2 was the weakest performer on effectiveness on outcomes and objectives among ### Effectiveness — outcomes and objectives - Module 2: four out of the five projects assessed were rated M. - Module 3: rated G overall; three-fifths of projects were rated VG to G. - Module 1: large dispersion — almost half of projects rated VG and 40 percent rated M. - Factors explaining variation: - Disagreements on recommendations regarding establishment of NR revenue units (or LTOs) within existing revenue authorities or on consolidating revenue collection responsibilities. - Issues arising from inadequate fiscal regimes. - Differences in scope and ambitiousness of outcomes and objectives. - Widely diverse initial conditions in resource sectors and political economy of reform. - Interagency cooperation, capacity, relations with resource companies, political environment, turnover of officials. ### Inputs, outputs, risk management and processes - Overall criterion: evaluated as G with a score of 5.2 and a SD of 0.71 (second lowest for the DAC criteria). - Lowest sub-category: risk management — risks often identified pro-forma or in generic terms; mitigating measures not explicitly incorporated into project design. - Notable exception: Iraq (Module 1). - Few proactive measures to address unforeseen risks that materialized during implementation (e.g., tensions among agencies). - Best-performing sub-categories: inputs and outputs. - Processes sub-category: - Modules 1 and 3: middle range. - Module 2: rated VG for processes but had the lowest rating for risk management. - Positive factors expediting implementation: joint action plans with timelines, continuity of engagement, hands-on TA delivery including STXs and RTACs. ### Efficiency - Definition: achieving outcomes and objectives at the lowest possible cost given project characteristics; influenced by TA delivery adequacy and adequacy of reporting/documentation to the SC. - Efficiency of TA delivery: - Rating: G with an average score of 5.2 and an SD of 0.71. - Distribution: one project rated E; 40 percent of projects rated VG; 40 percent rated G. - All three FAD modules: similar performance with G rating; module 2 assessed slightly lower. - Efficiency of reporting: - Rating: G with an average score of 5.4 and SD of 0.61 (lowest SD across criteria). - More than half of projects rated VG and 36 percent rated G. - Areas for improvement: timeliness and standardization of information in project summaries, proposals and assessments. - FAD modules: modules 1 and 3 rated VG; module 2 rated G (borderline VG). ### Impact - Criterion rating: G with an average score of 5.1 and SD of 0.99 (second highest SD). - Positive impacts observed: - Improved outreach to government and civil society via workshops and consultations on draft legislation. - Improved practices to manage natural resources (NR), sometimes improving inter-agency cooperation. - Broader impacts: improvements in resource investment environment, formalization of transactions, revenue administration indicators, and transparency of fiscal frameworks. - Dispersion in impact reflects factors outside IMF control (e.g., parliamentary approval of legislation). - FAD modules: all three rated G; more than half of projects rated E to VG. - Changes in petroleum and mining regimes can generate more visible impact than changes in tax revenue administration or PFM. - Contribution to international standards: - Assessment for whether TA contributed to improving/exploring international standards in recipient countries: G, borderline with VG. - Evidence: country project results profiled in international conferences; regional workshops; dissemination of TA reports. - Contribution of TTF country work to broader international standard improvements discussed in Appendix III (research projects and workshops). ### Sustainability - Criterion rating: G with an average score of 5.3, borderline with VG; SD of 0.72. - Strongest aspect: integration of the project into countries’ reform agendas — rated VG; reflects integration with other IMF-related activities such as non-TTF TA. - Weakest aspect: capacity building — need for sustained hands-on TA and long-term effort. - All three FAD modules rated G. - Module 1: half of projects rated VG. - Modules 2 and 3 combined: only 30 percent of projects rated VG or better. - The multi-year/modular approach enables integration of TTF TA into reform agendas and coordination with other TA providers; capacity building and integration into reform agendas necessary for sustainable impact. - Likelihood that changes brought about by TA will be sustained: scored 5.3 (borderline with VG) due mainly to integration with reform agenda. ### Comparison of DAC criteria across respondent types - Table summary (scores and ratings preserved as reported): - Relevance: Project Managers E (6.6) ... Experts VG (6.2) Government Officials VG (5.9) - Effectiveness Outcomes and Objectives: PM G (4.9) Experts VG (5.6) GOs VG (5.5) Evaluation Team G (4.5) - Effectiveness Project Design (I, O, RM, P): PM G (5.1) Experts VG (5.8) GOs VG (5.6) ET G (5.2) - Efficiency of TA Delivery: PM VG (5.6) Experts VG (6.1) GOs VG (6.0) ET G (5.2) - Impact: PM VG (5.9) Experts VG (5.7) GOs VG (5.5) ET G (5.1) - Sustainability: PM M (4.3) Experts VG (5.5) GOs VG (5.6) ET G (5.3) - Overall: PM G (5.4) Experts VG (5.7) GOs VG (5.7) ET G (5.2) - Survey response bias note: Government Officials participation rate was 23 percent (about half that of PMs and Es), possibly biasing GO scores upward. - Observed pattern: - Project Managers and the Evaluation Team tended to rate several criteria lower (notably effectiveness outcomes and sustainability) compared with Experts and Government Officials. ### Activity-level and country/module findings - Summary evaluation ratings by country and module preserve reported labels (examples): - Kenya: Module 1 — VG (6.3) overall; Module 1 component scores include Relevance 6.3 and other scores as reported. - Mongolia: overall VG/G/VG/VG across modules where evaluated. - Tanzania: overall G with module-level variation including an E in one module where only Relevance was evaluated. - Kenya case highlights: - Extractive industries recently assumed major importance; project complementary to East African projects under the TTF and FAD TA. - Government established an inter-agency committee; development partners set up an EI coordination group with IMF RR participation. - Outcomes almost fully achieved except for adequate capacity building at the Ministry of Mining within original timeframe and budget. - EI fiscal regime reform largely followed project recommendations; the new Production Sharing Agreement (PSC) scheme with the Ministry of Energy and Petroleum implements a scheme according to technical notes prepared under the project. *Mid-Term External Evaluation of the MNRW TTF, DevTech Systems, Inc.* ### 51. On legislation, in collaboration with the National Treasury (NT) and the Kenya Revenue ### mte2015i - 51. On legislation, in collaboration with the National Treasury (NT) and the Kenya Revenue Authority (KRA) ### Legislative and regulatory outcomes - A draft of new income tax provisions for mining and petroleum was completed in collaboration with the National Treasury (NT) and the Kenya Revenue Authority (KRA). - The revised Income Tax legislation was published in late 2014. - A full review of the draft Mining Bill was conducted; the National Treasury (NT) adopted the advice provided, and legislation was enacted in 2014. - Comments were provided to the new Petroleum Bill prepared by the World Bank. ### Capacity building and technical assistance (TA) activities - The project trained government officials on the FARI model and transferred petroleum FARI simulation models to agencies. - The National Oil Corporation of Kenya (NOCK) is operating the petroleum FARI model actively. - The FARI model is being simplified to accommodate weaker capacity in the Ministry of Mines (MOM) and the Petroleum Department at the Ministry of Energy and Petroleum (MEP). - Recommendation observed: participation in workshops by an expert in basic skills such as Excel or hands-on assistance to the MOM could have improved MOM participation. ### Project launch, delivery modalities, and stakeholder engagement - The project was launched with a diagnostic mission in February 2013. - Timelines for milestones in the reform program were formally defined. - TA delivery forms included: - workshops, - hands-on assistance, - advisory relations with government officials, - review of drafts, - creation of lateral communication channels among key agencies, - regular dialogue at Cabinet Secretary level, - consultations with private companies and CSOs. - The team held extensive consultations across government and with private sector and civil society representatives; these consultations contributed to fast-track approval of the legislation. - Workshops were delivered at the right level, contributing to government official motivation to be involved. ### Stakeholder commitment and uptake - Commitment to reforms of extractive industries (EI) fiscal regimes from NT, MEP, and KRA has been strong since the new government elected in 2013 reformed the structure of government. - Agencies such as NOCK are actively using the FARI model. - Staff capacity in the NT is under enhancement to use the FARI model. ### Visibility and dissemination - The project received a high profile at the “Kenya Rising” conference in Nairobi in September 2013. - The project was also highlighted at the EAC workshop on fiscal management of oil and gas in East Africa in January 2014. ### Views of government officials - Officials strongly praised the quality of the TA and highlighted its high professional level. - The IMF consulted officials on sequencing and took into account their preferences, with jointly agreed milestones. - Officials appreciated: - continuity of activities, - intensive and frequent interaction that helped keep momentum and achieve timely results, - the IMF’s flexibility in responding to their needs. - Technical-level staff was highly satisfied with the FARI model and the training provided to transfer technology and knowledge to Kenya. - Some senior officials suggested regional advisors with resource expertise from AFRITAC East would be helpful. - Officials who attended Workshop 1 in Tanzania were very satisfied with its format and contents, especially the provision of ample time for presentations from African officials. *Mid-Term External Evaluation of the MNRW TTF* ### 76. To evaluate outcomes and objectives at the project level, the TTF uses a scale from 0 to 4 ### mte2015i - 76. To evaluate outcomes and objectives at the project level, the TTF uses a scale from 0 to 4 ### Evaluation of the present RBM (results-based management) system - The Mid-Term External Evaluation of the MNRW TTF notes the TTF uses a 0 to 4 rating scale where: - 0 is not started - 1 is not achieved - 2 is partly achieved - 3 is largely achieved - 4 is fully achieved - Top-down approach limitations: - Aggregation obscures project-level patterns; the June 2014 Annual Report indicated that over half (17 out of 30) of targeted module outcomes were already fully achieved or largely achieved and that almost a third (8 out of 30) of targeted country-module outcomes had been partially achieved, but lack of information on modules and outcomes per module makes interpretation ambiguous. - The 0–4 rating is applied without timeline adjustment: an outcome receives the same rating whether the project has been in place for months or years, so indicators can improve over time even without new projects (as projects naturally progress). - When new projects are added, it is impossible to distinguish progress due to maturation of old projects from effects of new projects. ### Proposed bottom-up and complementary metrics - Proposed approach: complement top-down with a bottom-up approach aggregating results of individual project components (targeted outcomes) to generate composite results. - Design features of the bottom-up approach: - Each TTF project consists of one or more project objectives; each objective consists of one or more targeted outcomes. - Indicators must be SMART: Specific, Measurable, Actionable, Relevant, and Time-Bound. - Weighting system: - Each targeted outcome is weighted based on its relative importance within one module of a project. - Weighting across projects will be based on the budget for each project. - Two bottom-up scores proposed: - Composite score: a weighted average of individual scores for each targeted outcome; can be produced for each level (project module, type of module, region, TTF overall). It does not adjust for project life-stage; can change due to mixture of new and old projects. - On-track score: aggregates performance relative to project expectations by defining timelines for outcomes; calculated as the difference between actual score for the assessment period and the target score. ### Evaluation of the proposed system and issues identified - Strengths identified: - Produces quantitative indicators to assess TA implementation progress. - Improves standardization of outcomes across projects and comparability of assessments. - Weighting outcomes at project level gives a better picture of likelihood of achieving objectives and differentiates project performance better than simple averages. - Concerns and potential weaknesses: - Redundancy: unclear why two separate scores (composite and on-track) are needed rather than a single score that incorporates timeline/milestone features. - Ambiguity when composite and on-track scores move in opposite directions at aggregate level—interpretation of overall TTF performance would be unclear. - Using project budgets to weight outcomes across projects may introduce distortions: - Question posed: why should aggregate score differ if a small country with a small project budget achieves outcomes versus a large country with a large project budget? - Actual expenses can differ significantly from budgets; uncertainty whether weights should use budgeted amounts or actual expenses to date of assessment. - Example: Solomon Islands achieved objectives using only 21 percent of its budget, which was only a little more than 1 percent of the total TTF endorsed budget—raising the question whether crediting should reflect cost-efficiency. - Empirical exercise by the evaluation team (cost-per-point of effectiveness): - Module 1: cost ranged from a minimum of US$10,000 to US$151,000, with an average of US$79,000 and a median cost of US$80,000. - Module 2: cost ranged from US$59,000 to US$155,000, with an average of 92,000, and a median of US$84,000. - Module 3: cost ranged from US$41,000 to US$86,000, with an average of US$60,000 and a median of US$83,000. - Open question: how the top-down approach will be complemented by the bottom-up approach in reporting (e.g., whether both sets of scores will be used in the log frame). ### Governance structure and reporting practices relevant to RBM - Steering Committee (SC): - The SC (donor representatives and IMF staff) provides strategic guidance, endorses an indicative annual work plan, reviews progress and performance, and coordinates TA on managing natural resource wealth among TTF, donors, and stakeholders. - The SC is chaired by donors with a strong interest in TTF work and meets in person annually, with at least one additional mid-year meeting via video conference/teleconference; ad hoc meetings may be convened by the chair. - SC decisions are by consensus; if consensus cannot be reached, a proposed decision is postponed or withdrawn. - Secretariat and IMF reporting: - IMF staff serve as Secretariat: support SC work, propose initiatives, and provide information via annual and mid-year reports and a Donor Gateway. - TA departments implement the Work Plan, report progress, and propose new TA, research projects, and workshops. - Project submission and approval rules: - Proposed projects are submitted for SC endorsement via a Project Summary (PS); after endorsement a Project Proposal (PP) is made available to the SC through the Donor Gateway. - Approval of the PP by ICD activates the project's budget. - Changes to the Work Plan requiring SC endorsement include budget changes in excess of 10 percent, project extensions for periods longer than six months, changes to objectives of projects, and changes to the countries in the Work Plan. - There is a three-week waiting period for approval on a lapse-of-time basis. - Project budget changes approved in the annual WP by less than 10 percent and extensions to projects by less than six months can be made without SC endorsement. - Project Assessments (PAs) for annual SC meetings provide progress, performance ratings, notes for future work, and budget execution status; starting June 2014, PAs include risk and mitigation assessments across six standardized risk categories. - Other reporting: one-page project summaries (since June 2014), tables with selected results in annual reports, brief case study summaries, and monthly updates on TA mission activity to the Donor Gateway. - Governance findings relevant to RBM: - The SC has provided strategic guidance and monitored results; most donors are active and engaged. - SC priorities align with Program Document objectives; SC members emphasize roles in shaping strategy and monitoring project results, and providing guidance on country and project selection. - The SC has guided on country eligibility, selection criteria, research proposals, the RBM, donor coordination, and IMF reporting. - The SC’s focus on results has encouraged improvements in RBM log frame and standardization efforts; a 2014 workshop brought donors and IMF staff together on these issues. - IMF reporting to the SC is generally timely and appropriate but could be strengthened with more detail on main implementation problems, potential donor contributions, and complementarities with parallel IMF TA (e.g., TPA TTF, RTACs). - Project Summaries (PSs) and Project Proposals (PPs) vary in comprehensiveness; scoping missions at inception could strengthen PSs and PPs. - PPs and PAs require further standardization: distinctions between objectives and outcomes are sometimes unclear; inconsistent use of priorities, weights, and rating interpretations; occasional outcome changes during implementation without discussion in PAs. - Donor Gateway and mission reporting systems need usability and timeliness improvements; SC members reported difficulties finding materials and uncertainty about whether latest information was uploaded. ### Key policy-relevant implications and recommendations (from the evaluation) - Strengthen standardization and clarity in project design and reporting: - Standardize formulation of objectives and targeted outcomes, ensure SMART indicators, and clarify distinction between objectives and outcomes. - Use scoping missions to improve the comprehensiveness and specificity of Project Summaries and Project Proposals at inception. - Simplify bottom-up metrics: - Consider integrating timeline/milestone features into a single score that captures both achievement and on-track performance to avoid dual-score ambiguity. - Define milestones for outcomes so assessments can be made relative to milestone achievement (i.e., an outcome can be rated 4 if a milestone is achieved even if final completion remains). - Revisit aggregation weights and metrics: - Reassess whether budgets or actual expenses should be used as weights when aggregating scores across projects to avoid distortions and to recognize cost-efficiency (e.g., projects achieving outcomes using a small percentage of budget). - Consider using simple averages rather than budget-weighted aggregates if the objective is to treat project performance symmetrically across countries regardless of project size. - Improve IMF reporting and donor information systems: - Enhance PAs and PPs with clearer documentation of implementation problems, risk assessments, and complementarities with other IMF TA and donor programs. - Improve Donor Gateway usability and timeliness (better document search by country/module and clearer upload/version indicators). - Strengthen SC role in quality control: - Use SC engagement to promote standardization, ensure project timelines and milestones are clearly defined, and monitor consistency of rating interpretations across projects. *Mid-Term External Evaluation of the MNRW TTF — DevTech Systems, Inc.* ### 107. The evaluation of the TTF’s TA projects shows a solid and relatively even performance ### The evaluation of the TTF’s TA projects shows a solid and relatively even performance ### Overall performance and program outputs - The Mid-Term External Evaluation of the MNRW TTF finds the TTF’s TA projects show a solid and relatively even performance across the DAC criteria. - The TTF scored highest on the relevance of TA projects. - Performance on the effectiveness of objectives and outcomes was somewhat weaker and showed larger dispersion across projects; the evaluation notes this is the DAC criterion where the IMF has the least control on performance. - The TTF’s research and workshop program produced important products contributing to dissemination of findings and practical guidance on resource management relevant to TTF eligible countries. - The evaluation was not in a position to evaluate some workshops beyond their relevance. - Where ratings could be assigned, research projects and workshops showed considerable dispersion: - Two research projects and one workshop were rated favorably for impact. - One project and one workshop were assessed to have contributed to the improvement of international or regional standards in EI. - The evaluation notes occasional tensions between objectives to contribute to international EI standards and focusing on TTF-eligible developing countries. ### Findings at TTF level — standardization - The evaluation identifies areas where greater standardization would facilitate project management, cross-project comparison, and manager learning: - Definition and disaggregation of objectives, outcomes, verifiable indicators and completion dates, including on the weighting and prioritization system used. - Methodology used to rate performance on outcomes and objectives. - Designing timelines, including milestones. - Identification of risks and mechanisms to address risks that develop during implementation. - Handling delays in project implementation. - While project manager documents and IMF reporting to the SC deliver timely and appropriate information, improved standardization of documentation would enhance monitoring and quality control and the SC’s role in endorsing projects, monitoring progress and assessing results: - Improve and standardize the comprehensiveness of project summaries, proposals and assessments. - Standardize the way information is conveyed in the event of delays and/or changes in the definition of outcomes or in their completion dates, and the effect of such changes on timelines, milestones and performance. ### Findings at TTF level — MCM and STA modules - The contribution of MCM and STA to the TTF's TA has so far been very limited (STA's contribution will increase with the global project). - This limited contribution raises questions about whether modules 4 and 5 are seen by countries’ authorities as addressing their priorities and needs. ### Findings at TTF level — synergies, coordination, and LEG - The boundary between modules 1 and 3 has not always been clear; financial models of resource revenues were sometimes transferred from module 1 to module 3. - Some module 2 projects were hampered by lack of reform or delays in reform of inadequate fiscal regimes. - Coordination between modules 3 and 4, including on SWFs, could in some cases have been tighter. - LEG support was sometimes requested at short notice, affecting LEG’s assignment of experts due to limited availability of specialized EI legal expertise. ### Findings at TTF level — budget execution and inactive projects - There is great dispersion among projects regarding budget execution; some overestimation of project costs appears to have occurred. - A dozen projects had their budgets cut in December 2014. - The evaluation recommends a more proactive and structured approach, including termination, for projects endorsed but not launched or launched but inactive. ### Findings at TTF level — RBM and governance - The top down RBM system suffers significant weaknesses; it does not allow for a good understanding of progress over time. - The proposed bottom-up approach raises concerns: - It is not clear why two sets of scores, composite and on-track, are necessary. - The two sets of scores cannot be aggregated and could move in opposite directions, creating interpretation problems. - The proposal to weight projects by their budgets could introduce distortions. - Governance is sound: - The SC has provided strategic guidance and monitored results, taking an increasingly proactive role. - IMF reporting to the SC delivers timely and appropriate information but could be further strengthened. ### Findings at project level — external and domestic constraints - Implementation of TA advice can be affected by factors outside IMF control, including: - External events: elections, legislation approval by parliament, changes in government, continuity of relevant officials. - Shocks: Ebola. - Existing conditions that pose challenges include competition for resource rents and non-transparency, political support variability, and multiple stakeholders with different objectives. - Authorities’ actions depend on a broad array of circumstances (homogeneity of reform objectives within government, commitment, capacity, inter-agency cooperation) which the IMF can influence to some extent but may require more time and effort than anticipated. ### Findings at project level — characteristics of successful projects - Documentation indicates that many of the most successful TTF projects (examples include Kenya (1), Mongolia (2), Guinea (1), DRC (1), Sierra Leone (1) and Peru (3)) shared some TA characteristics: - Pre-project proposal mission or extensive discussions with authorities to delineate the TA. - Focused, realistic and well-defined objectives and outcomes aligned with existing capacity and potential capacity building. - Joint preparation of action plans and project sequencing with timelines and milestones. - Continuity of engagement, including intensive use of STXs. - Strong emphasis on capacity building and hands-on training. ### Project design and implementation weaknesses - Outcomes and objectives were sometimes overambitious relative to country implementation capacity and available TA resources, causing shortfalls in implementation. - Overambitious targets may stem from objectives and outcomes being defined at a very aggregated, generic level and carried into the RBM log frame. - Many projects required extensions beyond original completion dates due to: - Unrealistic initial time frames given scope and complexity. - More acute capacity constraints than anticipated. - Political issues and inadequate inter-agency cooperation. - Varying degrees of commitment to reform across agencies and over time. - Project summaries and proposals often lack well-defined timelines and milestones, making monitoring ad-hoc; completion dates for outcomes are often set only for the end-date of the project with no intermediate milestones or prioritized components. - Identification of project risks and proactive mitigation measures has largely been done in generic terms and needs improvement. - IMF made important efforts to ensure coordination and complementarity with other TA providers. ### RTACs role - In projects with RTAC participation, RTACs were instrumental in providing TA in specific areas and contributed to continuity in implementation and training. - RTACs constitute an efficient alternative to LTXs: - Economies of scale via RTAC-based regional experts. - Less likely to take over work that should be done by government officials (a risk posed by LTXs), thus strengthening ownership by authorities and technical personnel. ### Recommendations — summary and implementation context - The evaluation presents a set of strategic recommendations aimed at accelerating project implementation, and increasing the effectiveness and efficiency of TA delivery to enhance the TTF’s contribution to helping countries maximize benefits from natural resources. - Most recommendations require corrective actions rather than new instruments and can be implemented in the normal course of TTF work; sequencing may be needed but no time frame is prescribed. ### Recommendations at TTF level — Recommendation No.1: Appointing a supra-project manager in FAD - Proposal: appoint a supra-project manager with technical responsibilities on a full time basis in FAD—distinct from ICD’s administrative responsibilities—financed by the TTF. - Expected effects: - Standardize staff recommendations for projects encountering difficulties (scope and budget). - Distill lessons applicable across projects. - Partly offset cost by reduced workload on FAD project managers and improvements in budget management and project duration. - Functions of the supra-manager would include: - Dealing with all issues related to standardization. - Ensuring compliance with standardization in all projects through detailed review of all documents prepared by project managers, including for consistency with previous documentation, and across documents and years. - Ensuring compliance with the methodology used to assess project performance, both in the indicators for the RBM as well as the qualitative statements included in the project assessments. - Conducting the top down RBM exercise; providing assistance to project managers in the bottom-up exercise; consolidating the bottom-up results by outcomes, projects, modules, countries, regions. - Ensuring detailed assessment of project risks, of the risks that develop during project implementation, and responses to the risks. - Deciding together with project managers the transfer of inactive projects to an inactive project list, and on the early termination of projects which have not been launched within a year of endorsement and projects in the inactive list for a year. - Distilling lessons from the project assessments that can be applied across projects, both new projects and projects being implemented. ### Recommendations at TTF level — Recommendation No.2: MCM and STA modules - For efficiency reasons, consider providing this type of TA support through IMF non-TTF TA. - In respect to asset management, differently from establishment of a SWF (addressed by module 3), there is no need to put in place an investment framework and define an investment policy years in advance of actual implementation. ### Recommendations at TTF level — Recommendation No.3: Improving synergies and coordination - Modules 1 and 2: - Where inadequate and nontransparent fiscal regimes could significantly hamper progress under module 2, TA under module 1 could be made a requirement for provision of TA under module 2. - Modules 1 and 3: - Better synchronization requires high coordination with regular exchange of information. - Resource revenue experts who worked on module 1 should, to the extent possible, move to module 3 work when models are transferred until module 3 incorporates models into macro-fiscal modeling and forecasting. - Coordination with LEG: - LEG planning for deployment of resources and external experts depends on FAD planning; better planning and coordination would help LEG planning and enhance LEG's contribution. ### Recommendations at TTF level — Recommendation No.4: Improving project budgeting - Given the size of budgets under execution in many projects, including those launched in FY2012, TTF management should analyze individual projects to determine if there has been overestimation of costs, the reasons for it, and apply lessons learned to future budgeting. ### Recommendations at TTF level — Recommendation No.5: Dealing with inactive TA projects - Inactive project reserve list: - Inactive projects are those never launched or launched but with no activity for six months to a year. - Consider transferring the latter to an inactive project list with parallel transfer of unused budget funds back to the TTF; reactivation would be proposed to the SC if circumstances change. - The inactive project list would be assigned funds in the corresponding FY Work Plan depending on the number and size of projects in the list. - Alternatively, projects in the inactive list could retain around 20 percent of the unused funds in case they are reactivated within a year in the list. - Termination of projects: - Consider terminating projects never launched within a year of endorsement, and projects in the inactive list after a year on the list. - Early termination would allow redeployment of scarce TTF resources and more efficient fund management; this would benefit from the supra-manager’s comprehensive program view. ### Recommendations at activity level — Recommendation No.6: Project design, planning and TA delivery - Scoping missions: - Initial missions prior to submitting project summaries are recommended where there has been little or no IMF TA in the area in recent years, where clarification of objectives and authorities’ capacity is needed, or where previous TA take-up has been inadequate. - Scoping missions would help produce more focused project summaries, reduce differences between project summaries and proposals, and improve project budgeting. - Scoping missions could be financed from a dedicated TTF budget reserve, as done in some bilateral TTFs. - Project objectives and outcomes: - Objectives need to be focused and realistic given implementation capacity and available TA resources. - Consider sharing draft project proposals with authorities for input before finalization. - Project planning: - Improve planning by setting realistic time frames that account for capacity building and applying lessons from relevant TTF experience. - Resist pressure from authorities to adopt unrealistically short time frames. - Scoping missions would help narrow objectives to pragmatic priority reforms. - Timelines and road maps: - Prepare timelines with milestones for inputs, outputs, outcomes and objectives prior to preparing project summaries; high rate of return in performance. - Roadmaps should be agreed where relevant for sequenced processes and milestones, and should form the basis for defining milestones and verifiable indicators to facilitate implementation and monitoring progress. - Risk assessments: - The evaluation emphasizes the need for improved identification of project risks and proactive mitigation measures (discussion continues beyond the extracted text). *Source: Mid-Term External Evaluation of the MNRW TTF (DevTech Systems, Inc.)* ### 141. Strengthening risk assessment of TA implementation at the onset of a project would help ### 141. Strengthening risk assessment of TA implementation at the onset of a project would help ### Risk assessment and project design - Strengthening risk assessment at project onset would help design projects more effectively to address identified risks, and apply measures if those risks materialize. - The project proposal for Iraq is cited as an excellent example of the kind of risk assessment envisioned. - The TTF has already moved in this direction with project assessments and one-pagers. - Project summaries (PSs) should: - Be comprehensive and detailed. - Explain clearly the authorities’ needs and why the IMF proposes to undertake the TA under the TTF. - Include risk assessments of TA implementation and mitigation measures similar to those now included in project execution documentation. - Be revised and resubmitted to the SC for endorsement if there are significant changes in scope or budget between the PS and the PP, including changes in project objectives or budget above 10 percent during implementation. - Consider inclusion of finalized PPs in reports to the SC. ### TA delivery modes - TTF TA should rely on a balanced combination of activities that will in many cases change over the life of the project. - Initial stages: - HQ missions at a strategic level are needed to define the reform strategy, including through TA reports. - Later stages: - Active use of STXs to assist with implementation and capacity building. - Where relevant and feasible, use of RTACs to assist with implementation and training. - Continued strategic advice and participation of HQ staff in hands-on problem solving and implementation issues. ### Continuity of TA engagement and capacity building - Continuity of TA engagement helps generate project momentum, more buy-in, and commitment to reform. - There is scope for more intensive use of STXs; STXs carrying out work plans agreed with IMF HQ and the authorities can provide ongoing support to sequenced reforms. - STXs are key for capacity building through hands-on training and organization of focused workshops for relevant officials. - Officials in the three countries visited emphasized: - The importance of continuity of engagement. - The positive role of STXs and hands-on training relative to general workshops with wide participation. - RTACs and regional experts can strengthen capacity building features of projects. ### Recommendation No. 7 — Use of RTACs and Regional Advisors - TTF activities could be better integrated with the RTACs. - RTACs could participate more actively in TTF work, particularly in implementation and capacity building (not in strategy or policy). - RTAC contributions: - Provide follow-up and support for implementation of recommendations. - Send experts to countries to help maintain continuity of inputs and expand capacity of specialized assistance from FAD and STXs. - Assist with training (for example with FARI) via regional workshops or individual country hands-on training conducted by RTAC peripatetic advisors, exploiting economies of scale. - Regional advisors could be attached to the RTACs and financed by the TTF. - Coordination improvements: - More systematic sharing of early information about TTF work plans for the next fiscal year and needs for RTAC participation would help HQ and RTAC planning. - Multi-year work plans for TTF projects agreed with the authorities should include plans for RTAC participation if relevant. - In some countries, TA in the same area from the TTF and from RTACs could be better coordinated. ### Recommendation No. 8 — Research Projects and Workshops - Research agenda: - Objectives of the TTF's research project agenda could be more clearly defined in terms of the operational needs of the TTF. - Consider aligning research projects more closely to TTF objectives, focusing on issues faced by low income and lower middle income producers. - Need to clarify tension between contributing to improvement of international standards in EI and focusing research on TTF-eligible developing countries. - Given the requirement that research papers issued by the IMF must be cleared by IMF staff, and based on experience with Research Project 1, consideration should be given not to outsource TTF research papers. - Independent advice on policy issues could be obtained through STXs that provide inputs for research papers. - Workshops: - Regional workshops provide dissemination opportunities for lessons from the TTF's research and TA and focus on practical issues faced by participating countries. - Positive experiences cited: regional workshops in Tanzania and the Andean Regional project under module 1. - Benefits include peer-to-peer learning and takeaways from experience of other countries. - Ample participation of government officials as presenters, discussants, and panelists enhances relevance and effectiveness. - Consider breakout sessions on highly technical issues. ### RBM and reporting — Recommendation No. 9 (Enhancing the RBM) and No. 10 (Enhancing reporting efficiency) - Recommendation No. 9 — Enhancing the RBM as a monitoring tool: - Modify the log frame used for the top down approach to define objectives and outcomes in a more disaggregated fashion to allow better customization of projects, including on verifiable indicators. - Disaggregation will allow better definition of intermediate milestones. - There should be only one set of scores for the RBM’s bottom up approach that incorporates features of both the proposed composite and on-track sets of scores. - Establish a well-defined timeline so milestones can be defined for outcomes and assessment can be done with respect to achieving the milestone rather than the full outcome (example: if the project achieves the milestone, the outcome will be assessed as 4 even if it still has some way to go). - RBM scores need to be complemented by qualitative statements and contextual information regarding implementation, including: - Developments under the project. - The authorities’ efforts. - Main factors behind success or slow progress. - Prospects for the projects going forward, including risks faced by project implementation and factors beyond control of the authorities and the IMF. - Recommendation No. 10 — Enhancing the efficiency of reporting: - Standardize PPs and PAs further as a precondition for a successful RBM, including: - Formulation of objectives, outcomes and verifiable indicators in PPs. - Assignment of completion dates and of weights to outcomes. - Make interpretation of achievement ratings for objectives and outcomes in PAs uniform across projects. - Note and discuss in the PA text any changes to outcomes in PAs relative to the PP (number, contents, or completion dates). - Changes to project objectives should result in a revised PS submitted to the SC for endorsement as required in the Operational Guidelines, and a revised PP. - PAs should provide a balance of information between inputs and results, and between recent developments and earlier progress, including updated information on the country’s EITI status. - Final assessments for projects completed or terminated could include a section on lessons learned. - All project documents should be dated and updated for changes in project dates or budget. - Systems improvements: - Donor Gateway should be made more user-friendly to facilitate finding documents, including searching by country or by module; timeliness of placement of documents in the gateway should be ensured. - Mission reporting system should provide more information on missions and TA reports. ### Evaluation process, scope, and methodology (Appendix I) - Purpose and scope: - Purpose: assess extent to which the MNRW TTF is delivering planned objectives and outcomes at both activity and trust fund level, identify lessons learned, and make recommendations for improvements. - Evaluation rated performance of projects with respect to DAC criteria: relevance, effectiveness, efficiency, impact, and sustainability. - Study evaluated activities launched from April 2011 to April 2013; projects launched from May 2013 to April 2014 were only assessed for relevance. - From April 2011 through April 2014, the SC endorsed 34 modules (programs) in 17 countries and one regional project, five research projects and four workshops. - One TA project was launched after April 2014, leaving 33 projects to be evaluated. - Five of these projects were launched after April 2013 and only their relevance is evaluated. - Assessment covers developments for the period through end-2014 as reported in the December 2014 Report to the SC. - Methodology: - Bottom-up approach: each project assessed using questions grouped according to DAC criteria and sub-categories; questions customized to reflect TTF features and recipient country characteristics. - Data collection methods: desk review, interviews, surveys, and country visits. - Data and documents reviewed (desk review): - Annual and mid-year Steering Committee Reports (8) through December 2014. - Minutes of the SC meetings (9) through December 2014. - Project Proposals for TA projects (31). - Project Assessments for TA projects for 2012, 2013 and 2014. - Project Proposals for Research Projects (4) and Workshops (3). - Project Assessments for Research Projects and Workshops for 2014. - TA Reports (50) 2011-2014. - STXs Notes (3). - AFRITAC East 2014 Annual Report. - Based on documents reviewed, the team prepared a file for each of the 16 countries and the regional project, the 4 research projects and the 4 workshops that were evaluated. - Interviews and surveys: - Team met first with project managers as a group; eleven managers participated. - Interviews conducted with project managers of the 34 projects over a two-and-a-half week period, including participation by LEG in projects where it had been particularly active. - A questionnaire was sent in advance to interviewees. - Team met with FAD and MCM senior management, FAD staff in charge of the RBM exercise for the TPA-TTF, staff in charge of individual research projects and workshops, and staff managing the TTF budget process. - Team met with World Bank senior management on TA on NR, and staff in charge of one of the major projects on NR in SSA, and interviewed by phone a US Treasury official active on one of the TTF projects. - Representatives of five of the seven donors in the SC were interviewed on the phone by the evaluation team. - Surveys response rates: - Project managers: 46 percent (16 out of 35). - Experts: 43 percent (13 out of 30). - Government officials: 23 percent (7 out of 30). - Country visits: - Team visited Mongolia, Tanzania and Kenya. - Meetings held with government officials, the IMF office in Mongolia, AFRITAC East and IMF Resident Representatives, and with other TA providers such as AfDB, DFID, EU, Norway, and the World Bank. - Processing and analysis: - Desk review used to identify project timeline from PS to PP to PAs and one-pagers to comprehend rationale, objectives, outcomes, risks, inputs/outputs, progress, and reasons for fast/slow/no progress. - Material from desk review and interviews used to design a questionnaire to evaluate projects by DAC criteria and to standardize evaluation across countries/projects. - Rating system: - Ratings used: 1 to 7, grouped into five categories for effectiveness in achievement of outcomes and objectives: - Excellent (E): 6.5 to 7 — outcomes and objectives were achieved. - Very Good (VG): 5.5 to 6.4 — outcomes and objectives were substantially achieved. - Good (G): 4.5 to 5.4 — outcomes and objectives were largely achieved. - Modest (M): 2.5 to 4.4 — outcomes and objectives were partly achieved. - Poor (P): 1 to 2.4 — little or no progress in achieving outcomes and objectives. - ND utilized if there is not enough information to substantiate a rating; NA utilized when the question did not apply. - The five-category 1–7 system converts into the 1–4 system used by the TTF for projects that started as: - E: 6.5 to 7 is 3.8 to 4. - VG: 5.5 to 6.4 is 3.3 to 3.7. - G: 4.5 to 5.4 is 2.8 to 3.2. - M: 2.5 to 4.4 is [conversion range indicated in text]. *Mid-Term External Evaluation of the MNRW TTF (DevTech Systems, Inc.)* ### 1.8 to 2.7; and (e) P: 1 to 2.4 is 1 to 1.7. ### 1.8 to 2.7; and (e) P: 1 to 2.4 is 1 to 1.7. ### I.2.3 Evaluation Criteria — Overview - Evaluation categories: Relevance, Effectiveness, Efficiency, Impact, Sustainability. - Ratings for composite categories often computed as averages of specified sub-categories. ### Relevance — definition and sub-categories - Relevance relates to: - consistency with the MNRW TTF's selection criteria; - addressing the country's needs and recipient government priorities; - the country having the capacity or potential capacity to implement advice; - coordination with and complementarity to other projects and programs (multilateral, bilateral, other country initiatives); - appropriate sequencing with other TA, including other modules under the TTF; - substantive evidence of commitment by the authorities. - Six sub-categories (S) used to assess relevance (average rating across S1–S6): - S1: consistency with TTF selection criteria (country eligibility list, EITI candidate or EITI compliant, TA requested by authorities); - S2: addressed the country’s needs and recipient country’s priorities; - S3: capacity and potential capacity to implement advice; - S4: coordination and complementarity with other TA providers; - S5: appropriate sequencing with other TA, including other projects under the TTF; - S6: assessment of authorities’ commitment. ### Effectiveness — definition and assessment components - Effectiveness measures the extent to which TA attains its objectives and delivers results, including whether recipient authorities take actions to achieve outcomes and whether risks are identified and addressed. - Two categories assessed separately: - Effectiveness of outcomes and objectives (extent project delivers results). - Effectiveness of TA delivery (the “how” of project design), assessed across four sub-categories with the category rating as the average of those sub-categories. - Components evaluated under effectiveness of TA delivery: - Inputs: - missions (scoping, diagnostic, TA delivery), STXs, regional experts, RTAC work, workshops, hands-on assistance. - rating based on work plan preparation, sequencing of inputs aligned with capacity, TA delivery forms used, coordination with other IMF TA and providers. - Outputs: - TA reports, inputs into and comments on draft legislation and other materials, engagement with senior policy makers and government officials, coordination with other TA providers, engagement with private sector and civil society. - rating based on use of outputs to reach consensus for change, inform laws/regulations, raise stakeholders’ awareness, build capacity at policy and technical levels, extent advice was implemented, and contribution of TA reports to project effectiveness. - Risk Management: - assessment of mechanisms at inception, correct identification and addressing of risks in project design, addressing risks identified during implementation, and mitigation capacity. - Processes: - project design relative to TA under other TTF modules; discussion of objectives with senior policy makers and outcomes with senior management; consideration of authorities’ preferences and priorities; government agency involvement and inter-agency tensions; adequacy of time frame; feasibility and clarity of verifiable indicators; involvement of other IMF TA, RTACs, and regional experts; coordination with other TA providers. ### Efficiency — definition and assessment components - Efficiency relates to resources required to produce outputs: use of least resources to achieve desired results, timeliness of work-plan execution, quality control and monitoring arrangements, and reporting mechanisms delivering timely, appropriate information. - Two categories assessed separately: - Efficiency of TA delivery itself. - Efficiency of reporting (documentation prepared for the SC on project design, progress, and modifications to scope, budget, and project dates). - Reporting assessment criteria: - Timeliness: appropriateness of timing information was provided to the SC. - Completeness: extent documents delivered information required by the SC to monitor progress and make decisions. - Assessments of risks: identification and reporting of risks in documents. - Consistency: differences among documents. - Modifications: extent modifications over time were reported and discussed. ### Impact — definition and scope - Impact refers to positive and negative changes attributable to TA (changes in systems, procedures, policies). - Assessment of impact focuses only on positive changes and considers: - impact on systems, procedures, enactment of legislation, policies, capacity building, awareness of policy makers, government officials, and civil society; - contribution to improving and exploring best practices in EI. ### Sustainability — definition and assessment basis - Results are "sustained" when new capabilities (policies, capacity, institutions) are maintained and reforms implemented beyond TA completion. - Sustainability assessment based on evaluator opinion of likelihood reforms will be sustained, considering: - extent reforms/changes integrated with country’s reform agenda; - whether local institutions/capacities were strengthened to sustain results; - evidence that structures and processes were integrated into recipients’ institutional and legal arrangements. ### Appendix II — Desk Review Countries: Selected findings and country-level statistics - Overall approach: module-level averages, ratings, and scores reported; project extensions, budget changes, and budget execution figures quoted where relevant. - ANDEAN REGIONAL: G (5.2) - Module 1 Scores shown: 5.2 5.2 5.0 5.4 5.2 5.3 5.3 5.0 - Countries preferred knowledge-exchange formats (conferences, workshops) over conventional missions. - Strong support provided on furthering legislation in Bolivia, Colombia and Peru; capacity development was deprioritized altering project scope. - Regional conferences produced economies of scale and peer learning; March 2014 conference in Lima cited. - Project extended by 15 months to June 2015. - By October 2014, budget executed by 60 percent. - DEMOCRATIC REPUBLIC OF CONGO (DRC): G (5.4) - Module 1 Scores: 5.7 6.4 6.0 5.7 5.7 5.3 5.5 5.5 - Module 2 Scores: 5.1 6.0 4.2 5.4 5.5 4.7 5.0 5.0 - Module 3 Scores: 5.4 6.1 5.1 5.0 5.2 6.0 5.2 5.0 - Module 1: Significant progress toward a new draft Mining Code reflecting TA recommendations; momentum lost due to disagreements with mining companies. Project extended by 5 months to June 2015; budget reduced by 30 percent. By October 2014, budget execution was 44 percent. - Module 2: Initial slow progress, later pickup with measures improving information availability, audit and collection; integration of tax and nontax resource revenue remained elusive. Project extended by 9 months to June 2015; budget increased by 10 percent in December 2014. Budget execution as of October 2014 was 62 percent. - Module 3: Good progress after slow start; first consolidated state budget prepared with project support. Original timeframe too short; project extensions totaling two years to December 2015. Budget execution as of October 2014 was 89 percent. - GUINEA: VG (6.0) - Module 1 Scores: 6.0 6.7 6.2 6.1 5.7 5.5 6.2 5.8 - Module 3 (Relevance only) Scores: 6.6 - Field planning visit before Project Proposal enhanced relevance and focus. - Contributed to complete review and redrafting of fiscal provisions in the Mining Code; related fiscal regulations and a Model Agreement not yet adopted. - Project momentum affected by companies not providing data and the Ebola outbreak. - Project extended by two years to April 2015; budget reduced by 20 percent in December 2014. Budget execution as of October 2014 was 50 percent. - Module 3 inception mission produced a roadmap to foster inter-ministerial coordination; project budget reduced by 13 percent in December 2014. Budget execution as of October 2014 was 21 percent. - IRAQ: G (5.4) - Module 1/3 Scores: 5.4 6.4 3.3 5.2 6.0 6.3 - Module 4 (Relevance only) Scores: 4.6 - Unique project features contributing to effectiveness included detailed risk analysis, conditionality on data submission, a practical revenue forecasting tool, and formal counterpart team appointment. - Project management actions: module put on hold after six months of no activity; placed into reserves after one year of no progress on module 1 and six months on module 3; module 1 dropped six months later. Project effectively terminated when reserve list abolished in June 2013. - Module 1 started May 2011 but stalled due to lack of cooperation from the Ministry of Oil (MOO); put on hold in May 2012. - Some PFM and macro-fiscal missions funded under the project in 2012; module 4 endorsed April 2011 but terminated June 2013. - LAO PDR: G (4.5) - Module 2 Scores: 4.5 5.3 3.8 4.8 4.3 4.3 4.5 4.8 - At inception, complex and non-transparent fiscal regime exacerbated revenue administration problems; module 1 TA would have been beneficial. - Confirmation of TA request took place a year after endorsement. - Project Proposal would have benefited from a scoping mission; outcomes were wide-ranging and overly ambitious relative to capacity and timeframe. - Creation of a Natural Resource Unit with small staff was a notable achievement. - Project extended by twenty months to December 2015. Budget execution as of October 2014 was 60 percent. - MAURITANIA: VG (5.5) - Module 1 Scores: 5.5 5.7 5.8 5.2 5.7 4.3 6.0 5.5 - First and only formally completed TA project under the TTF. - New Mining Code had been adopted prior to project start; initial resistance to revisiting it and reluctance to renegotiate mining contracts required strategy adaptation. - Project recommended reforms in the general tax system with bearing on mining taxation, contributing to expected improved revenue performance and incentives for tax formalization. - TA assisted authorities in preparing draft legislation with drafting suggestions and options; efforts to improve inter-agency collaboration between MOF and Ministry of Mines. - Budget execution of the completed project was 70 percent. - MOZAMBIQUE: G (5.0) - Module 1 Scores: 4.8 6.1 4.0 4.6 4.0 5.7 5.0 4.5 - Module 2 Scores: 5.1 6.7 3.7 5.4 4.7 6.2 4.5 4.8 - Module 3 Scores: 5.0 6.2 3.3 4.8 4.8 5.5 5.5 4.8 - Module 5 (Relevance only) Scores: 5.0 - Mozambique was a candidate country for EITI since 2009 and had been recognized as having made “meaningful progress”. ### 2012. Requests for assistance under modules 1, 3 and 5 were received from both the Ministry of ### Mid-Term External Evaluation of the MNRW TTF ### Module 1 — Fiscal regimes and institutional coordination - Requests for assistance under module 1 were received from both the Ministry of Finance (MOF) and Ministry of Mineral Resources (MIREM); both ministries were committed to coordinating closely on reforms. - The time frame for completing the outcomes for module 1 was judged too ambitious given the project starting point and existing capacity; the ambitious timing may have reflected authorities’ plans to include the fiscal regimes legislation in a wider reform package intended for the legislature in the second half of 2012. - Early strong engagement: authorities agreed with the advice and were eager to proceed. - Progress stalled because authorities had problems reaching internal consensus over key parameters of the proposed fiscal regimes; members of government supported the planned reforms but differed on detailed solutions. - Responsibility for reform implementation was scattered among different agencies and required improved coordination. - Draft fiscal laws were sent to the Council of Ministers (COM) in February 2014 and the EI legislation was passed in August 2014. - The project team was not given the opportunity to review the drafts before submission to the COM. - The new laws were built on the project’s recommendations and represent an improvement over existing legislation but do not incorporate all of the advice provided. - Capacity development needs remained, specifically assistance on developing an institutional framework for FARI model ownership. - Project timeline and budget: - Project was extended by 15 months to September 2015. - By October 2014, execution of the budget amounted to 87 percent. ### Module 2 — Natural resources tax administration (MRA) - Recommendations of the missions were well received and engagement with the Mozambique Revenue Authority (MRA) was strong. - Detailed decisions on the organizational context and staffing of the specialized unit on natural resources tax administration were delayed by one year, impacting the efficiency of technical assistance delivery. - Progress impaired by differences in view between the MRA and the project team. - A Natural Resources (NR) Unit was created at MRA headquarters, planned for later transfer to operational level when Large Taxpayers Office (LTO) management has been strengthened. - Project team preference and compromise: - The project team preferred the NR Unit to be part of the LTO to achieve synergies from administration of large NR taxpayers and large taxpayers in general. - The project team recognized the need to strengthen LTO administration and MRA headquarters functions and therefore supported implementation of a preliminary headquarters unit with a plan specifying timeframe and criteria for transition. - Staffing and recruitment: - By end-2014, eight staff had been appointed to the Unit, recruited internally within the MRA. - Twelve additional experts will be recruited externally on a contractual basis. - Further recommendations (implementation not yet started): - Develop improved procedures, tools, and effective administrative processes to monitor NR taxpayers. - Put in place procedures for formal automatic exchange of information between relevant authorities within NR administration. - Establish procedures for public information, tax collection, and taxpayer information. - Implementation planning: - A plan comprising the core activities and deadlines for key deliverables has been prepared to step up implementation efforts. - Project timeline and budget: - Project was extended by 13 months to April 2016 to accommodate four planned STX visits. - By October 2014, execution of the budget amounted to 35 percent. ### Module 3 — Macro-fiscal analysis and Macro Fiscal Unit (MFU) - Progress was made in the first six months of the project following the diagnostic mission. - Subsequent missions were postponed on several occasions as authorities needed more time to absorb recommendations. - Continuation of project implementation hinged on establishment of a Macro Fiscal Unit (MFU) at the MOF; authorities were not able to implement the MFU due to budgetary restrictions and other priorities. - Implementation was also delayed because of political uncertainty with elections taking place in October 2014. - Authorities requested support from the World Bank to set up the MFU; work started in July. ### Cross-cutting observations and implementation issues - Strong initial engagement across modules, followed by delays stemming from internal consensus issues, scattered implementation responsibilities, budgetary constraints, staffing decisions, and political timing (elections in October 2014). - Several project outputs and legal reforms incorporated elements of the project’s recommendations but did not fully reflect all technical advice. - Need for strengthened institutional frameworks (e.g., FARI model ownership, MFU) and clearer coordination mechanisms across ministries and agencies to accelerate implementation. *Mid-Term External Evaluation of the MNRW TTF.* ### 2014. The TOR to design the organization of the unit were circulated to the project team for ### mte2015i - 2014. The TOR to design the organization of the unit were circulated to the project team for ### Project revisions, scope, and timelines - The MFU is the first step to implement parts of the action plan proposed by the initial project mission. - Agreement reached with the authorities to recalibrate the overall approach and submit a new project proposal to the SC that would focus on: - supporting the creation of the macro-fiscal and fiscal risks units through capacity building actions and TA; - strengthening fiscal risk management; - improving revenue forecasts. - The project was extended by 24 months to April 2016 to provide time for the new government that took office in February 2015 to implement recommendations of the diagnostic mission. - Module activations, reserve status, and follow-up: - Module 5 was put in the reserve list in April 2011, activated in May 2012, launched in November 2012; by December 2014 no activities had taken place though the project remained in the work plan and was extended to April 2016. - STA reported a TA mission under module 5 went to Mozambique in February 2015, identified major gaps in national accounts relating to mineral exploration and extractive facilities; authorities attached high importance and are likely to request additional TA on statistics for extractive industries; follow-up missions contingent on authorities’ action plan. ### Country and module-specific findings and performance metrics - Mozambique (Module 5 / Timing note) - The February 2015 STA mission identified major gaps in national accounts regarding mineral exploration and extractive facilities. - Given mission timing, ratings and scores on module 5 do not take it into account. - Niger (overall M rating 4.0) - Module 1: - Objective: address tax policy issues in Mining and Petroleum codes; build modeling capacity for negotiations with investors. - Outcome: no progress toward project's objectives; no follow up by authorities on recommendations; requested data for financial modeling not provided. - Project terminated in June 2014. - Actual expenditure on the project was 21 percent of budget. - Module 3: - Evaluated only for Relevance; Scores: 6.2 (Relevance). Not included in country average. - Papua New Guinea (Module 4 relevance only; Score 5.5) - Module 4 aims: develop SWF legal framework, investment mandate, policy/guidelines, governance/institutional framework, and capacity building for SWF investment strategy. - Prior IMF engagement: MCM provided assistance on SWF considerations and Organic Law; FAD TA provided in 2011. - Recommendation: split objectives into two modules (legal/governance under module 3; investment strategy capacity under module 4) and ensure close FAD-MCM cooperation. - Coordination: Australian government and World Bank provided complementary policy papers and public consultation advice. - Project extended by 19 months to December 2015. - By October 2014, budget execution amounted to 38 percent. - Peru (VG rating 6.1) - Minister of Finance requested FAD TA in August 2011 on structural fiscal accounts methodologies, distribution of mineral fiscal revenues, mineral fiscal regimes and modeling (the latter addressed through the Andean Regional project, module 1). - Capacity to implement advice: adequate at policy and technical levels. - Legislative reform: government-supported Committee of Experts prepared draft revised fiscal framework enacted in October 2013 (effective in 2015) introducing: - a structural fiscal rule; - new fiscal rules for sub-national governments; - creation of a fiscal council; - obligation for explicit reporting of contingent liabilities. - Reforms reflect advice provided under the project; distributional reforms proceeded slowly due to political sensitivities. - Project extended by 18 months to December 2014. - By April 2014, budget execution amounted to 97 percent. - Budget increased by 41 percent in June 2014; by October 2014, budget execution amounted to 87 percent of the revised budget, exceeding the original budget by 22 percent. - Sierra Leone (VG 5.5) - Country context: low income, EITI candidate; coordination with World Bank and donors was prioritized. - Module 1: - By May 2012 comprehensive new legislation (Extractive Industries Revenue Act — EIRA) drafted; September 2012 new petroleum agreements aligned with proposed legislation. - Revenue forecasting framework in place by May 2012 with inter-agency MRFU established and functioning. - Enactment delays: EIRA submitted to Parliament in December 2013; completion date for outcomes extended to June 2015 due to elections and Ebola outbreak; proposals to split EIRA into separate petroleum and mining bills may further delay enactment. - Project inputs delivered effectively: 4 HQ missions and 9 STX missions over 18 months; drafting, consultation, and modeling work continued between missions; civil society, industry, and development partner consultations conducted. - Budget history: original budget increased by 70 percent to accelerate implementation; one year later budget reduced to an amount about 30 percent above original; by October 2014, 64 percent executed. - Module 2: - Implementation brisk in Q3 2012 but slowed when EIRA enactment put on hold. - Acting Commissioner General’s confirmation renewed interest in EI unit at NRA; initial NRA proposal fragmented domestic tax operations; team advised on approach; EI unit later set up in line with team recommendations (head + three managers); support on initial EI unit set up on hold due to Ebola. - Project extended to April 2016. - By October 2014, budget executed by 35 percent. - Module 3: - Time frame too ambitious; fiscal rule recommendations lingered until May 2013; draft PFM legislation finalized for Cabinet approval end-2013. - Recommendation to split PFM Bill in two: PFM Act and separate SWF Law. - Expected PFM and SWF Acts to be finalized by May 2014 and tabled by September 2014 (pre-Ebola). - Medium-term expenditure framework developed and included in annual budget law. - Project extended by 24 months to April 2016. - By October 2014, budget executed by 80 percent. - Module 5: - TA delivery efficiency affected by weak capacity and lack of resources at Statistics Sierra Leone for surveys. - By end-2014, first estimates of iron ore industry production and income accounts compiled based on available source data; work on remaining mining industries launched but not completed due to data delays. - Progress interrupted by Ebola outbreak; follow-up missions contingent on travel resumption post-Ebola. - Project extended by 12 months to April 2016. - By October 2014, budget executed by 17 percent. - Solomon Islands (VG 5.8) - Module 1 scores: Average 5.8; Relevance 6.2; E/O&O 6.0; E/I,O,R,P 5.3; E/TA 5.7; E/R 5.4; I 6.0; S 5.8. - Outcomes: - New fiscal regime for mining enacted in April 2014 largely follows TA recommendations and drafting support under the project, albeit with significant delay relative to project completion dates; implementation issues remain. - The new regime provides a strong foundation for mining development provided high export duties on minerals are eliminated and implementation details are established. - Work on capacity for revenue forecasting and fiscal terms analysis still pending. - Less support than expected was necessary for legislation preparation; STX legal expert visit in September 2011 produced draft quickly ready for Cabinet; legislation enacted in April 2014 without further TA. - Project extended by 15 months to April 2015. - By October 2014, budget executed by 20 percent. - Timor Leste (M rating 4.2) - Country context: low income, heavily dependent on petroleum; FAD helped set up the Petroleum Fund in 2005; ongoing support on revenue forecasts and ESI calculations. - Module 1 (scores: Average 4.8; Relevance 6.3; E/O&O 3.7; E/I,O,R,P 5.2; E/TA 4.7; E/R 5.4; I 3.0; S 5.3): - Divided into three components: (1) ESI calculation preparation and training; (2) analysis of fiscal regimes for future petroleum development; (3) macro-fiscal modeling improvements and training. - Only the ESI component remains active. - Support since 2011 on ESI calculations and training including intra-mission internet training; ESI software simplified for Timor staff use. - Authorities gaining skills to undertake calculations without TA; 2015 expected to be last year TTF support required on ESI. - Module 1 work continued; project extended by 32 months to December 2015. - Budget increased by almost 80 percent in the project proposal due to widening scope to strengthen macro-fiscal forecasting capacity. - In December 2013, Module 3 work terminated and budget reduced by 19 percent. - Fiscal regime analysis experienced low engagement: Petroleum Authority declined to participate in initial fiscal regime analysis; no recent acreage releases requiring TA on fiscal regime analysis. - Macro-fiscal modeling: initial mission implemented refinements and provided capacity building; however reliance on non-Timorese contract employees raises uncertainty about permanence of capacity gains; macro-fiscal component terminated in December 2013. - Module 4 (scores: Average 3.6; Relevance 4.3; E/O&O 1.7; E/I,O,R,P 3.5; E/TA 4.0; E/R 4.4; I 3.0; S 4.0): - Authorities requested TA to operationalize a more complex Petroleum Fund portfolio and provide policy advice to Petroleum Fund Directorate and Central Bank. - After initial mission in February 2013, engagement lapsed; second mission postponed until 2015 as authorities work on first mission recommendations. - Tangible result: two self-assessments on alignment with the Santiago principles by authorities and one joint assessment with the mission. - Project scope too ambitious given limited institutional capacity and controversial policy decisions required on investment directives and risk tolerance. - Project proposal dates: SC-endorsed summary May 2012 with project dates May 2012–May 2013; project actually envisioned three-year duration. - Uganda (G rating 4.6) - Country context: low income with prudent policies; oil reserves ~2.5 billion barrels; large-scale commercial production expected within 5 years. - AFRITAC East providing TA and implementation support for TSA expansion. - Module 3 (scores: Average 4.6; Relevance 4.8; E/O&O 4.8; E/I,O,R,P 4.7; E/TA 4.7; E/R 4.7; I 4.0; S 5.0): - TA effectiveness and efficiency affected by issues not properly identified at inception: conflicts with oil companies and lack of an oil development policy within Uganda, resulting in major scope modifications. - On the PFA Bill: missions worked on finalizing the Bill, participated in parliamentary workshop, prepared comprehensive amendments and re-submitted Bill to Parliament; March 2014 mission raised concerns on lack of clarity and specificity for petroleum revenue management provisions. - Progress on the Bill significantly slower than anticipated though consultation process was extensive; proposed amendments, if adopted, would significantly improve the PFA. - Project envisioned to start May 2011 for 18 months, actually started March 2013 and extended by 14 months to June 2015. - By October 2014, execution of the budget amounted to 60 percent. - Module 4: - Placed in initial work plan; project proposal apparently not prepared; project terminated in June 2014. - In June 2013 project reported inactive due to uncertainties about petroleum development; IMF proposed removal from work plan but SC kept it under consideration; ultimately terminated in June 2014. *Mid-Term External Evaluation of the MNRW TTF, DevTech Systems, Inc.* ### APPENDIX III. EVALUATION OF RESEARCH PROJECTS AND WORKSHOPS ### APPENDIX III. EVALUATION OF RESEARCH PROJECTS AND WORKSHOPS ### Overview - Since the TTF's inception through April 2014, five research projects and four workshops were endorsed, with a total initial budget of US$718,500. - Four research projects have been evaluated (the fifth project, on Progressivity in Natural Resource Taxation, was launched after April 2014). - The evaluation could not fully assess some workshops beyond their relevance. - Performance on effectiveness of outcomes and objectives showed considerable dispersion: - Ratings ranged from an E rating for a workshop to M ratings for two research projects. - Research project 1 (savings and consumption guidelines for resource producers) was rated G. - Workshop 1 (resource fiscal management in East Africa) was rated VG. - STA’s research project 3 contributed to improvement of international standards in resource-related government statistics. - Workshop 1 disseminated African country approaches to revenue management and best practices, contributing to improvement of regional standards. ### Research Projects - Research Project 1: Savings and Consumption Guidelines for Oil and Mineral Producers - Placed in the Work Plan in April 2011. - Aim: review controversy over how fast to save and how fast to spend natural resource revenues; design an "optimal" fiscal framework for low-income countries linked to PFM and absorption capacity constraints. - Activities: commissioned two papers from academics to provide expert inputs. - Status as of late 2014: - Papers remained under revision following several rounds of IMF comments. - IMF‑satisfactory papers had not been delivered. - Draft materials were used in several contexts, including inputs to an IMF policy paper and to Workshop 1. - Implementation issues: - Prolonged revision process suggests inadequate framework for interaction and supervision. - Project framework did not result in an efficient completion process. - Rating: G. - Research Project 2: Mining Tax Administration - Placed in the initial Work Plan. - Aim: identify key practices to strengthen mining tax administration and develop a toolkit of good mining tax administration practices for developing countries. - Output: - In July 2014 a handbook on Administering Fiscal Regimes for Extractive Industries was published jointly by the IMF and the World Bank. - The handbook is a comprehensive study and best practice guide focusing on resource revenue administration issues faced by developing countries. - Collaboration: project was a collaboration with the World Bank; IMF revenue administration staff and experts developed initial material produced by the World Bank and incorporated results of IMF TA (including under the TTF). - Implementation issues: - Review process was lengthy and thorough. - Project was extended by one-and-a half years in total from its initial completion date because the initial time frame was ambitious for a project of this scope. - Research Project 3: Resource Revenue Database for Government Revenue Statistics - Placed in the original Work Plan. - Aim: develop a draft template for member countries to collect and report revenues from natural resources. - Outputs and milestones: - A draft template was presented in a paper circulated to the IMF's Executive Board and subsequently posted in January 2014 for consultation with the international community. - A revised paper was made available upon request to the general public in September 2014. - The template will be field tested in six countries. - Relevance and contribution: - Addresses issues relevant to the TTF, including TA under module 5 and module 3. - Contributes to improvement of international standards in resource-related government statistics. - Implementation issues: - Paper could have provided a preliminary discussion of specific data collection issues likely to be faced by TTF eligible countries. - Project started slowly and was completed in June 2014, a little less than a year later than its planned completion date. - Subsequent developments (not incorporated in ratings due to timing): - STA informed the evaluation team that “in February 2015 the EITI International Secretariat formally introduced a Summary Data Template, which is now a regular component of EITI reports and which includes three tables to summarize the detailed data presented in national EITI reports. One of these tables is directly based on STA’s standard template.” - STA also reported field-testing of the standard template in six countries under a new project funded by the MNRW‑TTF, with the first results in Norway and Mongolia described as “very positive.” - STA reported the project is on track and expected to be completed on time and within budget. - Research Project 4: Natural Resources Asset and Liability Management Handbook - Placed in the original Work Plan in April 2011. - Aim: assist resource-rich countries to enhance wealth management and take an integrated approach to sovereign asset-liability management (SALM). - Output: - The handbook Sovereign Asset-Liability Management - Guidance for Resource-Rich Economies was published on the IMF's website in June 2014. - Relevance and limitations: - Policy advice is mostly relevant to countries with sufficiently strong analytical and asset management implementation capacity in government, central bank and financial agencies. - These preconditions are unlikely to be in place in many TTF eligible countries. - While the paper has some references to developing countries, a greater focus on approaches for low income and lower middle-income resource rich countries with underdeveloped domestic financial markets and weak capacity would have been important. - Implementation timing: - Project was activated a year and a half after being placed on the Work Plan, suggesting premature inclusion before a project concept crystallized. - Once started, an ambitious completion date was set and the handbook was completed about a year later. ### Workshops - Workshop 1: Fiscal Management of Oil and Natural Gas in East Africa - Placed in the original Work Plan in April 2011; original intent was a workshop on oil revenue management frameworks for new African producers. - Modified in May 2012 at request of the East African Community (EAC) Secretariat to focus on fiscal management issues for new oil and gas producers in the region. - Organized by FAD in collaboration with AFR; held in Arusha, Tanzania in January 2014. - Features: - Drew on recent IMF analytical work and experience from TA. - Structure allowed ample scope for presentations by government officials from the region. - Impact and reception: - Impact was favorable; participant level of satisfaction was high. - Officials from Kenya and Tanzania interviewed praised the workshop’s usefulness and relevance. - Country presentations allowed sharing and comparing experiences. - Implementation timing: - Project was postponed several times and launched two and a half years after being placed in the Work Plan with a modified geographical focus, suggesting the original project may have been premature. - Rating: VG. - Workshop 2: Cross-Border Issues in Resource Taxation - Placed in the original Work Plan in April 2011. - Aim: explore international issues in fiscal regimes for natural resources to develop practical guidance for low and lower-middle income countries with resource production or potential. - Held in Washington D.C. in May 2012. - Assessment: - Evaluation team lacked enough information to rate criteria besides relevance. - Documentation indicates some participants praised the event as a significant success but noted variability in presentation types, with some too theoretical for practical concerns of developing countries. - Some participants wanted more time for country discussion and breakout sessions for more advanced subjects. - Production of an edited conference volume remained in progress as of late 2014. - Workshop 3: Management of Natural Resource Wealth in Central Asia - Placed in the Work Plan in November 2011. - Project was not launched and was removed from the Work Plan. - Background: - Project Summary indicated demand in Central Asian countries to learn from other countries' experiences in wealth management. - Under MCM lead, project would have included other IMF functional and area departments. - Status: - In 2012 the project was reported as inactive. - In June 2013 the Report to the SC indicated the project was not activated because the Turkmen authorities had withdrawn their initial request. - Secretariat proposed removal from the work program, which was endorsed. - Workshop 4: Management of Natural Resources in Sub-Saharan Africa - Placed in the Work Plan in November 2011. - Budget covered only part of the workshop cost, with the remainder financed by the IMF. - Held in Kinshasa in March 2012. - Assessment: - Evaluation team lacked enough information to rate criteria besides relevance. - Workshop aimed to provide a forum on macroeconomic policies, fiscal policies and public investment, governance and transparency, governments getting a fair share of resource wealth, and management of resource revenue when capacity is weak. - Agenda allowed only a few presentations by SSA officials. - Allowing more presentations from policymakers, as in Arusha two years later, would have enhanced the workshop’s relevance for African participants. --- _Source: https://www.imf.org/-/media/files/capacity-developement/mnrw/mte2015i.pdf_