## pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program

## Source details

**Canonical URL:** [pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program](https://www.imf.org/-/media/files/publications/pp/pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/pp/pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program.pdf.md)
- [Structured JSON version](/-/media/files/publications/pp/pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program.pdf.json)

---

### Evaluation criteria for development projects (Box 1)
- Four of five DAC criteria presented: Relevance, Effectiveness, Efficiency, and Sustainability.
- Definitions and core evaluation questions (drawn from RTAC evaluations):
  - Relevance:
    - Definition: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.
    - Key questions: Did TA meet priority needs of member countries? Did RTACs help define country TA priorities? Did the SC ensure strong country ownership? Are RTACs contributing to the Paris Declaration? Were activities well integrated with Fund HQ TA, surveillance, and lending?
  - Effectiveness:
    - Definition: A measure of the extent to which an aid activity attains its objectives.
    - Key questions: Were activities and outputs of good quality and timely? Did the RTACs contribute to developing institutions? Were activities well coordinated and leveraged with other donors and TA providers?
  - Efficiency:
    - Definition: Measures the outputs—qualitative and quantitative—in relation to the inputs.
    - Key questions: Were RTACs cost-effective? What was the quality and timeliness of backstopping by IMF HQ staff?
  - Sustainability:
    - Definition: Measures whether the benefits of an activity are likely to continue after the donor funding has been withdrawn.
    - Key questions: Did TA lead to tangible and lasting results? Did RTACs succeed in identifying and using local and regional expertise?

### Expansion, scope, and medium-term planning of RTACs
- Drivers and responses:
  - The Fund is expanding and extending its RTAC network because of the high demand for RTACs.
  - AFRITACs and PFTAC evaluations recommended expanding centers; AFRITACs also called for more resources for backstopping from HQ.
  - CARTAC evaluation recommended extending the center for a next phase.
  - Program documents lay out a 5-year plan with detailed deliverables for centers in each recipient country and TA area.
- Medium-term planning actions:
  - PFTAC: develop a strategy to intensify and extend regional approaches to build capacity.
  - CARTAC: formalize and strengthen country strategy meetings.
  - AFRITACs: adopt a three-year plan for each TA area in each country.
- Regional cooperation and SC strengthening:
  - AFRITAC evaluation: more could be done to foster regional integration and harmonization.
  - CARTAC: coordinates with CARICOM initiatives toward a single market and economy; coordinates with regional councils during Article IV consultations and annual regional conferences.
  - CAPTAC-DR: established specifically to support Central America’s regional economic integration efforts.
  - METAC and CARTAC evaluations: encourage SC networking, formalize minute taking and review, consider alternate SC members; SC meetings scheduled in spring 2010 to review suggestions.

### Effectiveness of RTAC TA — evaluation findings
- General findings:
  - RTAC TA judged generally of high quality.
  - Strengths: hands-on mentoring, experts’ active technical role, high credibility, valued regional workshops, appreciated Resident Advisor missions.
  - Challenges: unfinished agendas in many areas; difficulty distinguishing attribution vs contribution; need for coordination with other TA providers; slow implementation of advice.
- Management recommendation:
  - All evaluations recommended improving the results-focused management (RFM) framework to shift from mission-based inputs to managing for outcomes and results.

### Results-focused Monitoring and Evaluation (RFM)
- Purpose and key features:
  - Focus on well-formulated, longer-term outcomes and objectives measured through verifiable indicators.
  - Improve coordination with other TA providers and better integrate IMF TA into national strategies.
  - Facilitate the role of SCs.
  - Distinguish inputs and outputs (under RTAC control) from outcomes and objectives (under authorities’ control).
  - Alignment with HQ TA and embedding, where appropriate, in broader initiatives (such as PEFA).
- Pilots and operationalization:
  - Pilots underway with selected RTACs involving SC and core donors.
  - Fund is moving toward an RFM but has found it difficult to make operational for RTACs.
- Illustrative example (principles preserved):
  - Traditional M&E: monitors inputs and outputs.
  - Results-focused M&E: monitors implementation and assesses outcomes/objectives using measurable and verifiable indicators.
  - Example: inputs (TA missions exploring a large tax payer office); output (TA report designing the LTO); outcome (well-functioning LTO); indicators (establishment of the LTO or increase in revenue from large taxpayers).

### Donor coordination, dissemination, and transparency
- Evaluation highlights and Fund responses:
  - Need for strengthened donor coordination and improved websites for donor and regional coordination.
  - RTACs implementing the Fund’s new dissemination policy to allow sharing TA information, including reports, with SC members who sign confidentiality agreements and where TA recipient consent is given on a no-objection basis.
  - Continued donor briefings via resident representative offices and missions; RTACs without resident representation planning websites and annual reports.
  - HRD, with OTM and TA departments, to host and maintain recruitment-related external web content to increase transparency of expert recruitment procedures.

### Human resources policies and staffing
- Findings and actions:
  - METAC evaluation: importance of recruiting advisors with strong knowledge of Fund procedures; Fund staff leave-of-absence limits available experts.
  - Preparations to allow Fund staff appointment on a mobility-type arrangement to RTACs.
  - Recruitment transparency: continue selecting RTAC staff with enhanced transparency (regular information on job openings and appointments to the SC) and due consideration for suitable regional candidates.

### Efficiency, costing, financial reporting, and administration
- Evaluation findings:
  - RTACs are lean organizations, but measurement of cost indicators needs improvement.
  - Fund financial reporting did not allow accurate calculation of total costs including project management and backstopping.
  - CARTAC and METAC assessed efficiency based on stakeholder feedback due to lack of time-recording systems for HQ staff time.
  - High degree of satisfaction reported with quality of backstopping from Fund HQ; dedicated resources vary by TA department.
- Reforms and timelines:
  - Recommendation for a manual codifying organization, management, and administrative procedures for RTACs.
  - A new TA costing model identifies all actual costs related to a TA intervention (including cost of HQ staff time spent on TA delivery, project management and backstopping).
  - Existing RTACs moved to the Fund’s new external financing instrument; new RTACs start on this instrument.
  - Administration and start dates under new instrument:
    - CAPTAC-DR and AFRITACs East and West: already administered under the new instrument.
    - METAC: will follow on May 1, 2010.
    - AFRITAC Central: on November 1, 2010.
    - CARTAC: on January 1, 2011.
    - PFTAC: on May 1, 2011.
  - CARTAC conversion: set up in 2001 as a UNDP cost-shared project with the Fund as executing agency; conversion to be administered as other RTACs starting January 1, 2011.
  - New financing reporting procedures and administrative/management procedures are being incorporated in an “RTAC Handbook.”

### Sustainability of TA outcomes
- Evaluation findings:
  - RTACs produced tangible and lasting results, though attribution sometimes unclear.
  - Sustainability undermined by organizational weaknesses in executing agencies (particularly in post-conflict countries), agency staff turnover, lack of follow up, and insufficient resources for implementation.
  - METAC: stronger RFM would facilitate follow up.
  - CARTAC: medium-term need to continue TA within the region.
  - PFTAC: noted limited use of Pacific experts; RTACs generally helped build regional expertise and networks.
- Measures to enhance sustainability:
  - Strengthen country ownership through longer-term planning to support TA implementation.
  - Use RFM to assess absorptive capacity up front and monitor verifiable indicators to permit SC assessment of authorities’ follow up.
  - Wider dissemination of TA information to generate synergies and enhance sustainability.
  - Resident Advisors to indicate to donors whether funds are needed to implement recommendations (e.g., IT, software/hardware, surveys).
  - Moving to actual costs provides incentives to build regional expertise.
  - Recruitment transparency and consideration of regional candidates; periodic information to SC on job openings and appointments.

### Other TA evaluations (non-RTAC) — scope, findings, and common conclusions
- Coverage:
  - Nine additional TA evaluations covering Switzerland and Japan bilateral subaccounts, IMF backstopping and project management, CARTAC VAT implementation, FAD tax reforms in Mozambique, MCM bank supervision LTEs in Asia, STA post-conflict and China statistics work.
- Financial scale and time horizons:
  - Financial scale ranged from about US$1.5 million (STA post-conflict) to more than US$200 million (Japan subaccount).
  - Time horizons varied from 10–20 years for some projects to shorter periods for others.
- Common conclusions:
  - All non-RTAC evaluations gave high marks to Fund TA: good quality, generally well received, and well aligned with donor objectives.
  - Sustainability received the lowest marks among criteria.
  - Evaluators concluded IMF TA was good value for money and recommended strengthening financial reporting for more thorough future assessments.

### Evaluation recommendations (summary from multiple evaluations)
- Systematic backstopping is core to IMF TA quality control (Japan subaccount, IMF backstopping study, MCM on long-term experts).
- Recommend coordinated long-term strategies to identify priority TA (STA—People’s Republic of China).
- Improve results focus of project design and implementation (Japan subaccount, CARTAC VAT).
- Promote more regional approaches to build local expertise; regional fora and study tours can assist (CARTAC VAT; STA—People’s Republic of China).
- Improve dissemination of information (Japan subaccount).
- Sustainability depends on ownership and political commitment; should be addressed in project design and execution (Switzerland and Japan subaccounts; FAD; STA).
- Projects should be flexible to accommodate changing government priorities; TORs should not be unduly restrictive (Switzerland subaccount; MCM).
- Training is important to mitigate rapid staff turnover (Switzerland subaccount; CARTAC VAT; STA).
- Long-term experts (LTEs) are particularly suitable in early stages of reform and in behind-the-scenes activities; interpersonal skills and adaptability are critical (MCM).
- Recruitment procedures should be more transparent (Japan subaccount).
- Enhance donor visibility (Japan subaccount).

### Lessons learnt and programmatic responses
- Consensus: IMF TA is responsive, flexible, and high quality; room for improvement remains.
- Recent reforms underway:
  - Shift to the new TA costing model expected to increase efficiency of Fund TA.
  - Strengthening RFM framework for RTACs and in bilateral programs.
  - For TTFs, a clear RFM is key to programmatic design.
- Enhancing sustainability:
  - Move toward longer-term and programmatic engagements (including multi-year programs under bilateral subaccounts).
  - 5 year funding cycles for RTACs and TTFs.
- Strengthening partnerships and governance:
  - Bilateral donor relations to include periodic consultations.
  - Governance structure of multi-donor products (RTACs and TTFs) aims for close stakeholder involvement.

### TA Evaluation Program FY11-13 — scope and selected items
- EP for FY11-13 includes 18 evaluations.
- Projection:
  - Externally financed TA field delivery will increase to about 80 percent of total TA, and almost all of that is subject to periodic evaluations.
- Selected evaluations supported by OTM (with expected completion FY):
  - Belgium subaccount — 11
  - Norway subaccount—Capacity Building in Reserve Bank of Malawi — 12
  - CAPTAC-DR — 13
  - AFRITACs East and West — 13
  - TTF on AML/CFT — 13
  - JSA subaccount — Annual
- Departmental planned evaluations (selected):
  - FAD: several items with Expected Completion FY 11 and 12.
  - MCM: items with Expected Completion FY 11 and 12.
  - STA: statistical capacity building evaluations in Selected Countries (Expected Completion FY 11/12).

### Appendix II — RTAC-specific recommendations and implementing actions (selected)
- AFRITACs:
  - Recommendation 2: adopt a three year plan for each cluster of TA interventions in a country by end of calendar year 2010.
    - Implementing action: program documents propose medium-term plans, integrated with country reform programs, with verifiable indicators; annual reports to document progress.
  - Recommendation 6: OTM to prepare a manual codifying organization, management and administrative procedures for RTACs by end of FY2010.
    - Implementing action: work started; OTM expected to prepare manual in course of 2010; relevant parts to be shared with SC.
  - Recommendation 7: OTM to prepare a Ten Year Vision for RTACs as part of next RTAC Review.
    - Implementing action: OTM intends to prepare RTAC Review for discussion at IMF Executive Board in 2010.
- PFTAC:
  - Recommendation 1: develop a strategic plan setting out a five-year vision by end calendar year 2010.
    - Implementing action: program document for new cycle starting May 1, 2011; draft for discussion summer 2010.
  - Recommendation 2: reach consensus on priorities for additional Resident Advisors by end of FY2010.
    - Implementing action: TPRC in June 2009 decided to post an additional macroeconomic advisor starting 2010.
  - Recommendation 7: define medium term objectives and verifiable indicators by end of FY2011.
    - Implementing action: forthcoming program document will propose medium-term objectives and verifiable indicators.
- CARTAC:
  - Recommendation 1: re-consider reporting requirements to reduce compliance burden.
    - Implementing action: CARTAC under UNDP arrangement; program document for next funding cycle (starting January 1, 2011) will propose how to address.
  - Recommendation 8: extension into Phase IV (2011- 2013).
    - Implementing action: Fund will propose extension of CARTAC for 5 years into Phase IV; program document will lay out Phase IV objectives.
  - Recommendation 10: adopt a basic performance management reporting structure aligned to goals and programme objectives.
    - Implementing action: work in progress to create basic performance management reporting for all RTACs.
- METAC:
  - Recommendation 1: Phase III proposal to include clear objectives with defined performance indicators and assess role of TAIMS in monitoring and evaluation.
    - Implementing action: Phase III program document sets out verifiable indicators; TAIMS to play a more important role in monitoring and evaluating performance.
  - Recommendation 3: financial reporting to cover total cost of METAC’s TA delivery using the new SFA reporting system.
    - Implementing action: costing changed from standard to actual costs; financial reporting more detailed under new financing instrument.
  - Recommendation 4: upgrade METAC website and regional networking; involve GCC and North Africa to enhance knowledge transfer.
    - Implementing action: website upgraded; regional networking to more actively involve countries from GCC and North Africa.

### Appendix III — Ratings and recommendations of other TA evaluations (selected)
- Switzerland Subaccount (Support for IMF Capacity Building; July 2009)
  - Coverage: 1998-2009; Disbursements US$8.4 million.
  - Summary Evaluation (scale 1–4): Overall: Effectiveness 3.3; Efficiency 3.3; Relevance 3.1; Sustainability 2.5; Overall 3.1.
  - Recommendations: continue bilateral arrangements; lengthen and phase projects taking longer than envisaged; use TAIMS formats; provide staff training.
- Japan—Administered Account (JSA) (December 2009)
  - Coverage: 1990-2008; Disbursements US$279 million.
  - Summary Evaluation: Effectiveness: Good; Efficiency: Good; Relevance: Highly relevant; Sustainability: Achieved as far as under control of TA projects; Overall: TA delivered good value for money.
  - Recommendations: enhance transparency (publish project-level evaluation questionnaires; make recruitment more transparent); improve management and results-focus; improve visibility of JSA funding; strengthen recruitment of Japanese experts.
- CARTAC VAT assessment (July 2009)
  - Coverage: 2005-2008; Disbursements US$5.7 million.
  - Summary: Effectiveness: Yes; Relevance: Yes, with strong ownership; Sustainability: Yes, but risks must be addressed.
  - Recommendations: continue VAT implementation design; provide hands-on TA and training in complex auditing and legislative review; arrange regional forum for VAT coordinators; apply more results-oriented TA design.
- Selected other evaluations (high-level findings):
  - Mozambique tax reforms (FAD): IMF TA spurred and supported reform; implementation continues under Strategic Plan for 2008-2012.
  - Post-conflict Mozambique and Rwanda (STA): Rebuild institutions before TA can be absorbed; national statistical offices hampered by resource constraints.
  - Statistics in the People’s Republic of China (STA): TA highly appreciated; recommend shift to technical cooperation and consultation, expand modalities, develop coordinated long-term statistical strategy.
  - Bank supervision LTEs in Asia (MCM): LTEs successful; recommend LTEs in early reform stages; improve backstopping and HQ knowledge management.
  - Backstopping and project management study: backstopping is core IMF staff work and a comparative advantage.

### Selected planned evaluations and thematic studies (FY timing and focus)
- Development of Inflation Targeting and Monetary Modeling in Emerging Market Europe (MCM)
  - Fund provided TA to Croatia, Moldova, Romania, Russia, Serbia, Turkey, and Ukraine.
  - Proposed evaluation in FY2010 to cover cross-section of these countries.
  - Issues: links to central bank independence and financial market development; exchange rate regime implications; importance of pre-existing local capacity; effectiveness of delivery modes (resident experts vs periodic visits).
- Evaluation of risk-based supervision TA in Costa Rica and Nicaragua (MCM)
  - Program: 24-month work program ending April 2011; managed by external expert and backstopped by MCM HQ staff.
  - Proposed evaluation focus: progress in implementation of risk-based supervision in FY12; assess extent of use, adoption of recommendations, adequacy of manuals/procedures, training and coaching; Fund to offer TA to address shortcomings.
- Statistical capacity building in selected countries (STA)
  - Selected countries: Nigeria (AFR), Albania (EUR), Georgia (MCD), and Peru (WHD).
  - Evaluations to be carried out in FY11–12.
  - Methodology: mainly Fund staff assessments supplemented by short field missions; emphasis on outcomes, features of TA delivery conducive to capacity building, role of authorities, resource constraints, commitment/ownership.
  - Outputs: an evaluation report for each selected country.

*Source: https://www.imf.org/-/media/files/publications/pp/pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program.pdf*

### Box            1                                    Evaluation            Criteria            for Development Projects..

### Box 1                                    Evaluation Criteria for Development Projects

### Overview
- The four criteria presented are among the five endorsed by the OECD’s Development Assistance Committee (DAC).
- The five DAC criteria are the standard in this area; the box presents four: Relevance, Effectiveness, Efficiency, and Sustainability.
- Definitions and evaluation questions are drawn from RTAC evaluations.

### Relevance
- Definition: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.
- Questions:
  - Did TA meet priority needs of member countries?
  - Did RTACs help define country TA priorities?
  - Did the SC ensure strong country ownership?
  - Are RTACs contributing to the Paris Declaration?
  - Were activities well integrated with Fund HQ TA, surveillance, and lending?

### Effectiveness
- Definition: A measure of the extent to which an aid activity attains its objectives.
- Questions:
  - Were activities and outputs of good quality and timely?
  - Did the RTACs contribute to developing institutions?
  - Were activities well coordinated and leveraged with other donors and TA providers?

### Efficiency
- Definition: Measures the outputs—qualitative and quantitative—in relation to the inputs.
- Questions:
  - Were RTACs cost-effective?
  - What was the quality and timeliness of backstopping by IMF HQ staff?

### Sustainability
- Definition: Measures whether the benefits of an activity are likely to continue after the donor funding has been withdrawn.
- Questions:
  - Did TA lead to tangible and lasting results?
  - Did RTACs succeed in identifying and using local and regional expertise?

*Source: Box 1, "Evaluation Criteria for Development Projects," pp. 7–8.*

### 10.      The Fund is expanding and extending its RTAC network because of the high

### pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program - 10.      The Fund is expanding and extending its RTAC network because of the high

### Expansion and scope of RTACs
- The Fund is expanding and extending its RTAC network because of the high demand for RTACs.
- Evaluation recommendations:
  - AFRITACs and PFTAC evaluations recommended expanding the centers; AFRITACs also highlighted the need for more resources for backstopping from HQ.
  - CARTAC evaluation recommended extending the center for a next phase.
- In addition to extension, the AFRITACs and PFTAC evaluations propose expansion of existing centers.
- Program documents respond by laying out a 5-year plan with detailed deliverables of the centers in each of the recipient countries and each TA area.

### Medium-term planning and regional cooperation
- RTACs are strengthening medium-term planning:
  - PFTAC evaluation: develop a strategy to intensify and extend regional approaches to build capacity.
  - CARTAC evaluation: formalize and strengthen country strategy meetings.
  - AFRITACs evaluation: adopt a three-year plan for each TA area in each country.
- Regional cooperation:
  - AFRITAC evaluation: more could be done to foster regional integration and harmonization.
  - CARTAC: works with CARICOM initiatives toward a single market and economy, and coordinates with regional councils during Article IV consultations and annual regional conferences.
  - CAPTAC-DR: established specifically to support and leverage Central America’s regional economic integration efforts.
- SCs (Steering Committees) strengthening:
  - METAC and CARTAC evaluations proposed encouraging networking across SC members, formalizing minute taking and review, and considering appointment of alternate SC members.
  - SC meetings scheduled in spring 2010 to review these suggestions.

### Effectiveness of RTAC TA — evaluation findings
- General findings:
  - RTAC TA judged generally of high quality.
  - Hands-on mentoring, experts’ active technical role, and high credibility are strengths.
  - Regional workshops valued for peer learning; Resident Advisor missions particularly appreciated.
  - Challenges noted: unfinished agenda in many areas, difficulty distinguishing attribution vs contribution, need for coordination with other TA providers, and slow implementation of advice.
- All evaluations recommended improving the results-focused management (RFM) framework to increase RTAC effectiveness by shifting from mission-based inputs to managing for outcomes and results.

### Results-focused Monitoring and Evaluation (RFM) — design and implementation
- Purpose and features:
  - RFM aims to focus on well-formulated, longer-term outcomes and objectives measured through verifiable indicators, improve coordination with other TA providers, better integrate IMF TA into national strategies, and facilitate the role of SCs.
  - The Fund is developing an RFM framework that is simple, tailored, and meaningful for RTAC stakeholders.
  - The framework distinguishes inputs and outputs (under RTAC control) from outcomes and objectives (under authorities’ control).
  - Alignment with HQ TA and embedding, where appropriate, in broader initiatives (such as PEFA).
- Pilots and operationalization:
  - Pilots underway with selected RTACs involving SC and core donors.
  - The Fund is already moving toward an RFM in response to previous evaluations, but it has been difficult to make the framework operational for RTACs.
- Box 2 summary (principles preserved):
  - Traditional M&E: monitors inputs and outputs.
  - Results-focused M&E: combines monitoring of implementation with assessment of outcomes and objectives using measurable and verifiable performance indicators.
  - Example provided: inputs (TA missions exploring a large tax payer office), output (TA report designing the LTO), outcome (well-functioning LTO) and indicators (establishment of the LTO or increase in revenue from large taxpayers).

### Donor coordination, dissemination, and transparency
- Evaluations highlighted need for strengthened donor coordination and improved websites for donor and regional coordination.
- RTACs implementing the Fund’s new dissemination policy to allow sharing TA information, including reports, with SC members who have signed confidentiality agreements, provided the TA recipient has given consent (to be sought on a no-objection basis).
- RTACs will continue donor briefings (including via resident representative offices and missions) and RTACs without them are planning websites and annual reports.
- The Human Resources Department (HRD), in collaboration with OTM and TA departments, will host and maintain recruitment-related external web content to increase transparency of expert recruitment procedures.

### HR policies and staffing
- METAC evaluation noted importance of recruiting advisors with strong knowledge of Fund procedures and that Fund staff needing to take leave of absence limits available experts.
- Preparations under way to allow Fund staff to be appointed on a mobility-type arrangement to RTACs.
- Recruitment: Fund to continue selecting RTAC staff with enhanced transparency (regular information on job openings and appointments to the SC) and due consideration for suitable regional candidates.

### Efficiency, costing, and financial reporting
- Evaluation findings:
  - RTACs are lean organizations, but measurement of cost indicators needs improvement.
  - Fund financial reporting did not allow accurate calculation of total costs including project management and backstopping.
  - CARTAC and METAC assessed efficiency based on stakeholder feedback due to lack of time-recording systems for HQ staff time.
  - High degree of satisfaction reported with quality of backstopping from Fund HQ, with dedicated resources varying by TA department.
- Responses and reforms:
  - Recommendation for a manual codifying organization, management, and administrative procedures for RTACs.
  - A new TA costing model identifies all actual costs related to a TA intervention (including cost of HQ staff time spent on TA delivery, project management and backstopping), supporting more efficient allocation of resources and accountability.
  - Existing RTACs moved to the Fund’s new external financing instrument; new RTACs start on this instrument.
  - Administration and start dates under new instrument:
    - CAPTAC-DR and AFRITACs East and West: already administered under the new instrument.
    - METAC: will follow on May 1, 2010.
    - AFRITAC Central: on November 1, 2010.
    - CARTAC: on January 1, 2011.
    - PFTAC: on May 1, 2011.
  - New financing reporting procedures and administrative/management procedures are being incorporated in an “RTAC Handbook.”
- CARTAC administration:
  - CARTAC was set up in 2001 as a UNDP cost-shared project with the Fund as executing agency; CARTAC conversion to be administered as other RTACs starting January 1, 2011.

### Sustainability of TA outcomes
- Evaluation findings:
  - RTACs produced tangible and lasting results, though attribution sometimes unclear.
  - Sustainability undermined by organizational weaknesses in executing agencies (particularly in post-conflict countries), agency staff turnover, lack of follow up, and insufficient resources for implementation.
  - METAC: stronger RFM framework would facilitate follow up.
  - CARTAC: medium-term need to continue TA within the region.
  - PFTAC: noted limited use of Pacific experts; RTACs generally helped build regional expertise and networks.
- Measures to enhance sustainability:
  - Stronger country ownership through longer-term planning to strengthen TA implementation.
  - Use of RFM to better assess absorptive capacity up front and monitor verifiable indicators during execution to permit SC assessment of authorities’ follow up.
  - Wider dissemination of TA information to generate synergies and enhance sustainability.
  - Resident Advisors to indicate to donors whether funds are needed to implement recommendations (e.g., IT, software/hardware, surveys).
  - Moving to actual costs provides incentives to build regional expertise.
  - Recruitment transparency and consideration of regional candidates; periodic information to SC on job openings and appointments.

### Other TA evaluations (non-RTAC)
- Nine additional TA evaluations were undertaken covering:
  - TA to Central Asia under the Switzerland bilateral subaccount (external evaluation at donors' request).
  - All projects under the Japan bilateral subaccount (external evaluation at donors' request).
  - IMF’s approach to backstopping and project management (external evaluation at donors' request).
  - CARTAC evaluation on implementation of value added taxes (VAT).
  - FAD: reforms in tax policy and administration in Mozambique and macro policy lessons for fiscal decentralization.
  - MCM: TA on bank supervision by long-term experts in Asia.
  - STA: TA to Post-Conflict Countries and TA and training on statistics in the People’s Republic of China.
- Variation in focus, duration, and scale:
  - Some evaluations covered regional projects, some specific countries, and some modalities of TA provision.
  - Time horizons varied from 10–20 years for some projects to shorter periods for others.
  - Financial scale ranged from about US$1.5 million (STA post-conflict) to more than US$200 million (Japan subaccount).
- Common conclusions:
  - Echoing RTAC evaluations, all non-RTAC evaluations gave high marks to Fund TA.
  - IMF TA judged good quality, generally well received by country authorities, and well aligned with donor objectives.
  - Sustainability received the lowest marks among criteria.
  - Evaluators concluded IMF TA was good value for money and recommended strengthening financial reporting for more thorough future assessments.

*Source: https://www.imf.org/-/media/files/publications/pp/pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program.pdf*

### 30.      The evaluations made recommendations similar to the ones for the RTACs:

### pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program - 30.      The evaluations made recommendations similar to the ones for the RTACs:

### Evaluation recommendations (summary)
- Systematic backstopping is what sets IMF TA delivery apart from other providers, and is a key for quality control (Japan subaccount, IMF backstopping study and MCM on long-term experts).
- The STA evaluation on TA to the People’s Republic of China recommended developing a coordinated long-term strategy for all agencies to identify priority TA.
- The Japan subaccount and the CARTAC VAT evaluations highlighted the need to further improve the results focus of project design and implementation.
- Several evaluations (Japan subaccount, CARTAC VAT and the STA on the People’s Republic of China) recommended more regional approaches to build local expertise.
  - Regional fora for country officials (CARTAC VAT) or study tours (STA on the People’s Republic of China) can assist in building local expertise and develop a peer review network.
  - The use of regional experts helps with regional cooperation (CARTAC VAT).
- The evaluation of the Japan subaccount recommends improving the dissemination of information.

### Additional general observations from evaluations
- Sustainability of TA depends on ownership by authorities and political commitment (Switzerland and Japan subaccounts, FAD on fiscal decentralization, and STA on post-conflict and on the People’s Republic of China).
  - Sustainability issues must be covered in project design and execution (Japan subaccount).
  - Sustainability could be enhanced through various actions by implementing agencies (STA on the People’s Republic of China) and by providing adequate resources for implementation (FAD on Mozambique, STA on post conflict).
- Projects should be flexible enough to accommodate changing government priorities (Switzerland subaccount) and TORs should not be unduly restrictive (MCM on long-term experts).
  - Involvement of authorities, including middle management, in drafting TOR will better reflect the situation on the ground.
  - The flexibility of experts is as important as their expertise (MCM on long-term experts).
- Training is an important part of TA provision (Switzerland subaccount, CARTAC VAT and STA on the People’s Republic of China). More should be provided to mitigate the effect of rapid staff turnover.
- Long-term experts appear more suitable than other forms of TA in the early stages of a reform process (MCM on long-term experts).
  - They are most effective in less visible, more “behind the scenes” activities, and their interpersonal skills and willingness to adjust to local conditions are critical to their success.
- While the current recruitment system for experts is considered efficient, the procedures for selecting from the IMF roster of experts should be made more transparent (Japan subaccount).
- Donor visibility should be enhanced through greater use of donor logos and further informing recipients that TA has been financed by a particular donor (Japan subaccount).
  - Regional seminars could offer opportunities to give visibility to a donor and involve its nationals (Japan subaccount).

### Lessons learnt
- There is consensus that IMF TA is responsive to country needs, flexible and of high quality, as evidenced in high marks from recipient countries and donors alike.
- Evaluations noted room for improvements; many recent TA reforms (such as on TA costing and dissemination of information) have begun to bear fruit, but further work is needed to make Fund TA more results-focused and sustainable.
- All TA is benefitting from the Fund’s response to the recommendations.
  - The shift to the new TA costing model is expected to increase the efficiency of all Fund TA.
  - The Fund is strengthening its RFM framework for the RTACs and in bilateral programs.
  - For the TTFs—a main multi-donor product line—a clear RFM is key to their programmatic design.
- TA sustainability is being enhanced by:
  - moving toward longer-term and programmatic engagements (including through multi-year programs under bilateral subaccounts),
  - the 5 year funding cycles for the RTACs and TTFs.
- Sustainability and traction of Fund TA as well as information sharing and coordination will also benefit from strengthened partnerships.
  - Bilateral donor relations will include periodic consultations.
  - The governance structure of the multi-donor products—RTACs and TTFs—aims for close involvement of stakeholders.

### TA Evaluation Program FY11-13 (overview)
- The EP for FY11-13 includes 18 evaluations.
- Over the longer term, for most Fund TA evaluations will become standard practice.
- Externally financed TA field delivery will increase to about 80 percent of total TA, and almost all of that is subject to periodic evaluations.
- Evaluations of externally financed TA supported by OTM during FY11-13:
  - Several RTAC evaluations as they reach the halfway point of their new funding cycles: CAPTAC-DR, and AFRITACs East and West.
  - First evaluation of the TTF launched in 2009 on Anti-Money Laundering/Combating the Financing of Terrorism (AML/CFT).
  - Evaluations of several bilateral subaccounts (Belgium, Japan and Norway), all conducted by independent evaluation teams.
- Evaluations conducted by TA departments:
  - FAD envisages three evaluations to be completed by August 2010 (all delayed from the FY09-11 EP) and one new evaluation (on treasury single accounts).
  - MCM envisages one evaluation delayed from the FY09-11 EP and two recast evaluations.
  - STA envisages four evaluations of country programs in four regions.

- Table 3 (selected items from TA Evaluation Program FY11-13; Department — Topic/Program — Expected Completion (FY)):
  - OTM — Belgium subaccount — 11
  - OTM — Norway subaccount—Capacity Building in Reserve Bank of Malawi — 12
  - OTM — CAPTAC-DR — 13
  - OTM — AFRITACs East and West — 13
  - OTM — TTF on Anti-Money Laundering and Combating the Financing of Terrorism — 13
  - OTM — JSA subaccount — Annual
  - FAD — Experience with the implementation of the “upstream” approach for delivery of TA in revenue administration — 11
  - FAD — Tax policy TA on revenue aspects of trade reform — 11
  - FAD — The role and modalities of public financial management TA in an environment where other TA providers are intensely involved: the Cambodia and Mozambique examples — 11
  - FAD — Treasury Single Account — 12
  - MCM — Program of Technical Assistance Cooperation to Nigeria — 11
  - MCM — Development of inflation targeting and monetary modeling in emerging market Europe — 11/12
  - MCM — Evaluation of risk-based supervision TA in Costa Rica and Nicaragua — 12
  - STA — Statistical Capacity Building in Selected Countries with Significant TA Delivery--Nigeria — 11/12
  - STA — Statistical Capacity Building in Selected Countries with Significant TA Delivery --Albania — 11/12
  - STA — Statistical Capacity Building in Selected Countries with Significant TA Delivery --Georgia — 11/12
  - STA — Statistical Capacity Building in Selected Countries with Significant TA Delivery --Peru — 11/12

*Source: https://www.imf.org/-/media/files/publications/pp/pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program.pdf*

### Appendix II

### Appendix II

### AFRITAC Recommendations and Implementing Actions
- Recommendation 1: OTM presentations during the negotiations for the next financial replenishment should argue for additional resources to strengthen the human and financial resources of the AFRITACs and necessary support from HQ together with implementing the recommendations of the evaluation.
  - Implementing action: The program documents for all AFRITACs propose a budget incorporating a request for more resources (for existing AFRITACs) or are based on a larger role of the center (new AFRITACs). In addition to the expansion of the AFRITACs, the IMF is planning to step up TA to Sub-Saharan Africa through topical trust funds.
- Recommendation 2: The three AFRITACs should, in coordination with the TA Departments, by the end of calendar year 2010 adopt a three year plan for each cluster of TA interventions in a country that sets out the strategic objectives and outcomes that the capacity building initiative expects to achieve and provides a framework with indicators against which progress can be monitored.
  - Implementing action: The program documents for all AFRITACs propose medium-term plans for the covered countries for each cluster of TA interventions. These plans are integrated with the relevant country reform program, which also set out the strategic objectives of each AFRITAC’s initiatives. The proposals include verifiable indicators against which progress can be monitored. The annual reports of each AFRITAC will document the progress made toward these indicators.
- Recommendation 3: While recognizing that beneficiary countries should lead donor coordination, all three AFRITACs need to strengthen their donor coordination and information dissemination strategies.
  - Implementing action: Coordination with donors will be strengthened in a variety of ways, including by sharing information after AFRITAC mission and enhanced communication through new websites for each AFRITAC. In April 2009, the IMF adopted streamlined and strengthened procedures for the wider dissemination of TA-related information, including its mission planning, regional TA strategies and specific TA reports.
- Recommendation 4: All AFRITACs, the SCs as well as the Center Coordinators, Resident Advisors, the TA Departments and OTM must do some strategic thinking about how to strengthen TA sustainability; provide more follow up and financing to support implementation of TA recommendations.
  - Implementing action: Strengthened coordination with donors and other TA providers. Resident Advisors through periodic missions would indicate needs for funds to implement recommendations (e.g., information technology, software and hardware, support for surveys). Needs to be highlighted at SC meetings. Follow-up TA needs disseminated to downstream TA providers through regular donor updating by resident experts.
- Recommendation 5: Each AFRITAC should prepare a plan to indicate how it will use a regional approach to facilitate development of macroeconomic tools necessary to support regional integration and harmonization and present the plan to their SCs in 2010.
  - Implementing action: Program documents for all AFRITACs set out how they would support regional integration and harmonization. Each AFRITAC will report progress at the SC to facilitate coordination with other TA providers.
- Recommendation 6: By the end of Financial Year (FY2010) OTM should prepare a manual that codifies the organization, management and administrative procedures for the RTACs.
  - Implementing action: Work started. OTM hosted a 2-day internal RTAC retreat with Center Coordinators and IMF HQ staff in December 2008, identified action items and timetable. OTM expected to prepare this manual in the course of 2010. While most of the manual would be on internal procedures, relevant parts would be shared with the SC.
- Recommendation 7: As part of the next RTAC Review, OTM should prepare a Ten Year Vision for RTACs that outlines the strategic implications for IMF.
  - Implementing action: OTM intends to prepare a RTAC Review for discussion at the IMF Executive Board in 2010, which will lay out the IMF’s long-term vision on the RTACs.
- Recommendation 8: By the end of 2009 the AFRITACs and OTM should develop formal action plans, identifying the necessary resources and monitoreable benchmarks to implement the accepted evaluation recommendations and report the implementation status to the SCs in FY2010 and FY2011.
  - Implementing action: Progress on implementation of the recommendations will be reported at the SCs in FY2010 and FY2011.

### PFTAC Recommendations and Implementing Actions
- Recommendation 1: By the end calendar year 2010, the Tripartite Review Committee (TPRC; the PFTAC Steering Committee) and IMF should develop a strategic plan that sets out a vision for where PFTAC should be in five years time.
  - Implementing action: Based on a discussion at the TPRC in June 2009, work is in progress on a program document for PFTAC’s new cycle starting on May 1, 2011. The program document will set out the strategic plan. A draft for discussion will be sent to donors and recipient countries in the summer of 2010.
- Recommendation 2: By the end of FY2010, consensus should be reached at TPRC on the priorities for additional Resident Advisors.
  - Implementing action: The TPRC in June 2009 decided to post an additional macroeconomic advisor starting 2010. The forthcoming program document will propose an expansion of PFTAC for the next cycle.
- Recommendation 3: Beginning in FY2010 PFTAC should make greater use of short term experts to leverage the expertise and associated fixed costs of Resident Advisors, particularly in the financial sector supervision and statistics areas.
  - Implementing action: Some expansion in the short-term expert program in PFTAC’s current funding cycle. The program document will propose a further expansion for the next cycle.
- Recommendation 4: By the end of FY2010 the TPRC, PFTAC Coordinator, Resident Advisors and the TA Departments should develop a strategy to strengthen the likely sustainability of the benefits of PFTAC assistance.
  - Implementing action: A strategy will be proposed in the context of the forthcoming program document.
- Recommendation 5: Beginning in FY2010 PFTAC should make a concerted effort to develop and use Pacific expertise.
  - Implementing action: The forthcoming program document will propose how to further develop and use Pacific expertise.
- Recommendation 6: By the end of FY2011 TPRC and PFTAC should develop a strategy to intensify and extend the use of regional approaches to build capacity in the Pacific Region.
  - Implementing action: A strategy will be proposed in the context of the forthcoming program document.
- Recommendation 7: By the end of FY2011 PFTAC should, in coordination with the TA Departments, clearly define medium term objectives to be achieved in each functional area in each country and verifiable indicators against which progress can be monitored.
  - Implementing action: The forthcoming program document will propose medium-term objectives in each functional area in each country, including verifiable indicators.
- Recommendation 8: By the end FY2010 PFTAC should develop formal action plans for each recommendation accepted by the TPRC, identifying the necessary resources and monitoreable benchmarks to implement those recommendations and report on the implementation status of the action plans to the TPRC in FY2011 and again in FY2012.
  - Implementing action: The forthcoming program document will propose an action plan for discussion at the TPRC in FY2011.

### CARTAC Recommendations and Implementing Actions
- Recommendation 1: CARTAC should re-consider the current reporting requirements with the IMF to reduce the burden of compliance with Fund reporting.
  - Implementing action: CARTAC is under a different arrangement with UNDP as project manager and the Fund as executing agency; CARTAC must report to both institutions. If recommendations 12 and 13 were implemented, CARTAC would be administered as any other RTAC, for which the Fund is in the process of establishing streamlined reporting.
- Recommendation 2: Country strategy meetings should be formalised and followed by a clear process for resident advisers to integrate outcomes into workplans and link with the wider Fund TA programme through the RSNs.
  - Implementing action: Work in progress on a program document for the next funding cycle starting on January 1, 2011; a draft for discussion will be sent by mid-2010. The program document will propose how to address this recommendation.
- Recommendation 3: CARTAC should develop a country-level window on its website to summarise activities and plans for each CARTAC country and include country strategy meeting summaries.
  - Implementing action: The program document will propose how to address this recommendation.
- Recommendation 4: The website should be updated to reflect the core objectives and mission of CARTAC rather than being only a statement of technical expertise and source of information.
  - Implementing action: The program document will propose how to address this recommendation.
- Recommendation 5: CARTAC should ensure a particular focus is placed on Haiti and develop a coherent strategy to increase TA delivery in Haiti within current resourcing.
  - Implementing action: The program document will propose how to address this recommendation. Discussions with donors are under way to provide scaled-up TA to Haiti through other channels.
- Recommendation 6: CARTAC considers a more equitable approach to cost recovery for TA delivery in middle income countries, including charging and/or larger annual contributions.
  - Implementing action: The program document will propose how to address this recommendation. Planned changes to TA delivery that the Fund is implementing will also have an impact.
- Recommendation 7: There should be an exit strategy for CARTAC in terms of donor support; CARTAC may progressively become regionally owned over the longer term.
  - Implementing action: The program document will propose how to address this recommendation.
- Recommendation 8: There should be an extension into Phase IV (2011- 2013).
  - Implementing action: The Fund will propose an extension of CARTAC for 5 years into Phase IV. The program document will lay out CARTAC’s objectives for Phase IV.
- Recommendation 9: The Steering Committee should consider measures to improve meeting effectiveness and information accessibility (including limiting multiple donor attendees, improving reporting formats, encouraging networking, formalising minute taking, and releasing draft minutes for comment).
  - Implementing action: The program document will propose how to address this recommendation.
- Recommendation 10: CARTAC should adopt a basic performance management reporting structure that aligns activities to the goal and programme objectives, moving toward full performance management once Fund framework is developed.
  - Implementing action: Work in progress to create basic performance management reporting for all RTACs.
- Recommendation 11: CARTAC’s financial reporting and management information should be modernised to reflect resources put into individual programmes and projects and integrated into an overall performance management framework.
  - Implementing action: The program document will propose how to address this recommendation. See also response to recommendation 1.
- Recommendation 12: CARTAC should assume all administrative, reporting and consular activities currently undertaken by the UNDP at the commencement of Phase IV.
  - Implementing action: The program document will propose how to address this recommendation.
- Recommendation 13: A portion of the existing management fee could cover increased administrative and management costs, including improved performance reporting; the bulk should be applied to more technical assistance in the region.
  - Implementing action: The program document will propose how to address this recommendation.
- Recommendation 14: The UNDP continues to engage with CARTAC on the steering committee.
  - Implementing action: The program document will propose how to address this recommendation.

### METAC Recommendations and Implementing Actions
- Recommendation 1: The Phase III proposal should include a clear statement of METAC’s objectives with defined performance indicators, articulation of linkages between activities and objectives, and key assumptions; develop a more results-focused reporting system and assess the role of TAIMS in monitoring and evaluation.
  - Implementing action: The program document for Phase III sets out verifiable indicators in each area of METAC’s activities. Objectives are linked directly to country activities. A results-based reporting system will help assess performance and measure results. METAC’s annual work plan will be linked to the objectives and METAC will report on them at each SC meeting. Following recent improvements, TAIMS will play a more important role in monitoring and evaluating performance.
- Recommendation 2: The Phase III project should address improving METAC TA performance through more focus on support to implementation, regional networking and information sharing and communication, including assessing costs and organizational requirements.
  - Implementing action: Phase III program document’s regional focus strengthened through regional networking, establishment of regional organizations promoting information sharing and problem solving. Regional workshops/seminars will continue and METAC’s revamped website will report main findings. METAC will focus more on following up recommendations though implementation remains with beneficiary countries.
- Recommendation 3: Financial reporting should cover total cost of METAC’s TA delivery and include explicit performance indicators, using the new Selected Fund Activities (SFA) reporting system.
  - Implementing action: Costing of METAC activities will be changed from standard to actual costs, and financial reporting will become more detailed under the IMF’s new financing instrument, the SFA.
- Recommendation 4: The increased focus on regional networking should be developed as part of information and training strategies and more effective use of METAC’s website; consider events linked to SC meetings to increase profile.
  - Implementing action: METAC’s website has been upgraded to include relevant documents produced by the IMF and METAC. Issuing press releases and holding press conferences have been practiced (e.g., SC meeting in May 2009). Regional networking will more actively involve countries from the GCC and North Africa to enhance knowledge transfer.
- Recommendation 5: To support increased information and training focus, METAC should consider strengthening capacity for training coordination and management via (a) short-term Training Coordination consultant; (b) new part-time permanent Training Coordinator; or (c) training and development of existing Administrators.
  - Implementing action: Recommendation needs further discussion. Strengthening training administration might be useful but expensive and possibly too ambitious given current training agenda. If donors provide additional funds, action could be reassessed.
- Recommendation 6: Build on the Office Procedures Manual by strengthening process management throughout METAC operations to ensure consistency and provide basis for performance measurement; drive this with a broader set of performance measures.
  - Implementing action: A broader set of performance measures has been developed and included in METAC’s program document for Phase III.
- Recommendation 7: The SC needs to play a more effective role in representing all beneficiary organizations, via appointment of alternates, subcommittees or networks, and better distribution of proposed Work Plans and Minutes to designated authorities; supplement single annual SC meeting with better communication (website, telephone conferencing, ad hoc meetings).
  - Implementing action: Next SC will discuss appointment of an alternate SC member from the next most important organization. METAC will encourage SC members to strengthen discussion with beneficiary agencies. Proposed work plans and minutes are already being distributed to SC members, who are responsible for distribution to counterparts. Communication with SC members is being done and will be enhanced.

*Appendix II — Recommendations RTAC Evaluations and Implementing Actions*

### Appendix III

### Appendix III — Ratings and Recommendations of Other TA Evaluations

### Switzerland Subaccount for Selected Projects — Support for IMF Capacity Building (July 2009)
- By independent consultant; Covers 1998-2009; Disbursements US$8.4 million.
- Summary Evaluation (scale of 1 (poor) to 4 (excellent)):
  - Overall: Effectiveness 3.3; Efficiency 3.3; Relevance 3.1; Sustainability 2.5; Overall 3.1.
  - External Debt Project: Effectiveness 3.1; Efficiency 3.2; Relevance 2.8; Sustainability 2.2; Overall 2.9.
  - Government Securities Market Project: Effectiveness 3; Efficiency 3.2; Relevance 2.9; Sustainability 2.3; Overall 2.9.
  - AML/CFT Project: Effectiveness 4.0; Efficiency 3.7; Relevance 4.0; Sustainability 3.0; Overall 3.7.
- Recommendations:
  - Continue bilateral arrangements between Switzerland and the IMF.
  - Regional approach did not contribute to success of multi-country projects.
  - Lengthen and phase projects that have taken longer than envisaged.
  - Allow some flexibility within each phase to increase likelihood of success when government priorities shift.
  - Do not begin 2nd phase of the securities market project in Azerbaijan until the PDA has a strong head and staffing conditions are right.
  - Use TAIMS reporting formats to simplify project monitoring and reporting.
  - Provide staff training to maintain capacity.

### Japan—Administered Account for Selected IMF Activities (JSA) (December 2009)
- By Mitsubishi UFJ Research and Consulting; Covers 1990-2008; Disbursements US$279 million.
- Summary Evaluation:
  - Effectiveness: Good.
  - Efficiency: Good.
  - Relevance: Better than other international organizations; Highly relevant.
  - Sustainability: Achieved as far as under control of TA projects.
  - Overall: TA delivered good value for money.
- Recommendations:
  - Transparency:
    - Publish project-level evaluation questionnaires in Annual Report on the JSA.
    - Make more transparent the recruitment process for IMF roster of experts.
    - Institutionalize sharing with other donors of information on TA contents.
  - Improvement of Management:
    - Improve evaluation and assessment systems to sharpen results-focus and increase accountability.
    - Develop strategy for selection of TA projects paying attention to IMF and country context, time-bound targets, and an exit policy.
    - Focus more on "sustainability" and "ownership" in project design and execution.
    - Create more efficient system for website dissemination of project information to donor.
  - Improvement of Visibility:
    - Clearly indicate JSA funding on all relevant TA material.
    - Provide better information to recipient governments on JSA funding of TA.
  - Enhance Japan's Involvement:
    - Strengthen recruitment of Japanese experts for short-term assignments.
    - Hold regional forums in Japan on IMF/JSA TA.

### Assessment of CARTAC Technical Assistance to Support VAT Implementation (July 2009)
- By independent consultants; Covers 2005-2008; Disbursements US$5.7 million.
- Summary Evaluation:
  - Effectiveness: Yes.
  - Efficiency: Not evaluated.
  - Relevance: Yes, with strong ownership.
  - Sustainability: Yes, but risks must be addressed.
- Recommendations to CARTAC:
  - Continue current design of VAT implementation but consider supporting parallel reforms to tax administration.
  - Provide hands-on TA and training in complex auditing and legislative review.
  - Assist with compiling legal interpretations ("rulings").
  - Arrange regional forum for VAT coordinators.
  - Continue regional horizontal capacity building model.
  - Apply a more results-oriented TA design and planning methodology.

### Evaluation of Reforms in Tax Policy and Administration in Mozambique and Related TA (September 2009)
- By FAD staff; Covers 1994-2007; Outlays 25 person years.
- Evaluation findings:
  - IMF TA evaluated against success in modernizing tax system, rectifying flaws in tax and customs administration, creating modern revenue administration institutions, and improving revenue collection.
  - Report develops a methodology for quantitative assessment of success achieved.
  - IMF TA appears to have spurred and supported reform in key areas and played a decisive role in developing modern, effective legal and institutional instruments.
  - Implementation of reforms continues under Mozambican tax authorities' Strategic Plan for 2008-2012.

### Evaluation of Technical Assistance to Post-Conflict Countries — Mozambique and Rwanda (February 2009)
- By STA staff; Covers 2000-2008.
- Summary Evaluation:
  - Effectiveness: Most effective for central bank produced statistics (BOP and monetary).
  - Efficiency: Not evaluated.
  - Relevance: Very useful in meeting countries' priority needs.
  - Sustainability: National statistical offices hampered by resource constraints.
  - Overall: Favorable.
- Conclusions:
  - Rebuild institutions before TA can be absorbed in post-conflict countries.
  - Country commitment to and ownership of TA is vital.
  - Statistical legal and institutional frameworks are important.
  - Adequate resources, including staff counterparts, are necessary.
  - Reforms must be prioritized and linked to the budget.
  - Coordination with donors and Fund departments is necessary to permit flexibility in supporting interlinked reforms.
  - The GDDS framework is valuable for coordinating documentation and updating progress.
  - Address language constraints.
  - Expert selection and mix of TA delivery are important for STA.

### Review of Technical Assistance and Training on Statistics in the People's Republic of China (November 2009)
- By STA staff; Covers 2002-2009.
- Summary Evaluation:
  - TA and training missions were highly appreciated.
  - Efficiency: Not evaluated.
  - Relevance: All five recipient agencies strongly agreed that TA responded to their priority needs.
  - Sustainability: Risks must be addressed to sustain momentum.
  - Overall: TA program contributed significantly to substantial progress in all areas of macroeconomic statistics in recent years.
- Recommendations:
  - Shift focus from traditional TA to technical cooperation and consultation.
  - Expand TA delivery modalities (staff secondments to STA, contacts with agencies in other countries, study tours, host joint seminars with experts).
  - Develop strategies to deal with rapid staff turnover.
  - Develop coordinated long-term statistical strategy for all agencies to identify priority areas for TA.
  - Prepare for problems associated with IMF TA charging regime.
  - In the medium-term, refine training activities in line with suggestions from Chinese agencies.
  - Chinese agencies should address issues raised by data users regarding usability, accessibility, credibility, timeliness and frequency of data.

### Evaluation of Technical Assistance on Bank Supervision by Long-term Experts in Asia (June 2009)
- By MCM staff; Covers 2002-2009; Disbursements US$6.8 million.
- Summary Evaluation:
  - Effectiveness: Major progress made.
  - Efficiency: Not evaluated.
  - Relevance: Areas of TA and broad goals highly appropriate.
  - Sustainability: Not evaluated.
  - Overall: Consensus that LTEs have been successful.
- Recommendations:
  - LTEs are more suitable than other forms of TA in early stages of reform.
  - LTEs may be more effective in less visible, "behind the scenes" activities.
  - LTEs must have proper interpersonal skills and willingness to understand and adjust to local conditions.
  - TORs should permit flexibility to adjust to situation on the ground; involve host institution including middle managers.
  - Improve backstopping, quality control and knowledge management at IMF headquarters.
  - Amend special reporting status of long-term central bank advisors.
  - Arrange donor funding to cover backstopping and administration of TA projects.

### Macro Policy Lessons for a Sound Design of Fiscal Decentralization (July 2009)
- By FAD staff.
- TA Evaluation:
  - Effectiveness of Fund advice in fiscal decentralization has varied significantly across countries and time.
  - Degree of political support for reforms is a fundamental explanatory factor.
  - Easier to implement piece-meal reforms than comprehensive ones that achieve more efficient and lasting outcomes.

### The Framework for Technical Assistance in the IMF: Backstopping to Project Management (September 2009)
- By an independent consultant.
- Summary Evaluation:
  - Backstopping is core IMF staff work in direct support of experts in the field.
  - Backstopping differentiates IMF TA delivery from other TA providers lacking an institutional framework for systematic backstopping.
  - Through backstopping the IMF comparative advantage is realized.

*Source: pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program - Appendix III*

### 6.      Development of Inflation Targeting and Monetary Modeling in Emerging

### 6.      Development of Inflation Targeting and Monetary Modeling in Emerging Market Europe (MCM)

### Summary of context and Fund involvement
- Several European emerging market countries have moved toward inflation targeting and central banks have been actively developing concomitant monetary modeling capacity.
- The Fund has provided technical assistance in this area to many European countries, including Croatia, Moldova, Romania, Russia, Serbia, Turkey, and Ukraine.
- A recent Board paper on inflation targeting addressed these issues in depth.
- An evaluation, to be conducted in FY2010, would cover a cross-section of these countries.

### Issues proposed for evaluation
- Linkages between progress in inflation targeting/monetary modeling and progress in other areas, such as de facto central bank independence and financial market development.
- How the design of inflation targeting and monetary models depends on the exchange rate regime.
- The importance of pre-existing local capacity.
- The effectiveness of different modes of delivery (e.g., resident experts versus periodic visits).

### Relevance
- The results of the evaluation would be of relevance to other emerging market countries in Europe and elsewhere.

---

### 7.      Evaluation of risk-based supervision TA in Costa Rica and Nicaragua (MCM)

### Program description
- With TA from the Fund, Costa Rica and Nicaragua have embarked on a major effort to introduce risk-based supervision.
- The TA focuses on review and suggestions for improvement to facilitate risk-based supervision in two major areas:
  - Prudential regulatory framework, including for consolidated supervision, prudential treatment of certain banking activities such as leasing and factoring, capital adequacy, and credit, liquidity and foreign exchange risks.
  - Examination practices and methodology (on-site and off-site), including bank rating system, bank management evaluation methodologies, supervision reporting system, and supervision strategy and planning.
- The TA, which ends in April 2011, also aims at laying a solid foundation for risk-based supervision in both Costa Rica and Nicaragua by developing manuals and procedures for risk-based supervision, and training and coaching for staff engaged in bank supervision.
- It consists of a 24-month work program managed by an external expert on banking regulation and supervision, supported by the supervisory authorities of a number of Latin American countries, and backstopped by MCM staff at headquarters.

### Proposed evaluation focus and timing
- The evaluations will focus on progress made in the implementation of risk-based supervision in both Costa Rica and Nicaragua in FY12.
- If agreed by the authorities in both countries, the evaluation would assess:
  - The extent of the use of risk-based supervision in Costa Rica and Nicaragua by then.
  - The progress made in adopting the recommendations associated with the different components of the technical assistance.
  - The adequacy of the manuals, procedures for both on-site and off-site supervision, training, and coaching.
- The evaluation would also, if necessary, provide specific actions to address any shortcomings in the implementation of risk-based supervision.
- The Fund would offer TA to put in place such actions.

---

### 8.      Statistical capacity building in selected countries with significant TA delivery—Nigeria, Albania, Georgia and Peru (STA)

### Rationale and country selection
- STA has adopted a regional approach to provide strategic and country perspectives to its TA program.
- Periodic evaluations of TA provided to selected countries in each region would help assess the effectiveness of TA delivered.
- Proposed selected countries: Nigeria (AFR), Albania (EUR), Georgia (MCD), and Peru (WHD).
- Selection is based on diversity in stages of statistical development along with significant TA delivery.

### Evaluation coverage and emphasis
- The evaluations will cover the outcomes of the TA program in each selected country in the recent period, offering a clear assessment of effectiveness and providing lessons STA can draw on.
- Particular emphasis will be directed at the features of TA delivery most conducive to building capacity for the compilation and dissemination of macroeconomic statistics.
- The evaluation will consider:
  - The role played by the country authorities benefiting from the TA.
  - Constraints posed by resource inadequacies.
  - The extent of commitment and ownership by the beneficiaries.

### Methodology and timing
- The evaluations will be undertaken mainly using assessments by Fund staff, supplemented by field missions of a few days’ duration in each country to review TA delivered.
- Feedback from country authorities will be a particularly important component of the evaluation.
- The exercise will entail an examination of documents associated with TA delivery for each country, including back-to-office and mission reports, responses to questionnaires, and interviews with representatives of country authorities and with Fund staff.
- The evaluations will be carried out in FY11–12.
- An evaluation report will be prepared for each of the selected countries.

*Source: Excerpt from pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program*

---


_Source: https://www.imf.org/-/media/files/publications/pp/pp4251-technical-assistance-evaluation-program-findings-of-evaluations-and-updated-program.pdf_