October 22, 2002

The IEO provides objective and independent evaluation on issues related to the IMF. The Office operates independently of IMF management and at arm's length from the IMF's Executive Board.

Draft Work Program for Fiscal Year 2003–04: Some Options

Discussion Paper for Comments

1.  The work program of the IEO in its first full year of operation FY 2002-03 consisted of three projects chosen from a list of 15 identified as possible evaluation projects which would be undertaken over a three-year period. The first IEO evaluation on Prolonged Use of IMF Resources has been completed and is available on the IEO website along with a summary of the Executive Board discussion. The other two studies on The Role of the IMF in Capital Account Crises: Korea, Indonesia and Brazil and on Fiscal Adjustment in IMF-Supported Programs are scheduled to be completed by the end of April 2003.

a) The PRSP/PRGF experience based on full PRSPs.

b) Country case study of Argentina or Turkey programs.

c) The role of the IMF in providing Technical Assistance.

d) The IMF's surveillance function.

e) The IMF's approach to capital account liberalization.

4.  The issues involved in each of the five studies listed above are outlined briefly in this note. Comments are invited from all interested parties on which subjects should be included in the work program and also on specific aspects of these subjects which may not be adequately covered in the note.

Evaluation of the PRSP/PRGF experience

5.  Through the creation, in 1999, of the PRGF, the IMF shareholders have given it a responsibility to help low-income countries to deal with structural balance of payments problems in a manner which encourages pro-poor growth based on Poverty Reduction Strategy Papers (PRSPs) which emerge from a consultative and participatory process. This is intended to ensure a country driven agenda of reforms. The PRSP is also expected to provide a mechanism for effective collaboration with the World Bank, which has primary responsibility for many of the structural reform areas, and for coordination with other donors.

6.  The initial experience based on 19 "Interim-PRSPs" and associated PRGF arrangements between 1999 and 2001 showed only tentative improvements over the earlier ESAF programs, but this could be because the early PRGFs were based on Interim PRSPs. Since then, about 18 countries have completed full PRSPs and 15 are engaged in PRGF arrangements based on these PRSPs. Since PRSPs involve both the IMF and the World Bank it is proposed that aspects of the evaluation related to the PRSPs be undertaken in parallel with the World Bank's Operations Evaluation Department (OED). However, the IEO and the OED would each produce their own evaluation report, to be submitted to their respective Boards.

7.  The evaluation would seek to present an overview of experience in all countries which have completed PRSPs based on cross country data combined with a detailed evaluation of between six to eight case studies. Since the evaluation will be undertaken in the course of calendar year 2003, it is too early to evaluate performance against long-term objectives such as the Millennium Development Goals, but the available information on the first two and in some cases three years under PRGF-supported arrangements will provide useful insights on many important aspects.²

Some of the questions which will be addressed are:

• Has the PRSP approach helped countries adopt broadly owned, effective poverty reduction strategies?

• Has the PRSP/PRGF process helped to define more clearly the role of the IMF in low-income countries in a useful way?

• Have the macroeconomic programs underlying PRGF arrangements been derived from broader strategies for fostering growth and reducing poverty set out in the PRSPs and do the latter provide sufficient basis for such macroeconomic frameworks?

• Has the approach to negotiation and program design adopted for PRGF-supported arrangements been substantively altered to conform to the PRGF operational guidelines? Should these guidelines be further amended?

• How effective has Bank-Fund collaboration been in implementing the PRGF/PRSP approach?

• Can any conclusions be drawn at this stage about the impact of the PRGF on growth and/or poverty reduction?

• What does experience to date suggest about the IMF's longer-term role in low-income countries, especially "post-stabilization" cases, and the need for an "exit" strategy.

In addressing these questions, the evaluation would seek to look beyond process issues and to assess the substantive policy changes brought about by these initiatives. It will also assess performance in the two or three years on which data will be available.

8.  In announcing the work program for FY 2002-03, it was indicated that the PRGF/PRSP study would be undertaken in FY 2003-04 and feedback received since then has reinforced this view. IEO has therefore commenced preliminary work on the issues paper for this study. The PRGF/PRSP evaluation would be a large project, involving considerably more resources than an "average"-sized project. If this project is included in the final list as is very likely, the total number of projects may have to be restricted to three.

Case study of Argentina or Turkey

9.  It has been argued that the IEO should routinely do ex post evaluations of all large programs. Argentina and Turkey represent recent cases of large program support and could be considered as individual case studies. The results of the evaluation would build further on the findings of the ongoing study of capital account crises in Korea, Indonesia and Brazil.

(a) Argentina

10.  The crisis in Argentina with the collapse of the currency board arrangement has raised a number of questions about the effectiveness of IMF crisis prevention and the quality and impact of its policy advice. This is especially so since Argentina was continuously engaged in IMF-supported programs for most of the period since the adoption of the Convertibility Law in 1991 and was often presented as a success story.

11.  The evaluation would focus on the IMF's involvement with Argentina from 1991 until the recent crisis and would seek to address two sets of questions:

(i) Did the IMF provide adequate and effective policy advice? Issues to be looked at under this heading would include the quality, timeliness and impact of the IMF's policy advice on the currency board and possible "exit" strategies, on fiscal policy and debt sustainability, and on structural policies required for the successful operation of the currency board arrangement. Reasons for nonimplementation of IMF policy advice would also be examined.

(ii) Was it appropriate for the IMF to remain continuously engaged in Argentina with lending arrangements over that period? Issues to investigate would include program design adequacy, possible moral hazard effects of continued IMF involvement on both the authorities and financial market participants, the modalities of the internal decision making process at key stages of Argentina's programs--including possible defensive lending considerations and internal governance issues--and private sector involvement.

12.  The primary focus of the evaluation would be on drawing lessons from this experience that would contribute to improve the effectiveness of IMF lending and surveillance activities in future.

(b) Turkey

13.  Turkey experienced severe financial instability in the past few years leading to a request for a stand-by arrangement at the end of 1999. The original intention was to protect the exchange rate which had come under pressure and was viewed as an essential nominal anchor. This effort proved unsustainable and was abandoned in February 2001 when the lira was floated. The amount of the stand-by arrangement was increased substantially to SDR 15 billion in several rounds between December 2000 and May 2001. This arrangement was cancelled in February 2002 with SDR 11.7 billion drawn and a new SBA adopted for SDR 12.8 billion expiring in December 2004.

14.  An evaluation of the IMF's experience in Turkey would provide useful lessons on the effectiveness of Fund surveillance in the pre-crisis period and the reasons for several strategic changes in the program including especially the switch from defending the exchange rate initially followed by floating. The study would address a number of questions. What are the lessons on how to stabilize an inflationary economy where important structural factors impair progress on the fiscal side and remain a major source of overall financial vulnerability? What is the critical progress needed in the quasi fiscal area (state enterprise reform, extrabudgetary funds, etc.) and the banking sector prior to embarking on an exchange rate based anti-inflationary program? How can a program expect to quickly change market expectations and real interest rates in situations where the latter constitute a major burden on public finances?

15.  In view of the constraints on the total number of projects that can be undertaken by the IEO in FY 2003-04 the Argentina and Turkey case studies might be considered as alternatives. Since the current SBA approved in February 2002 will expire in December 2004 it may be more appropriate to take up the Turkey study in FY 2004-05 as the evaluation could cover the entire period 1999-2004.

Technical assistance by the IMF

16.  Technical assistance (TA) is regarded as critical for capacity building and fostering ownership, which are viewed as increasingly important in both surveillance and program contexts. TA is potentially extremely important in low-income countries, which often suffer from severe capacity constraints that limit the effectiveness of policy formulation and implementation. It is also important in other member countries especially where the growing concern with financial sector stability points to many new areas where TA could help to strengthen the financial system and thus help crisis prevention.

17.  A review of the IMF's TA activities was undertaken by an internal evaluation group in 1999 and a number of shortcomings were identified, including uneven effectiveness, in part owing to weak follow up by the Fund and the lack of satisfactory internal evaluation and reporting procedures, and an insufficient complementarity between surveillance and TA activities. Several initiatives were taken to address these shortcomings including the establishment of a centralized Office of TA Management, the adoption of a policy framework to ensure that TA supply is suitably prioritized, and the opening of regional TA centers in the Caribbean and in Africa. In the past few years new areas of TA have emerged, including especially in the financial sector, where an evaluation of what has been done would be useful before expanding the scale of the effort.

18.  TA raises somewhat different questions in low-income and in middle-income countries. The focus of the evaluation and the choice of case studies would take account of the coverage provided by other projects retained in the final work program

19.  Some of the questions which the study could address are:

• Is the framework used to allocate TA resources across countries and across subjects appropriate? Is adequate attention paid to ensuring cooperation with other TA providers?

• How effective has IMF TA been in building institutional capacity and fostering sustainable reforms? How is absorptive capacity taken into account and how effectively is the policy advice tailored to the circumstances of each country? How is transfer of knowledge emphasized and what measures are taken to make TA more goal oriented? Is there sufficient consultation with countries before the finalization of terms of reference for TA missions and resident experts?

• Is there enough complementarity between TA and other major IMF activities, such as surveillance and use of IMF resources?

• What is the interaction between TA and the Financial Sector Assessment Program (FSAP) and initiatives to strengthen standards and codes?

• Are the IMF's internal TA evaluation procedures adequate? The evaluation will cover the outcome of ongoing efforts to establish a common "best practice" evaluation methodology for all TA-providing departments.

IMF surveillance

20.  Surveillance is a core activity of the IMF and consists of bilateral surveillance conducted primarily in the context of Article IV consultations and multilateral surveillance on the world economy. The quality of the IMF's bilateral surveillance of some borrowing countries has already been reviewed as part of the evaluation of Prolonged Use of IMF Resources and will be extensively studied as part of future IEO studies of specific IMF-supported programs (including the ongoing evaluation of capital account crisis cases). This evaluation would thus focus on bilateral surveillance of industrialized countries and multilateral surveillance, which are closely connected. Effective surveillance of industrialized country policies combined with strong multilateral surveillance is key to enable the IMF to make valuable inputs in the process by which the international community achieves a shared perception on the state of the world economy.

21.  The IMF's surveillance activity was reviewed by an external evaluation panel in 1999 and more recently in the summer of 2002 in the context of the biennial internal review of surveillance, which led to a clarification of surveillance guidelines. A review in FY 2003-04 would therefore be an appropriate vehicle to examine the extent to which problems identified by the external review panel and those identified subsequently have been addressed. The study would address the following issues:

• The role and value added of the World Economic Outlook and International Capital Markets Report exercises in forecasting prospects for the global economy and identifying global vulnerabilities.

• The interaction between multilateral surveillance and bilateral surveillance of the major industrial countries. What is the value added and how could the impact of surveillance be improved?³

• The effectiveness of the IMF's inputs into the deliberations of various international groups such as the G-7 and various regional groupings (such as APEC and the Manila Group).

• What has been the impact of the increased emphasis on capital market issues in multilateral surveillance and the enhanced interactions with private financial market participants?

• Has multilateral surveillance been constructively used to help anticipate and prevent crises? Were the implications of international developments for individual countries appropriately analyzed and incorporated into surveillance of those countries? Did multilateral surveillance identify vulnerabilities associated with volatility of capital flows? Were contagion risks properly assessed?

The IMF's approach to capital account liberalization

22.  The IMF has been widely criticized for pushing capital account liberalization in developing countries in situations where the financial sector in those countries was not sufficiently sophisticated to avoid the build up of vulnerabilities associated with short-term capital flows. In this respect, critics argue that the IMF not only failed to prevent crises but may even have contributed to their emergence in the first place. In response to these criticism the IMF has often said that whatever may have been the approach prior to 1997, the IMF's approach to capital account liberalization has been much more nuanced subsequently. Specifically, it is argued that there is no effort to push countries to liberalize the capital account prematurely and there is clear recognition of the importance of sequencing and establishing the pre-conditions for successful liberalization of the capital account.

23.  The evaluation could examine IMF-supported programs and surveillance activity from 1998 onwards in order to determine what are the characteristics of the current approach. Specific questions to be addressed could include the following:

• Considering that the IMF does not have a formal mandate to get involved in capital account liberalization issues, what are the scope of and rationale for its involvement in such issues? Are they appropriate?

• Is the policy advice given by the IMF on capital account issues consistent across members? Does it take adequate account of differences between members' financial sector development and soundness, exchange rate regime, and general vulnerability?

• Should the IMF adopt a formal policy framework for its approach of capital account liberalization issues?

* * * * *

Comments on the proposals in this note are invited by November 20. Based on comments received and ongoing consultations being undertaken by the IEO, the work program for FY 2003-04 will be finalized by early December and subsequently submitted to the Executive Board for review as required under the IEO Terms of Reference. Comments may include suggestions for covering topics other than those discussed above.

APPENDIX I   

Items Previously Identified for Work Program

1. Poverty Reduction and Growth Facility (PRGF)/Poverty Reduction Strategy Paper (PRSP) (to be undertaken jointly with the World Bank's OED)
2. The IMF's advice on financial sector restructuring after a crisis
3. Structural conditionality in Fund-supported programs
4. The role of IMF surveillance in crisis prevention
5. The IMF's advice on exchange rate policy
6. Financial Sector Assessment Program (FSAP)/Financial System Stability Assessments (FSSA)
7. IMF Technical assistance
8. Private Sector Involvement (PSI)
9. The IMF's approach to capital account liberalization
10. The role of multilateral surveillance
11. Additional country case (possibly Argentina or Turkey)
12. Low-income country case