IEO logo
IEO Home

Independence

Goals & Instruments

Transparency

Work Program

Ongoing Projects

IEO Publications

Staffing

Terms of Reference

Origins of the IEO

Links to Related Sites


Independent Evaluation Office - Work Program

October 23, 2003

The IEO provides objective and independent evaluation on issues related to the IMF. The Office operates independently of IMF management and at arm's length from the IMF's Executive Board.

WORK PROGRAM FOR FISCAL YEAR 2005
Discussion Paper for Comments


1. The work program of the IEO is determined by the Director on the basis of consultations with a broad range of stakeholders. To facilitate these consultations, this paper presents a short list of topics from which the program for FY 2005 (May 2004 to April 2005) will be chosen. It is expected that this work program will consist of four evaluations.

2. The short list has been chosen from the medium-term work program published by the IEO in November 2001, which suggested 15 possible evaluation topics (see Appendix 1). Six of these have already been taken up in the first two full years of operation. The remaining 9 topics (nos. 7 to 15 in the Appendix) remain relevant today. A tenth topic which may be added is a case study of Turkey, which was originally envisaged as an alternative to Argentina.


3. The selection criteria used earlier gave priority to topics that (i) have been the subject of controversy or criticisms; (ii) have the greatest interest for a wide range of the membership; (iii) have the greatest learning potential; and (iv) for which the IEO has a clear comparative advantage. Applying these criteria, we have identified the following short list of seven topics.

A. The IMF's experience with financial sector assessment programs (FSAP) and the associated financial sector stability assessments (FSSA);
B. A case study of the role of the IMF in Turkey;
C. The IMF's approach to capital account liberalization;
D. Structural conditionality in IMF-supported programs;
E. Private sector involvement in the prevention and resolution of crises;
F. The role of multilateral surveillance; and
G. A low-income country case study.

4. The IEO welcomes suggestions on which of the topics listed above should be included in the final list of four evaluations to be undertaken in FY 2005. Topics not included in the short list can also be considered if there are good reasons for preferring them on the basis of the criteria outlined above. Comments can be addressed to ieo@imf.org.
5. The main issues in each of the seven topics listed above, and some considerations relevant in determining priority, are spelled out below.

A. Financial Sector Assessment Program (FSAP) and Financial Sector Stability Assessments (FSSA)

6. The FSAP/FSSA is a major initiative undertaken by the IMF, in collaboration with the World Bank, in response to the crises of the late 1990s. It is designed to strengthen bilateral surveillance of member countries and contribute to crisis prevention by helping to identify financial sector vulnerabilities and generate policy recommendations to overcome them. The experience under this initiative was last reviewed by the Executive Board in March 2003, based on an internal joint assessment by IMF and World Bank staff. Approximately 70 assessments are expected to be completed by the end of FY 2004. An independent evaluation of this important, and resource-intensive, activity is therefore timely.

7. The proposed evaluation could focus on areas of broad program design and integration with surveillance functions, as well as implementation aspects, including:

Program design

  • How has the Fund's involvement in financial sector stability issues aligned with its core macro-stability mandate? What has been the overarching framework guiding the design of the FSSA/FSAP and its integration into the Fund's surveillance function?

  • What has been the role of the Fund in financial sector surveillance vis-à-vis other IFIs? What has been the division of labor, including on leadership for LDCs, emerging countries, and advanced economies; and on development, and soundness and stability issues?

Implementation

  • Risk Assessment: How well has financial sector assessment under the FSAP and associated FSSAs identified the major risk factors in a country's financial system? Are best approaches to diagnose vulnerabilities being used?

  • Addressing Vulnerabilities: Have FSSAs proposed appropriate and feasible measures to address vulnerabilities? Are recommendations articulated as a coherent "blueprint" for reform with proper regard to suitability, sequencing, and implementation capacity? Have actions/reforms been effectively carried out? How has the Fund operated to induce countries to address financial sector vulnerabilities in its surveillance and UFR operations? Has the exercise led to a better understanding of TA needs in the context of strengthening the financial sector, and has this led to appropriate IMF response in the provision of TA?

  • Assessment of Value Added: How do country authorities assess the FSAP/FSSA exercise? Has it deepened the quality and coverage of discussions with authorities on financial sector issues? Have authorities perceived an effective reduction of vulnerabilities? How has the private sector reacted to the new information published in FSSAs and how has it impacted their activities and risk perceptions?

  • How effective has been the coordination efforts with the World Bank? Is there an appropriate balance between industrialized and developing countries covered, given resource requirements and potential benefits? Does the FSAP/FSSA achieve its goals in a cost effective manner? Is the current degree of transparency (which involves voluntary publication of FSSAs) appropriate? What has been the role of the FSAP and FSSAs in recent crisis intervention cases?

8. The evaluation would likely have to be conducted at two levels. Some issues would be examined on the basis of a cross country assessment of experience in all the FSAP cases. This cross-country assessment would have to be supplemented by in-depth study of a sample of country cases.

9. Since the FSAPs are undertaken jointly with the World Bank, it would be desirable to undertake the study in parallel with an evaluation by the World Bank's Operations Evaluation Department of the Bank's experience in this regard. Preliminary discussions with OED have been held to this end.

B. A Case Study of Turkey

10. Turkey provides another example of a capital account crisis involving large access, similar in some ways to the cases in Korea, Indonesia, and Brazil already studied by IEO and the study of Argentina which is currently under way. The country experienced severe financial instability in 1999 leading to the request for a stand-by arrangement for SDR 9.3 billion (US$12.7 billion) at the end of 1999, to protect the fixed exchange rate which was viewed as an essential nominal anchor. The amount under the arrangement was subsequently augmented, but the effort to maintain the exchange rate peg proved unsustainable and was abandoned in February 2001 when the lira was floated. The 1999 arrangement, with SDR 3.3 billion undrawn, was cancelled in February 2002, and a new SBA for SDR 12.8 billion was put in place. This program is expected to end in December 2004.

11. An evaluation of Turkey's experience, including the process by which key decisions on IMF-supported programs were taken, should provide additional lessons on the effectiveness of Fund surveillance in crisis prevention and the Fund's approach to crisis resolution. It would be most useful if the study could cover a longer period after the initial crisis than was possible in the earlier IEO studies of crisis cases, as this would enable the IEO to focus on the post-crisis period in greater depth. The exact period to be covered under the evaluation will be chosen so as to avoid interfering with ongoing operations. The experience with the original 1999 arrangement which was cancelled in February 2002 can be examined without interfering with ongoing operations. The evaluation could be extended to cover issues relating to the second 2002, arrangement after December 2004, when the present arrangement is set to expire.

12. The evaluation would seek to address several questions related to the adequacy of IMF diagnosis and policy advice, including:

(i) Surveillance in the pre-crisis period will be critically examined to assess whether it identified the problems which surfaced later, especially regarding inflation stabilization, the use of the exchange rate as a nominal anchor, and consideration of vulnerabilities, risks, and exit strategies.

(ii) Were the programs well designed from the point of view of assessing fiscal sustainability, both before the crisis and during the period of crisis management?

(iii) Did program design focus adequately on the structural factors that have impaired fiscal adjustment and have been a source of financial sector vulnerability?

(iv) Were programs designed on the basis of a realistic assessment of the extent to which the proposed reforms were owned by the authorities, as well as the authorities' capacity to implement key measures?

(v) Did crisis management, including the defense of the exchange rate regime, take adequate account of banking sector problems?

(vi) How well did IMF policy advice in the post-crisis period cope with the challenges of stabilization and continuing vulnerabilities, including debt sustainability and banking soundness? Parallels with Argentina in this respect would be interesting.

C. The IMF's Approach to Capital Account Liberalization

13. The issue of liberalization of the capital account has been the focus of a great deal of attention and generated considerable criticism of the IMF. Liberalization of capital account transactions is not an explicit part of the mandate of the Fund but policies relating to liberalization of the capital account have figured in Fund surveillance. Critics have argued that the IMF has encouraged emerging market countries to liberalize capital flows in a manner that has subjected them to severe stress because of the unexpected volatility of these flows. Some have argued that rapid liberalization, with insufficient attention to sequencing and establishing the preconditions for effective integration with global financial markets, has been responsible for a great deal of the financial instability and associated economic distress experienced by many emerging market countries.

14. The IMF has argued that its approach to capital account liberalization has become much more pragmatic after the East Asian crisis and there is now a clear recognition of the need to build strong financial systems as a precondition for capital account liberalization. The proposed evaluation would assess whether indeed the Fund's policy advice on the scope, pace and composition of capital account liberalization has changed after the late 1990s. The evaluation would be largely based on a selective study of surveillance and program documents in a representative subset of countries before 1997 and after 1999, supplemented by an examination of other internal documents, if any.

15. Specific issues which could be addressed by the evaluation include the following:

D. Structural Conditionality in IMF-Supported Programs

16. The IMF was criticized in many quarters for the widening of both the scope and the volume of structural conditionality over the last decade. The Managing Director responded to this criticism by issuing, in the Fall of 2000, a set of interim guidelines for staff on streamlining conditionality and, shortly thereafter, launching a thorough review of its conditionality. This led to the adoption by the Executive Board of revised conditionality guidelines in September 2002. These guidelines call for streamlining conditionality to achieve parsimony while ensuring that conditionality covers all policies critical to the achievement of macro-economic objectives. They recognize that critical policies would include policies for which other institutions, notably the World Bank, have primary responsibility. They also emphasize the importance of ownership for conditionality to be effective.

17. The evaluation could examine the internal consistency of the new guidelines, including (a) the possible tension between emphasizing parsimony and also covering all policies critical to the achievement of macroeconomic objectives, and (b) the tension between an approach that emphasizes country-owned adjustment programs but also retains the traditional dependence on conditionality. It could assess the extent to which the new guidelines have brought about a significant change in actual practice in focusing on the four core areas of structural conditionality—fiscal, trade, financial and privatization. This would require comparison of the situation before September 2002 (as already documented in a staff review) and the current position reflected in programs approved after the September 2002 guidelines were issued. Specific issues to be investigated could include:

  • What is the evidence about the effectiveness of structural conditionality? How have programs dealt with the issue of ownership and what are the implications for the design of structural conditionality? In addressing these issues, and others listed below, the evaluation will need to distinguish between PRGF and non-PRGF cases and for the former would draw upon any lessons emerging from the ongoing evaluation of the PRSP/PRGF Initiative.

  • How has the potential conflict between the objective of achieving parsimony and the objective of ensuring that conditionality cover all policies critical to achieving macroeconomic objectives of the program been resolved in practice? Have programs identified core essential elements and non-core desirable elements while limiting conditionality explicitly to the former?

  • How have staff approached the task of assessing macroeconomic relevance in deciding whether or not to include particular structural measures in IMF conditionality?

  • Has the concept of division of labor led to a redesign of programs to give greater emphasis to the areas of conditionality that are within the competence of the IMF, with less emphasis being given to the others. Where other areas of conditionality remain important, how is the objective of parsimony achieved in collaboration with other institutions?

  • How has enhanced collaboration with the World Bank in the design of conditionality been implemented in practice?

E. Private Sector Involvement (PSI) in Crisis Resolution

18. The need to involve the private sector effectively in the resolution of financial crises in order to contain the size of access and also to create market-based systems of resolution with the right mix of incentives has been much discussed. In principle, an evaluation of recent country programs could be undertaken to address issues such as:

  • How are private market creditors being involved in the resolution of financial crises and are principles in this respect being applied consistently? Is debt sustainability/solvency analyzed in a consistent manner? Is there a tendency to be over-optimistic about the expected return to normalcy, e.g., expected reductions in risk premia?

  • What has been the catalytic role of Fund-supported programs and official financing packages in mobilizing private capital (in terms of both volume and terms of financing)? Why in many cases were private flows less than assumed in the program?

  • How effective have been the various concerted efforts, short of a standstill, to secure private sector involvement in major crisis uses? What has been the outcome in terms of burden-sharing among creditors? Are further institutional changes required to facilitate PSI?

  • What lessons do the early cases suggest about the impact of concerted PSI on the real economy on the resumption of voluntary flows, and on the availability of financing to other countries?

While these are important issues, an argument against undertaking an evaluation at this stage is that policy in this area is currently being discussed and there are significant differences within the official community on how the issues should be handled. The management initiative on introducing a Sovereign Debt Restructuring Mechanism was potentially an important element of private sector involvement but there is no agreement on this proposal as yet. There is general agreement on the usefulness of Collective Action Clauses (CAL), but too little experience.

F. The Role of Multilateral Surveillance

19. Multilateral surveillance is a core activity of the IMF that aims to provide a basis for analyzing the forces driving the world economy, identifying global vulnerabilities, and advising on appropriate policies, especially in the most systemically important countries. Potential issues to be addressed would include the following:

G. An Additional Country Case Study

20. The IEO could undertake an in-depth examination of a recent country program with the focus on the effectiveness of program design and conditionality. The primary goal would not be to assess whether the IMF's actions were right or wrong in a particular case (although that would be an additional outcome), but to investigate in some detail what the case suggests about the IMF's recent approach to program design as well as the structure and focus of its conditionality. Programs chosen could be from either 2000 or 2001, allowing evaluation of outcomes in the post-program period. Two possible options could be considered:

(i) An investigation of program design in a middle-income country, but not one of the major "capital account crisis" cases, which are being examined in other evaluations. Possibilities would include Bulgaria, Peru, Romania, Sri Lanka, or the Ukraine.

(ii) A low-income country that has not been able to advance to the PRSP stage owing to weak policy formulation processes, fragile institutions, and a variety of governance and other political problems, including post-conflict situations4. An in-depth examination of the IMF's role in one such case could help generate lessons on what the IMF's role and approach should be in such cases.

  • The focus would be on the relevance and effectiveness of the IMF's role in such cases, including whether the traditional approach to program design and conditionality is well-suited to situations where weak governance and institutions are the core problems to be addressed.

* * * * *

Comments are invited from all interested parties on the priority that should be attached to the seven topics discussed in this paper in choosing 4 topics for the IEO work program for FY 2005. Suggestions for including topics not discussed in the paper are also invited.

The deadline for submitting comments is Monday, December 22, 2003. Comments can be submitted through the IEO website www.imf.org/ieo. The final work program will be determined after taking account of comments received and will be posted on the IEO website sometime in January 2004.


1Capital Account Liberalization and Financial Sector Stability—Considerations for Sequencing (SM/01/186, 6/25/2001).
2There is some potential overlap between this issue and an evaluation of multilateral surveillance (see topic F).
3The 1999 External Evaluation of IMF Surveillance identified an inadequate cross-fertilization between multilateral and bilateral surveillance as a problem and made a number of recommendations for improvements. The study could also assess the impact of these recommendations
4The ongoing evaluation of the IMF's role in the PRSP and the PRGF is focusing on those countries that have completed "full" PRSPs. This evaluation is expected to be completed in Spring-2004 and will include six background case studies. However, only 32 of the 77 countries eligible for IMF concessional lending through the PRGF have actually completed PRSPs as of end-September 2003.

Appendix

Proposed Medium-Term Work Program1
FY 2002/2003
1. Prolonged use of IMF resources
2. The role of the IMF in three recent capital account crises, Indonesia, Korea, and Brazil
3. Fiscal adjustment in IMF-supported programs
FY 2004
4. An evaluation of the PRGF/PRSP experience(to be undertaken jointly with the World Bank's OED)
5. Argentina case study (In the original medium-term program, this item indicated Argentina and Turkey as alternative possibilities)
6. IMF technical assistance
Remaining topics
7. The IMF's advice on financial sector restructuring after a crisis
8. Structural conditionality in Fund-supported programs.
9. The role of IMF surveillance in crisis prevention
10. The IMF's advice on exchange rate policy
11. FSAP/FSSA
12. Private Sector Involvement (PSI)
13. The IMF's approach to capital account liberalization
14. The role of multilateral surveillance
15. Low-income country case

1The list of topics is taken from the original list of 15 topics in "Proposed Work Program of the Independent Evaluation Office (IEO)" November 21, 2001 which can be accessed at www.imf.org/ieo. The order in which the topics appear has been changed to reflect the selections already made in the FY 2003 and FY 2004 work programs.