IEO logo
IEO Home

Independence

Goals & Instruments

Transparency

Work Program

Ongoing Projects

IEO Publications

Staffing

Terms of Reference

Origins of the IEO

Links to Related Sites


Independent Evaluation Office - IEO Publications

January 31, 2003
The IEO provides objective and independent evaluation on issues related to the IMF. The Office operates independently of IMF management and at arm's length from the IMF's Executive Board.

Evaluation of Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility

Issues Paper/Terms of Reference


This document is available in French (352 kb PDF file), Portuguese (349 kb PDF file), and Spanish (363 kb PDF file) - [Disclaimer: Reasonable efforts to ensure accuracy of these documents have been made; however, only the English version produced by the IEO is the authentic version]

Full text in PDF Format (English) (348Kb) 

I. Background Information

II. Terms of Reference of the Evaluation

III. Envisaged Methodological Approach

Bibliography

List of Abbreviations



Evaluation Of PRSPs and the PRGF

Issues Paper/Terms of Reference

In late 1999, to enhance the contribution of their interventions to international poverty reduction efforts, the International Monetary Fund (IMF) and the World Bank (WB) adopted a new strategy for their assistance to low-income countries. The main planks of this strategy were twofold: (i) both institutions were to base their concessional lending and debt relief to low-income countries on Poverty Reduction Strategy Papers (PRSPs) prepared by the countries themselves; and (ii) IMF concessional lending was to be provided through a revised lending facility, the Poverty Reduction and Growth Facility (PRGF)--with a stronger poverty reduction focus.

Of the 77 countries that are eligible for the PRGF, 28 have produced interim PRSPs and another 23 have produced a full PRSP. Almost all of the latter countries are currently engaged in PRGF-supported programs.1 There should now be sufficient evidence on how the approach is working in practice for an independent evaluation to be undertaken, so as to allow at a still relatively early stage for any adaptations of the approach that might appear desirable. In view of the relevance of the PRSP for the World Bank and of the joint IMF-WB nature of the many components of the process, the evaluation undertaken by the IMF's Independent Evaluation Office (IEO) will be conducted in parallel with and complemented by an evaluation of the PRSP process by the Operations Evaluation Department (OED) of the World Bank.2

The purpose of this paper is to lay out the main issues to be addressed by the evaluation. Section 1 provides some background information on the objectives of the PRGF and PRSP initiatives, the main findings of internal reviews undertaken to date, and key criticisms formulated by external stakeholders. Section 2 presents the main questions to be addressed by the evaluation. Section 3 discusses the methodological approaches envisaged. The latter two sections constitute the terms of reference for the evaluation.

A draft of this issues paper was circulated earlier to seek inputs from all interested parties on the questions to be addressed and the methodology to be followed. Comments or contributions on the substance of the issues covered are now invited from all interested parties and may be submitted at any time by e-mail addressed to ieo@imf.org.

I. BACKGROUND INFORMATION

Main objectives of the PRSP and PRGF initiatives

From 1987 to 1999, the bulk of IMF concessional lending was provided under the Enhanced Structural Adjustment Facility (ESAF). Internal and external evaluations conducted in 1997 and 1998, respectively, identified a number of problems that hindered the effectiveness of programs supported by this facility.3 In particular, the external evaluation highlighted the following as major problem areas: (i) lack of national ownership; (ii) weaknesses in the analytical and empirical bases of the social policy content of programs; and (iii) insufficient attention to tradeoffs involving policy choices that imply significantly different paths for growth and social welfare.

Initially, after considering the lessons from the two reviews, the Executive Board endorsed several recommendations of IMF staff and the external evaluators aimed at strengthening the effectiveness of ESAF-supported programs. Subsequently, it became clear that bolder and broader steps were needed, in particular to ensure that debt relief under the enhanced HIPC Initiative--which was also agreed upon at that time--had a significant poverty reduction impact.

A new approach to supporting reform and adjustment programs in low-income countries was therefore developed by the staffs of the World Bank and the IMF and subsequently endorsed by their respective Boards. The key element was to be the development by countries themselves of poverty reduction strategies, set out in a new vehicle: the Poverty Reduction Strategy Paper (PRSP). While stating clearly that the ultimate goal of the new policy was poverty reduction and the achievement of related international development goals, the initial policy framework did not spell out explicitly the objectives of PRSPs. It merely listed a number of process requirements, as well as expected outputs and intermediate outcomes.

  • Process:

The process was to be based on five key principles: (i) country driven, with broad-based participation of civil society; (ii) results-oriented; (iii) long-term perspective; and (iv) comprehensiveness (to address the multi-dimensional nature of poverty and the policies needed to reduce it); (v) partnership (with all stakeholders and with/among donors, particularly between the IMF and the World Bank). The World Bank and IMF support the process, with their operations oriented around the objectives and strategy set out in the PRSP. In approving this framework, IMF Executive Directors stressed that its implementation would require a closer collaboration between the staffs of the Bank and the Fund, and they welcomed the clear delineation of responsibilities proposed, with the IMF focusing on macroeconomic policy and related structural aspects. They emphasized that IMF staff should not be expected to become directly involved in areas that are primarily the responsibility of the Bank.

  • Primary outputs:
    • focused document laying out realistic but challenging poverty objectives, along with the policies needed to achieve them;
    • suitable basis for IMF and World Bank concessional lending;
    • key instrument for low-income countries' relationships with the donor community

  • Intermediate outcomes:
    • development of participatory processes for the setting of poverty reduction goals and the monitoring of the implementation of poverty reduction strategies
    • improved understanding of links between policies and poverty reduction outcomes
    • setting of priorities and design of public actions to achieve poverty goals
    • development of appropriate diagnostics and indicators of progress in poverty reduction
    • deepening of a shared vision across civil society regarding poverty reduction strategy
  • To complement this new tool, the IMF Executive Board endorsed the Managing Director's proposal to transform the ESAF into the PRGF. Prominent features that were to distinguish PRGF from ESAF-supported programs included:
  • Broad participation and greater country ownership;
  • Embedding the PRGF-supported program in a broader set of measures set out in an overall strategy for growth and poverty reduction;
  • Government budgets that are more pro-poor and pro-growth
  • Ensuring appropriate flexibility in fiscal targets;
  • More selective structural conditionality;
  • Emphasis on measures to improve public resource management/accountability; and
  • Social impact analysis of major macroeconomic adjustment and structural reforms.

IMF Executive Directors, upon endorsing this change, again stressed that the IMF did not have the panoply of expertise needed to assess the quality of social spending and related issues and, therefore, that the social components of IMF-supported programs should draw, to the fullest extent possible, on the work of the World Bank or other relevant institutions.

Main findings of internal reviews to date

Since the adoption of the PRSP/PRGF approach, progress reports and internal reviews of early experience were conducted by IMF and World Bank staff in 2001 and 2002.4 Given the limited amount of time elapsed and of full PRSPs available, these reviews focused on process issues and on experience with I-PRSPs. The 2002 reviews also drew upon extensive external consultations (discussed in the next Section). One of the purposes of the current evaluation will be to assess how much things have changed now that a larger number of full PRSPs is available. The main findings of these internal reviews are summarized here to provide background to the evaluation. At this stage, the IEO does not have its own view on these findings.

Regarding the PRSP5

  • broad agreement among all stakeholders involved on the validity of the objectives of the PRSP approach and its usefulness in tackling poverty;

  • general recognition that there have been improvements over time in PRSP process and content;

  • widespread agreement that the four key achievements of the PRSP approach to date are: (i) a stronger sense of ownership among most governments; (ii) a more open dialogue; (iii) a prominent place for poverty reduction in policy debates; and (iv) an acceptance by the donor community of the principles of the PRSP approach;

  • focus should now shift beyond process to content and implementation; hence the importance of improving knowledge of the linkages between policies and poverty outcomes; and of utilizing and building local capacity in core areas;

  • areas that will need continuing attention are: realism in the setting of goals and targets as well as in managing expectations; the importance of openness and transparency; the importance of flexibility, to allow for different country circumstances; the desirability of debate about alternative policy choices; and the importance of patience and perseverance with implementation;

  • problems identified in a significant number of cases included: the lack of involvement of specific groups, including Parliaments, in the participatory process; unsatisfactory modalities for donor involvement (apart from the Bretton Woods Institutions); lingering weaknesses in poverty data and analysis and in target setting; lack of adequate poverty and social impact analyses; lack of satisfactory monitoring indicators and/or institutional arrangements; insufficient prioritization and specificity of public actions identified in PRSPs; uneven coverage of cross-cutting issues such as gender, HIV/AIDS, good governance and rural development; persistent weaknesses in public expenditure management systems and accountability in general; uneven integration of the PRSP process into existing decision-making processes; and need for more opportunities for learning and disseminating lessons and good practices.

While still refraining from providing strict directives as to the content of PRSPs and the implementation of the process, these internal reviews nonetheless identified a number of "good practices" in all the areas of the PRSP process.6

Regarding the PRGF

  • In all three fundamental areas for change, namely program content, country ownership and the IMF's role, there has been substantial progress, but more can be done.

  • Areas of progress include consistency between the policy goals and macroeconomic frameworks of PRGF-supported programs and PRSPs, increased allocation of resources toward poverty reduction spending, streamlined structural conditionality and better coordination with the World Bank.

  • Areas leaving most scope for further improvements include the more systematic incorporation of poverty and social impact assessments (PSIAs) into program design, increased efforts towards improving the quality and management of public spending, and a perceived need to encourage deeper and broader discussion and analysis of the macroeconomic framework and the policies in PRGF-supported programs, with increasing focus on the analysis of the sources of growth.

Main external criticisms of the PRGF and PRSP initiatives7

While the PRSP process has been welcomed by many external commentators as putting increased emphasis on a country-driven, participatory process and on the role of poverty diagnostics in policy design, a number of concerns have also been raised. The following brief summary is not meant to be a comprehensive review of the various critiques, and the IEO does not yet have its own view on the issues raised, but the following broad sets of issues that have been raised are relevant for the evaluation.

Roles of the IMF and the World Bank

Some external critics have argued that the continuing dominance of the Bretton Woods Institutions in agenda setting and the choices of the reforms to be pursued, as evidenced by the need for a joint staff assessment (JSA) of the PRSP, "clearance" of this assessment by the Boards of the two institutions,8 and the HIPC link, remains a serious impediment to country ownership. Some characterize this process as little more than a relabelling of previous IMF and World Bank policy practices. Some observers have also argued that this dominance is an impediment to meaningful donor alignment since donor views are still not sufficiently taken into account.

Participatory process

Concerns also relate to the nature and extent of domestic participation. Some external critics argue that while stakeholders have been consulted, their influence on the choice, design, and implementation of policies has not increased markedly. This is partly related to the limited technical capacities of civil society and other groups that represent the poor but it also reflects, in their view, flaws in the design of the process, including an underestimation of the time and resources required to build the capacity to enable civil society to contribute effectively to policy formulation. In this context, some commentators see the need to limit the ambition and content of the PRSPs, which are currently too detailed and all encompassing, in order to avoid the risk of large divergences between realizable resources and the goals to be addressed.

Some critics also suggest that the level of civil society participation tends to decline or break down altogether as the PRSP process approaches key final stages. They argue that the final document tends to be substantially different in language from the penultimate one discussed with stakeholders. Thus, while the participatory process initially raises a number of expectations on the evolution of policies, the final version of the PRSP does not always reflect conclusions of the civil-society based consultation process. Other observers argue that the inability of the participatory process as implemented to address conflicting interests/policy tradeoffs means that it inevitably has a limited impact at the implementation stage. Such problems may be greatest for macro-economic policy formulation--where the IMF has primary responsibility.

Some external critics also suggest that the participatory process gives insufficient weight to the views of parliamentarians, the private sector, and, within civil society, those of rural dwellers--with adverse consequences for the comprehensiveness of the policy debate. In contrast, other observers have argued that the objectives set for the PRSP with regard to participation are unrealistic and vague, especially in the area of macroeconomic policies, and that in practice the PRSP process is often not well integrated with the framework for domestic macro-policy formulation, which can cause problems for the links between the PRSP and the PRGF (see below). They have also expressed concern that a participatory process that is largely separate from existing political institutions, including local governments, risks undermining these institutions.

Links between the PRSP, PRGF, and HIPC

Some external commentators see the need to delink the PRSP process from the HIPC framework arguing that the linkage has led to a rushed PRSP process.

With respect to the PRGF, commentators note that, for the reasons discussed above, the macroeconomic policies embedded in PRGF-supported programs are not always well integrated with PRSPs. In some cases, this reflects weaknesses in PRSPs themselves. It is thus difficult to tell at this stage the extent to which the PRSP process has informed the policy content of PRGF-supported programs. Some critics therefore see the policies supported by PRGF as traditional structural adjustment programs in another guise. More generally, there is a concern that by putting the emphasis on process changes, the PRSP/PRGF initiative overlooks the need for a reexamination of the traditional policy planks of structural adjustment programs based on a rigorous analysis of their impact on growth and poverty reduction.

Some civil society representatives and in-country NGO groups have also objected to the methods used to evaluate and monitor the impact of the PRSP/PRGF processes, which are often based on hastily collected and inadequate data, especially about poverty, and using methods on which there is little consensus. They highlight the need for independent and credible monitoring mechanisms, as part and parcel of the participatory framework.

One response made to these various criticisms is that the PRSP/PRGF initiative is meant to begin a fundamental change in approach, which cannot happen immediately, and that the degree of progress will depend on each country's particular institutions and starting conditions. Commentators who take this view argue that the key test is whether approaches in practice (including within the IMF and World Bank) are making significant progress in the right direction.

II. TERMS OF REFERENCE OF THE EVALUATION

The purpose of this evaluation is to assess the IMF's role in the PRSP process as well as the PRGF. Recognizing that a parallel evaluation of the PRSP is being undertaken by the World Bank's OED, the IEO will not assess important issues where the primary responsibility lies with the World Bank. The evaluation will seek to assess, in keeping with standard practice in the area of aid effectiveness, the following five aspects: efficiency, effectiveness, impact, relevance and sustainability. This approach effectively entails addressing three broad sets of questions:

  • Are the objectives of these initiatives suitably defined and is their design consistent with the achievement of cost-effective and sustainable progress?

  • Is the IMF delivering on its commitments embedded in the PRGF/PRSP and with what results?

  • What accounts for any shortcomings diagnosed (including systemic constraints) and what suggestions emerge from the evaluation about how they can be remedied as far as the IMF is concerned?

A few important considerations should be acknowledged from the outset, all dictated by a need to keep the scope of evaluation reasonably focused and consistent with the IEO's comparative advantage:

  • First, besides the PRGF, the evaluation will concentrate on the role of the IMF in the PRSP process and on those dimensions of the PRSP initiative that are directly relevant to the IMF's mandate. Clearly though, the ultimate success or failure of the PRSP and PRGF initiatives will depend to a considerable extent on factors that are beyond the scope of this evaluation, such as the role of other stakeholders, in particular country authorities and the World Bank, as well as the wider donor community, the global evolution of the volume and nature of ODA flows and broader developments in the world economy.

  • Second, the evaluation will focus on the experience of countries with full PRSPs, based on the assumption that this experience has the greatest potential for generating lessons of importance for the IMF's role in the future. This focus implies that start-up problems (such as the time pressure initially caused by the linkage of I-PRSPs with the HIPC initiative), owing to their one-off nature, will not be directly assessed. Nor will the evaluation examine directly issues related to countries encountering protracted difficulties either to enter into the PRSP process or to move from the I-PRSP stage to the full PRSP (including post conflict cases). While we recognize that these issues are important, we think that they would be better addressed in a separate study and at a later stage when it is easier to distinguish countries facing truly protracted difficulties from those just requiring a little more time to complete the process of drawing up a full PRSP.

  • Third, at this stage of the process, the evaluation will not be able to cast much light on final outcomes for growth and poverty.9 The focus will therefore be on inputs (the PRSP process and PRGF-supported programs' formulation), outputs (i.e., PRSP contents and PRGF-supported programs' design) and intermediate effects (i.e., institutional and policy changes).

In spite of these limitations, we expect this evaluation to have a substantial value-added over the internal review undertaken in late 2001/early 2002, for the following reasons. First, the sample of full PRSPs to be looked at will be expanded from 6 in the internal review to 23 in this evaluation, and six of those coincided with or preceded the adoption of a new PRGF arrangement (compared to none in the internal review sample). Second, most of the interactions with PRSP/PRGF stakeholders planned for this evaluation will take place in the second and third quarters of 2003, i.e. 1½ years after the consultations held as background to the internal review--a period during which significantly more experience has been gained by all stakeholders. Third, taking account of the particular comparative advantage of the IEO as an evaluation unit that is independent but also has access to internal documents and IMF staff, the evaluation aims to cast light not just on what has happened under these initiatives, but also on the why of any shortcomings in the approach (i.e. by examining constraints, incentives, and possible side effects that appear to be of general applicability) and on suggesting specific solutions.

The broad questions outlined above lead to a range of more specific issues involving the three dimensions just mentioned along with the overall architecture of the initiative. These issues are outlined below. Not all will be addressed in the same depth in the evaluation. Rather, we expect that a prioritization will emerge as the evaluation proceeds and as major issues are identified. For all issues, the evaluation will seek to investigate the causes of any shortcoming identified, and attempt to suggest remedies. To avoid undue repetition, this point has not been duplicated in every set of questions. The methodology envisaged to address these issues is discussed in Section 3.

One way of thinking about the scope of the evaluation would be in the context of a logical framework diagram (see White (1999) for an example of this approach). While we have not prepared a detailed mapping of this framework, Figure 1 gives a broad indication of the different stages. The evaluation will focus on only part of the overall framework, namely those elements of the first three stages (i.e., process, outputs, and effects) that directly concern the IMF's role.

Schematic Logical Framework of the Evaluation

A. Process issues

PRSP Process and IMF Inputs Into It

1. To what extent are PRSPs country-driven documents in the areas of the IMF's primary responsibility? Specifically, does IMF involvement strike the right balance between providing needed support and allowing the process to be genuinely driven by country authorities? Do the methodological tools provided by the IMF in its areas of expertise (e.g. the relevant sections of the PRSP Sourcebook) provide adequate guidance and allow sufficient scope for country-driven choices between alternative policies? Is IMF policy advice (including in the Sourcebook) based on concrete evidence of the links between policies and growth/poverty reduction?10

2. What is the nature of the IMF's contribution to the participatory process, in particular, in terms of : (i) provision of information to the various stakeholder groups? (ii) interaction with the donor community and civil society? Is the process by which the macroeconomic framework of PRSPs is formulated participatory and, if not, what are the practical obstacles to a participatory approach on such issues?

3. In the areas within its mandate, does the IMF adequately assess the ability of countries to undertake an effective PRSP process, including poverty and social impact analyses and what steps does it take--either directly or in coordination with other providers of technical assistance--to help countries strengthen their capacity for domestic policy formulation, implementation and monitoring?11

PRGF process

4. Has the process of program formulation changed under the PRGF with respect to: (i) the IMF's own internal procedures (including the cycle of missions); (ii) the scope, depth, and transparency of IMF staff interactions with various stakeholders (including line ministries, parliaments, private sector, the poor, CSOs, and donors); and (iii) the analytical basis of the macroeconomic framework? In particular, how are poverty/social impact assessments (PSIA) incorporated into the design of the policy framework and what are the obstacles to greater use of PSIA? Where the PRGF program was preceded by a PRSP, was it clearly derived from the former?

B. Immediate outputs

PRSP Contents

5. In the areas of the IMF's primary responsibility, to what extent do PRSPs have the general components expected from them, including an analysis of the impact on growth and income distribution of past macroeconomic and related structural policies, a robust macroeconomic framework, along with related priority public actions and a credible financing plan ?

6. Do PRSPs improve the clarity of policy choices in the macroeconomic and budgetary areas by providing realistic goals and projections, an analysis of the costs and benefits of alternative policy options and the tradeoffs between them, and a suitable prioritized plan of actions?

7. In the areas of the IMF's primary responsibility, do PRSPs provide a framework suitably linked to implementation, including monitoring arrangements and feedback mechanisms? While taking into account the expanded time horizon over which their results will be seen, are PRSPs also sufficiently connected to shorter-term policy-making instruments (in particular the budget and the medium-term expenditure framework), so as to provide a framework within which a PRGF-supported program can be derived?

Joint Staff Assessment

8. Are JSAs clear and candid in their assessments of suitability of PRSPs as a basis for IMF concessional lending ? How are inputs from other stakeholders handled in making such assessments?

9. What is the assessment of the usefulness of JSAs by: (i) country authorities? (ii) the Executive Board? (iii) other PRSP stakeholders ? To what extent do JSAs play a signaling role of their own to aid donors?

PRGF-Supported Program Design/ content

10. To what extent do PRGF-supported programs incorporate the key features intended to differentiate them from ESAF-supported ones? In particular, to what extent are these programs aligned on PRSPs, in terms of their macroeconomic framework, targets and policy objectives, linkage and support to budgetary processes, and use of monitoring indicators? Is program design influenced by PSIAs?

11. Is streamlined structural conditionality suitably prioritized in PRGF-supported programs to those actions judged critical to successful implementation of macroeconomic policy? Is the World Bank's conditionality taken into account in that process? What has happened to the aggregate level of conditionality and how are gaps and overlaps between the agendas and priorities of both institutions handled?

12. Are program targets and underlying projections realistic (and consistent with the World Bank's assumptions)? Are they set in sufficiently flexible terms to allow a timely adaption in case of unexpected shocks (be they positive or negative)?

13. How are external financing requirements derived in PRGF-supported programs and how are these estimates linked to the longer-term objectives set out in PRSPs, including debt sustainability? How is the tradeoff between accommodating the higher spending plans embedded in PRSPs and preserving/progressing toward macroeconomic stability being handled in PRGF-supported programs?

C. Intermediate effects and outcomes12

Institutional Effects13

14. In macroeconomic and related areas, are PRSPs contributing to substantial improvements in domestic policy formulation and implementation? For example, are they contributing to improvements in the analytical bases (including relationships between macroeconomic and related structural policies and poverty outcomes, monitoring tools etc.) for policy debates and economic policy decisions? In particular, do the monitoring and evaluation arrangements provided for in PRSPs effectively feed back to policy formulation in these areas?

15. Are PRSPs linked to countries' budgets and public expenditure management frameworks on a permanent, institutionalized basis, as an integral part of domestic policy formulation?

16. Have the various institutional arrangements surrounding PRSPs brought about a closer, more effective collaboration between the IMF and the World Bank (including in their analytical work and setting of priorities for their respective country operations)? What has been the performance of the Joint Implementation Committee (JIC) in that respect?

Impact on Economic Policy and Preliminary Evidence on Outcomes14

17. How has the PRSP process affected the nature and scope of the policy debate, within the country and between the country and the IMF, as well as policy outcomes on key issues that are in the IMF's primary area of responsibility? How has PRGF-supported program implementation been affected? For example, what changes in policies have resulted from a more country-driven process?

18. What can be said of the evolution of expenditure composition parameters vis-à-vis suitable comparator group(s), looking at both budget allocations and actual spending? What has been the impact on tax structure/budgetary process and public expenditure management frameworks? What do the intermediate indicators used to monitor progress towards the MDGs suggest (focusing on those indicators that are most relevant to the IMF's role)?

19. While it is too early for a systematic assessment of the impact of the PRSP/PRGF on final objectives, the evaluation will collect what evidence is available on the evolution of various program outcomes (e.g. GDP growth and key macro indicators) compared, for example, with earlier ESAF-supported programs.

D. Questions of architecture and design of the PRSP/PRGF initiatives

The evaluation will also address several issues involving the design and architecture of the PRSP/PRGF approach, in order to assess its relevance and sustainability with respect to the ultimate objectives. Taking into account the parallel evaluation by the OED the goal would not be to examine all questions involving the overall architecture, but to address some specific questions concerning the IMF's role:

20. Does IMF staff receive sufficiently clear guidance from both internal and public policy documents regarding: (i) their involvement in the PRSP process? (ii) the formulation of PRGF-supported programs? (iii) the depth and modalities of their collaboration with World Bank staff? Are the supporting institutional arrangements and the incentives faced by staff consistent with a uniform and effective delivery of the promises embedded in the PRSP/PRGF approach as far as the IMF is concerned?

21. Are the IFI's expectations regarding the PRSP spelled out clearly enough for PRSPs to provide both an effective, operational strategic policy framework for poverty reduction and a vehicle for effective donor coordination ?

22. Are the function of the JSA process and the IMF's role in it appropriate? In particular, does the need to gain JSA endorsement lead to a form of self-censorship on the part of countries? Should changes be envisaged to the JSAs, e.g., by making them more inclusive of other stakeholders' views, so as to make judgments on PRSPs less of a "Washington thing" while providing a frank and clear assessment?

23. What actions could/should the IMF take, within its mandate and area of expertise (e.g., as part of its broader surveillance role), to link the PRSP objectives with policies in the advanced economies that would improve the external economic environment faced by the PRSP countries?

24. What does the evaluation suggest regarding what should be the long-run role of the IMF--and the PRGF in particular--in relation to the PRSP process in countries: (i) with no immediate short term balance of payments need? and (ii) with clear short-term balance of payments needs but where the poverty reduction strategy or its implementation is judged inadequate? In those cases, what suggestions emerge from the evaluation about what would be a suitable strategy for IMF involvement and what functions should the IMF perform in those cases, based on its comparative advantage?

III. ENVISAGED METHODOLOGICAL APPROACH

In order for the IEO to concentrate on its own areas of expertise and avoid duplicating work undertaken by others, while at the same time letting its own judgments be informed by external analysis, the following three-pronged approach is envisaged:

  • The IEO's original contribution would focus on the areas where it has a comparative advantage, namely IMF operations in the relevant countries (e.g., IMF inputs to the PRSP process, PRGF-supported programs, surveillance, and technical assistance activities) and internal IMF processes.

  • Aspects of shared interest between the IMF and the World Bank will be reviewed in collaboration with OED.14 IEO and OED will undertake some evaluative work jointly (see below) and will also share the findings of any work done independently. However, each evaluation unit will retain its independence of judgment on the issues at stake.

  • Many of the cross cutting issues related to the PRSP process have also been studied by various research and development institutions in both the low income countries themselves and in donor countries, including through case studies. While final judgments from the evaluation will remain the exclusive responsibility of the IEO, this material will be systematically reviewed and, where appropriate, drawn upon by the IEO to inform its own analyses. A survey of this literature will be part of the evaluation.15 Authors of such evaluative or analytical material related to the issues discussed above are invited to bring their work to the attention of the IEO.

With respect to IEO's original contributions, the evaluation will follow a three-tiered approach consisting of: (i) in-depth cases studies; (ii) cross-cutting analyses of the full sample of countries with a full PRSP; (iii) control-type analyses. The case studies will contribute to answering all the questions, while cross country or institution-wide analyses will be undertaken to test how generalizable findings from the case studies are, and to provide additional evidence on each set of issues. The matrix presented in Annex II provides a summary mapping of key tasks and methodologies envisaged to tackle them.

The case studies would involve both deskwork (including reviews of relevant IMF documentation, both published and unpublished, as well as reviews of external evaluative evidence) and field work. They would seek the views of all relevant stakeholders (country authorities including line ministries and parliaments, civil society, donors and IMF and World Bank staff) through a combination of interviews and surveys. They would place particular emphasis on analyzing changes in the nature of the policy debate by following through a few critical policy issues in each country and assessing how their treatment has evolved under the PRSP/PRGF initiative. Six case studies will be undertaken by the IEO, chosen so as to reflect diverse regional experiences and economic performance, and to offer a combination of recent and mature PRSPs and PRGF-supported programs, and of HIPC and non HIPC countries. The specific countries envisaged as case studies are the following:16
(i)  Tanzania*
(ii)  Nicaragua*
(iii)  Mozambique*
(iv)  Tajikistan
(v)  Vietnam
(vi)  Guinea*
Case studies would include field trips and consultations with local stakeholders expected to take place through July 2003. The timing of the country visits will be announced in advance on the IEO website and will also be communicated to a broad range of stakeholders. Submission to the IEO of external inputs on PRSP/PRGF experiences in the country case studies would be welcome at any time.17

Cross-country analyses would be undertaken on the full sample of countries with a full PRSP as of December 2002. Their purpose would be to test on a broader scale the relevance of the findings made in the case studies as well as to bring out other messages of general significance. These analyses would rely upon systematic reviews of IMF staff reports on surveillance and lending arrangements, and IMF databases on program design/ implementation and economic outcomes; surveys of country authorities, IMF staff, donors and other stakeholders (including the civil society networks that took part in the PRSP participatory process). Care would be taken, in the conduct of these analysis, to distinguish within the sample between different vintages of PRSPs and PRGF arrangements, in order to identify any learning effects.

Control-type analyses are also envisaged, essentially of two sorts: first, broad-based statistical analyses of the full sample of PRGF-eligible countries, in order to compare the performance of program versus non program countries, PRSP versus non PRSP countries and ESAF versus PRGF-programs.18 Second, tentatively, a desk review of a small number of PRGF-eligible countries that chose not to engage in the PRSP process may be undertaken, in order to highlight any differences in the process of policy formulation and in actual policy choices and outcomes compared to PRSP countries. There are obvious problems of sample-selection bias in any choice of such a limited control group, but it may highlight some useful qualitative messages.

The modalities for cooperation between the IEO and the OED on their respective evaluations, will be guided by the following broad principles: (i) each institution will produce its own evaluation report in keeping with its own procedures, and these reports will be presented to their respective Boards; and (ii) the inputs to the evaluations will be organized in such a way as to avoid duplication of demands on member countries (e.g. with respect to country case studies). Joint case studies are expected to involve, inter alia, joint country missions and a common stakeholder survey, as well as single country reports.

The timetable of the evaluation contemplates submission of the final report to the IMF's Executive Board in the first quarter of 2004.

Back to top

Bibliography

Action Aid, 2002. Inclusive Circles Lost in Exclusive Cycles, Contribution to the First Global Poverty Reduction Strategies Comprehensive Review, mimeo, January.

Bevan, David and Christopher Adam, 2001. Poverty Reduction Strategies and the Macroeconomic Policy Framework. Mimeo- Department of Economics, University of Oxford, January.

Bretton Woods Project and World Vision, 2002. Blinding With Science or Encouraging Debate? How World Bank Analysis Determines PRSP Policies. London, September.

Campbell, Bonnie and Bruno Losch, 2002. Les Pauvres, Bénéficiaires ou otages des Stratégies de Réduction de la Pauvreté? Politique Africaine, October.

Catholic Relief services, 2001. Review of the Poverty Reduction Strategy Paper Initiative Based upon the Experiences and Comments of CRS Partners in Bolivia, Honduras, Zambia and Cameroon, December. Baltimore; Catholic Relief Services World Headquarters.

Christian Aid, 2002. Quality Participation in Poverty Reduction Strategies: Experiences From Malawi, Bolivia and Rwanda.

Christian Aid, 2001. Ignoring the experts: Poor People's Exclusion From Poverty Reduction Strategies. October.

Cling, Jean-Pierre, et al., 2002. The PRSP Initiative: Old Wine in New Bottles? Paper presented to ABCDE-Europe conference 2002.

European Network on Debt and Development (EURODAD), 2001. Many Dollars, Any Change? Part 1: The Changing Nature of Development Cooperation: Building Ownership. Brussels, October.

______, 2001b, Many Dollars, Any Change? Part II: Have Structural Adjustment Policies Failed the Poor? Brussels, October.

Gupta, Sanjeev, et al. 2002. Is the PRGF Living Up to Expectations? An Assessment of Program Design. IMF Occasional Paper No. 216.

Hubbard, M. 2001. Attacking Poverty, a Strategic Dilemma for the World Bank. Journal of International Development, Vol. 13, pp. 293-298.

International Monetary Fund, 1997. The ESAF at Ten Years: Economic Adjustment and Reform in Low-Income Countries.

______, 1998 a. External Evaluation of the ESAF: Report by a Group of Independent Experts.

______, 1998 b. Review of the IMF's Approach to Social Issues and Policies.

______, 1999 a. Economic Adjustment and Reform in Low-Income Countries. Studies by the Staff of the International Monetary Fund (Bredenkamp and Schadler, editors)

______, 1999 b. Poverty Reduction and Growth Facility: Operational Issues.

______, 2000. Key Features of IMF Poverty Reduction and Growth Facility Supported Programs.

______, 2001. IMF Lending to Poor Countries: How does the PRGF Differ From the ESAF?

______, 2002 a. Review of the Key Features of the Poverty Reduction and Growth Facility--Staff Analyses. (February)

______, 2002 b. Review of the Poverty Reduction and Growth Facility: Issues and Options. (February)

International Monetary Fund and World Bank, 1999 a. Heavily Indebted Poor Countries (HIPC) Initiative--Strengthening the Link between Debt Relief and Poverty Reduction

______, 1999 b. Poverty Reduction Strategy Papers--Operational Issues. (December)

______, 2000. Progress Report on Poverty Reduction Strategy Reports.

______, 2001. Poverty Reduction Strategy Papers--Progress in Implementation. (April)

______, 2002 a. External Comments and Contributions on the Joint Bank/Fund Staff Review of the PRSP Approach. Volume 1--Bilateral Agencies and Multilateral Institutions. (February)

______, 2002 b. External Comments and Contributions on the Joint Bank/Fund Staff Review of the PRSP Approach. Volume 2--Civil Society and Individual Contributions. (February)

______, 2002 c. Synopsis of External Comments and Contributions on the Joint Bank/Fund Staff Review of the PRSP Approach. (February)

______, 2002 d. Review of the Poverty Reduction Strategy Paper (PRSP) Approach: Main Findings. (February)

______, 2002 e. Review of the Poverty Reduction Strategy Paper (PRSP) Approach: Early Experience with Interim PRSPs and Full PRSPs. (February)

Klasen S., 2001, In Search of the Holy Grail: How to Achieve Pro-Poor Growth?, Background Paper to SPA Task Team: Growth and Equity. Eschborn: GTZ.

Maxwell S., 2001, Innovative and important, yes, but also instrumental and incomplete: the treatment of redistribution in the New Poverty Agenda, Journal of International Development, Vol. 13, pp. 331-341.

Mosley, P., 2001, Attacking Poverty and the Post-Washington Consensus, Journal of International Development, Vol. 13, pp. 307-313.

Oxfam International, 2001. Are PRSPs Working? Contribution to the World Bank/IMF PRSP review process. Washington DC.

Structural Adjustment Participatory Review International Network (SAPRIN), 2002. The Policy Roots of Economic Crisis and Poverty. A Multi-Country Participatory Assessment of Structural Adjustment.

United Nations Conference on Trade and Development (UNCTAD), 2002. Economic Development in Africa. From Adjustment to Poverty Reduction; What Is New? New York and Geneva: United Nations.

United States General Accounting Office (GAO), 2001. Few Changes Evident In Design of New Lending Program for Poor Countries. Report No. GAO-01-581. Washington DC.

White, H. (1999), Dollars, Dialogue and Development, Stockholm, SIDA Evaluation Report.

World Bank, 1999. Building Poverty Reduction Strategies in Developing Countries

List of Abbreviations

BWI Bretton Woods Institutions
CSO Civil Society Organization
ESAF Enhanced Structural Adjustment Facility
HIPC Heavily Indebted Poor Countries
IDA International Development Association
IEO Independent Evaluation Office
IMF International Monetary Fund
I-PRSP Interim Poverty Reduction Strategy Paper
JSA Joint Staff Assessment
MDGs Millennium Development Goals
NGO Non Governmental Organizations
OED Operations Evaluation Department
PRGF Poverty Reduction and Growth Facility
PRSP Poverty Reduction Strategy Paper
PSIA Poverty and Social Impact Analysis
WB World Bank

ANNEX I

PRSP and PRGF Status of PRGF-eligible countries as of December 2002.
  PRSP date1 PRGF approval date3 PRGF status3 HIPC status4
A--Countries with full PRSP5 (21)
Albania June-02 June-02 Board approval NE
Bolivia June-01 September-98 off track CP
Burkina Faso July-00 September-99 5th rev completed CP
Ethiopia September-02 March-01 3rd rev completed DP
Gambia, The July-02 July-02 Board approval DP
Guinea July-02 May-01 1st rev completed DP
Guyana September-02 September-02 Board approval DP
Honduras October-01 March-99 3rd rev completed DP
Malawi August-02 December-00 off track DP
Mauritania January-01 July-99 5th rev completed CP
Mozambique September-01 June-99 4th rev completed CP
Nicaragua September-01 March-98 off track DP
Niger February-02 December-00 3rd rev completed DP
Rwanda July-02 July-02 Board approval DP
Senegal Oct-Dec 02 April-98 completed DP
Tajikistan Oct-Dec 02 Dec-2002 Board approval NE
Tanzania December-00 March-00 4th rev completed CP
Uganda May-00 November-97 completed CP
Vietnam June-02 April-01 2nd rev completed NE
Yemen July-02 SMP NE
Zambia May-02 March-99 4th rev completed DP
B--Countries with PRSP pending6 (29)
Armenia Jan-Mar 03 May-01 2nd rev completed NE
Azerbaijan Jan-Mar 03 July-01 1st rev completed NE
Benin Jan-Mar 03 July-00 3rd rev completed DP
Bosnia and Hrz Apr-Jun 03 SBA NE
Cambodia* Jan-Mar 03 October-99 5th rev completed NE
Cameroon Jan-Mar 03 December-00 2nd rev completed DP
Cape Verde Apr-Jun 03 April-02 Board approval NE
Central Afr. Rep. Jul-Sep 03 no PRGF E
Chad Jan-Mar 03 January-00 3rd rev completed DP
Comoros Jul-Sep 03 SMP E
Congo Dem. Rep. unspecified Jun-02 Board approval E
Cote d'Ivoire Jan-Mar 03 March-02 Board approval E
Djibouti Apr-Jun 03 October-99 2nd rev completed NE
Dominica Apr-Jun 03 SBA NE
Georgia Jan-Mar 03 January-01 2nd rev completed NE
Ghana Jan-Mar 03 May-99 4th rev completed DP
Guinea Bissau Jan-Mar 03 December-00 off track DP
Haiti Jul-Sep 03 no PRGF NE
Kenya Jan-Mar 03 August-00 off track NE
Kyrgyz Republic* Jan-Mar 03 December-01 1st rev completed NE
Lao, PDR Jan-Mar 03 April-01 2nd rev completed NE
Lesotho Jan-Mar 03 March-01 2nd rev completed NE
Macedonia FYR unspecified December-00 off track NE
Madagascar Jan-Mar 03 March-01 1st rev completed DP
Mali Jan-Mar 03 August-99 4th rev completed DP
Moldova Jan-Mar 03 December-00 1st rev completed NE
Mongolia Jan-Mar 03 September-01 off track NE
Nepal Jan-Mar 03 no PRGF NE
Pakistan Jan-Mar 03 December-01 2nd rev completed NE
Sao Tome and Pr. Jan-Mar 03 April-00 off track DP
Sierra Leone unspecified September-01 1st rev completed DP
Sri Lanka Jan-Mar 03 SBA NE
C--Countries not yet at I-PRSP stage (27)
Afghanistan       NE
Angola       NE
Bangladesh Jan-Mar 03     NE
Bhutan       NE
Burundi Apr-Jun 03     E
Congo, Republic of       E
East Timor Apr-Jun 03     NE
Eritrea       NE
Grenada       NE
India 2/       NE
Kiribati       NE
Liberia       E
Maldives       NE
Myanmar       E
Nigeria Jan-Mar 03     NE
Samoa       NE
Solomon Islands       NE
Somalia       E
St Lucia       NE
St Vincent & Grenadin.       NE
Sudan       E
Togo Jan-Mar 03     E
Tonga       NE
Vanuatu       NE
Zimbabwe       NE
1Actual date of review by IMF Board for group A countries; expected date of review by the Board for group B countries; expected date of endorsement of I-PRSP for group C countries.
2India is eligible to the PRGF but the authorities have indicated that they do not intend to avail themselves of resources under this facility.
3SBA: stand-by arrangement
  SMP: staff monitored program
  off track: indicates absence of review since at least 12 months.
4CP: completion point reached
  DP: decision point reached
  E : eligible but still to be considered
  NE: not eligible to HIPC debt relief.
5Shaded cells indicate countries chosen as case studies for the IEO evaluation.
6Countries marked with an asterisk had completed their PRSP as of 12/31/2002 and transmitted it to the IMF but Board review was still pending.

ANNEX II

Task (review of...)

Methodology*

 

Case study

Data-
base analysis

Review of policy guidelines**

Review of staff country reports

Survey (S)/ interviews (I)

Literature review

S I
IMF inputs and processes
- IMF inputs and
  contributions to
  process
- IMF policy advice
- IMF institutional
  arrangements
- External perception
  of IMF contributions


X



X
X

X


X1



X1


X



X
X
 

X




X


X






X
 
Joint Staff Assessments
- Consistency with
  guidelines
- Value-added and
  focus of JSAs


X


X
   

X




X




X




X
PRSPs
- Consistency of
  contents with
  "standards"
- "Washington driven"
  nature
- Operational value
  for domestic policy
  making

X



X

X
   
X



X

X




X

X
 
X



X

X
PRGF-supported programs
- Observance of "7
  key features"
- Realism of program  
  design
- Effectiveness of
  WB-IMF collaboration


X

X

X


X2

X2,3

X2
 

X

X

X


X




X
 

X
PRSP/PRGF outcomes & Impacts
- Program
  implementation
- Changes in economic
  policies
- Changes in
  macroeconomic
  performance
- Institutional
  changes (including.
  monitoring/feedback)


X

X

X



X


X2

X3

X3
 





X



X









X









X




X

X


X
*The case study methodology will include interviews with country specific stakeholders, interviews with IMF and World Bank staff, review of unpublished IMF material and database analysis. Other elements of the methodology matrix refer to the key techniques that will be applied to the broader sample of countries. Techniques not checked may be used as well, but it is not expected at this stage that they would yield major insights.
**Including published and unpublished policy documents and guidance notes to IMF staff.
1Primarily budget reporting system database.
2Primarily MONA and ad hoc PDR-generated databases.
3Primarily WEO and similar macroeconomic databases.


1Details on PRSP and PRGF status of PRGF-eligible countries as of 12/31/2002 are provided in the Annex.
2Information on the evaluation to be undertaken by OED is available online at this address: www.worldbank.org/oed/prsp.
3See IMF, 1998 a; IMF, 1997 and IMF, 1999 a.
4See IMF 2002(a) and 2002(b) and IMF and World Bank, 2001; 2002(d) and 2002(d).
5In the interest of completeness, this summary covers all the main findings of the internal reviews, including those concerning areas that are primarily the responsibility of the World Bank. It is not intended that the IEO evaluation would cover these latter areas.
6See IMF and World Bank, 2002 d) and e) and The PRSP Sourcebook, available online on the World Bank's internet site at http://www.worldbank.org/poverty/strategies/sourctoc.htm.
7This summary of issues raised by the external commentators draws upon the Synopsis of External Comments and Contributions on the Joint Bank/Fund Staff Review of the PRSP Approach, IMF (2002c) as well as on a number of additional commentaries cited in the attached bibliography. A review of the external literature on the PRSP/PRGF process will be part of the evaluation.
8Technically, the two Executive Boards do not clear the PRSPs themselves, but make a judgment that they constitute a suitable basis for their own lending programs.
9While, in a sense, this is a serious limitation, it should also be acknowledged that conclusive data on final outcomes is unlikely to be available for a number of years, i.e. at a time when any substantial changes in the design or implementation of these initiatives that might prove necessary would be long overdue.
10In some areas, there is a broad consensus, based on concrete evidence, that certain policies have an adverse effect on growth (e.g. highly distorted multiple exchange rates or rapid inflation)-thus leaving little space for considering a wide variety of such policy options. In many other areas, however, the evidence is less conclusive, and one would therefore expect the IMF to show a greater openness to country-driven alternatives to what it views as its "first best" prescriptions. The evaluation will assess whether this is the case.
11The PRGF/PRSP evaluation will not include a comprehensive assessment of the IMF's technical assistance (TA) activities in the countries concerned. (An evaluation of IMF TA will be undertaken as part of the FY2004 work program of the IEO. See www.imf.org/ieo for details). However, the current project will examine how the IMF assesses a country's capacity to undertake the IMF-related parts of the PRSP process and how this assessment is linked to a strategy to help the countries concerned improve their capacity to effectively drive the process over time and implement the PRSP.
12To address the issues raised under this heading, the evaluation will focus on post full PRSP developments, including where relevant a review of PRSP progress reports or amendments to the initial PRSP.
13In addressing some of these issues, in particular issue #16 and parts of issue #18, the IEO will also draw substantially on the parallel evaluation work done by the OED.
14This would include (but is not limited to) JSAs, Bank-Fund collaboration issues, aggregate conditionality and the analytical bases of the macroeconomic and related policy elements of PRSPs.
15The evaluation will also draw substantively on the external comments and contributions made at the time of the Bank/Fund staff review of the PRSP (see IMF and World Bank, 2002a and 2002b).
16Case studies marked with an asterisk are expected to be undertaken jointly with OED. In addition to these six "new" case studies, the evaluation would draw on the findings of the case studies of Pakistan and Senegal undertaken in the context of the recently completed IEO evaluation of The Prolonged Use of IMF Resources.
17The expected timing of the country visits will also be brought to the attention of those civil society groups who participated in the PRSP process in each country, in order to allow time for any submissions to be prepared. However, submissions can be made at any time during the evaluation.
18In addition to the usual methodological problems associated with the endogeneity of participation in the PRGF/PRSP, it must also be recognized that, in practice, the transition from ESAF-supported to PRGF-supported programs has been a gradual, rather than a clear-cut one. Many of the early PRGF-supported programs were relabeled ESAF programs, with a gradual introduction of the elements specific to the new approach.