## External Evaluation of the Independent Evaluation Office (Content unit: _010113) ## Source details **Canonical URL:** [External Evaluation of the Independent Evaluation Office (Content unit: _010113)](https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/np/pp/eng/2013/_010113.pdf) ## Other formats - [Markdown version](/-/media/websites/imf/imported-full-text-pdf/external/np/pp/eng/2013/_010113.pdf.md) - [Structured JSON version](/-/media/websites/imf/imported-full-text-pdf/external/np/pp/eng/2013/_010113.pdf.json) --- ### Purpose, scope, and methodology - Executive Board requested an external review of the IEO; second external evaluation in the IEO’s twelve year history (first = “Lissakers Report” presented in 2006). - Focus: IEO activities since 2006 and evaluation of IEO performance against Panel terms of reference. - Consultations and inputs: - Interviews with current and three former Managing Directors; current and two former Deputy Managing Directors; all members of the Executive Board; former Chair of the Evaluation Committee; Chair and a member of Lissakers Panel; current and two former IEO Directors and IEO Deputy Director; Directors of several IMF Departments; Director of the G-24; Head of the World Bank’s Independent Evaluation Group. - Three external expert consultations (Washington D.C., London, Buenos Aires); two civil society consultations (Washington D.C., London). - Conference contributions and IEO-organized conference on first decade of work; Ruben Lamdany and Hali Edison (eds.), Independent Evaluation at the IMF: The First Decade, 2012. - IMF staff survey with questions largely coinciding with Lissakers Report survey plus new/reformulated questions. - Panel had full access to IMF and IEO information deemed relevant. ### Report structure - Seven parts: Part 1 Introduction; Part 2 Overall evaluation; Part 3 Review of Lissakers follow-up; Part 4 Independence and mandates; Part 5 Detailed assessment of evaluation cycle; Part 6 Staffing and budgetary issues; Part 7 Summary of main recommendations. ### Terms of reference highlights - Panel to assess IEO effectiveness on: (i) appropriateness of evaluation topics; (ii) IEO independence; (iii) cost-effectiveness and operations; (iv) appropriateness and adequacy of evaluation process including how IEO recommendations are endorsed by the Board and implemented. --- ### Overall evaluation — mandates, strengths, and areas for improvement - Mandates at inception: - Enhance the learning culture of the Fund. - Support institutional governance and oversight. - Strengthen the Fund’s external credibility. - Promote greater understanding of the work of the Fund throughout the membership. - Panel view on mandates: - First two mandates are most important; third largely a result of fulfilling the first two. - Panel recommends dropping the fourth mandate. - Broad assessment: - IEO successful in improving governance, transparency, and promoting a learning culture. - Widely considered the most independent evaluation office among international financial institutions. - Strong consensus that IEO strengthened IMF external credibility. - Strengths: - Generally good topic selection and report quality; focus on longer-term cross-cutting issues. - Use of solid evidence from wide sources, including governments. - Full access to internal information. - Analyses provide alternative views sometimes accepted and leading to changes. - Context and tensions: - IMF operates in “risky and messy business”; pressures from influential members noted. - Some interviewees argued IEO did not always recognize operational constraints; Panel judges transparency and openness to evaluation have contributed to, not damaged, external credibility. - Areas for improvement (Panel): - Follow-up process: lacks strong Board ownership, involves conflict of interest for Management, poorly suited for broader substantive recommendations, bureaucratic; recommendations tend to be watered down at each stage — Panel recommends overhauling follow-up and involving IEO in follow-up without weakening Board and Management responsibilities. - Attitudes within Management and Staff: defensive/antagonistic attitudes need addressing to derive full benefit. - Topic selection and scope: clarify “current operations”; focus recommendations more on policy and culture rather than processes; revisit key issues previously evaluated. - Awareness and in-reach: Staff survey shows awareness surprisingly low and lower than 2006; propose measures to increase in-reach and outreach. - Risk of routinization: avoid becoming “bureaucratized, routinized and marginalized”; Executive Board must use IEO for oversight; IEO should engage more with IMFC. - Resource implications: - Changes in staffing rules to facilitate mobility to/from IEO and a small budget increase. --- ### The Lissakers Report — summary and Panel assessment - Lissakers (2006) assessed first five years; concluded independent evaluation unit needed; safeguards required to ensure independence. - Panel findings relative to Lissakers: - IEO remains highly relevant and independent. - Staff mix more skewed to “outsiders”; Panel finds mix of insiders and outsiders appropriate. - Access to information and safeguards concerns from Lissakers appear resolved. - Topic selection: agree IEO should evaluate controversial issues and should not evaluate “current operations” without clarifying definition. - Report quality: IEO generally high quality; now regularly uses external expertise; recommendations since 2006 became more process-oriented — Panel urges outcome-focused recommendations. - Follow-up and impact: formal follow-up mechanism introduced post-Lissakers but panel judges it not working properly and needing revamp. - Role in Board oversight: Board uses IEO less than it should; Panel proposes concrete steps for more proactive governance role by Board. - Outreach: IEO has improved external outreach; however in-reach inside IMF has become weaker and needs improvement. --- ### Independence and mandates (detailed) - Independence assessment: - IEO retains actual and perceived independence across staffing, topic selection, and conduct of evaluations. - Evidence: independent capacity to conduct evaluations; independence in staff selection, external consultants and advisory panelists; full access to required information; freedom to comment on any Fund policy/process. - Perceptions and survey: - 2012 staff survey: 59 percent considered IEO either independent or very independent (2006 = 65 percent). Only one in ten respondents considered IEO not independent. - Exercise of independence and relationships: - IEO maintained arms-length relationship with Board, Management, Staff; as IEO matured relations became more tense/formalized — trend judged detrimental and should be reversed to restore greater collaboration. - Tensions identified: prioritization of recommendations; follow-up process; Management signaling relevance to Staff; Board ownership — need to better codify processes and restore informal links. - Mandates and policy recommendation: - Two key mandates identified by Panel: - Enhance the learning culture of the Fund. - Support institutional governance and oversight as instrument for Board. - Third function (strengthening external credibility) is a result of first two. - Fourth mandate (promote greater understanding throughout membership): Panel recommends dropping it. --- ### Detailed assessment of the evaluation cycle A. Nature of reports and topic selection - Outputs and patterns: - Over twelve years IEO presented nineteen reports — slightly less than two per year. - Most reports focus on long-term cross-cutting IMF issues; two country reports (Argentina, Jordan) were completed during first five years (outside current evaluation period). - Topic selection process: - Director chooses topics after open consultation with country authorities, Board, Management, Staff, external stakeholders; suggested topics posted on IEO website for comment. - Panel considers Director should retain full freedom to choose issues. - Definition of “current operations”: - Existing rules constrain topics to avoid interfering in ‘current operations’; Panel concerned by lack of clear definition. - Recommendation: Board should define ‘current operations’ narrowly as current lending programs, allowing IEO review of recent/current activities that do not involve lending programs and past lending programs even if countries have new programs. - Distinction urged between ‘current operations’, ‘current issues’, and ‘recurrent activities’. - Number and format: - Two full-scale evaluation topics per year appropriate given Office size. - Panel does not favor expanding to many shorter reports to avoid jeopardizing rigor. - Country-specific evaluations: - Recommendation: evaluate country programs more systematically but focus on strategic/generic issues; periodic evaluations typically every two years of a sample of EPAs and EPEs. - If IMF were to undertake ex-post evaluations of all programs, IEO would evaluate a sample of all IMF programs. - Access and report length: - Full access to needed information; report length appropriate with technical aspects in appendices. - Executive summaries could be improved to better convey main analysis and recommendations. B. Quality of reports and nature of recommendations - Overall quality: - Evaluation of eleven reports since 2006: generally high quality. - Methodology: - Use of Advisory Groups and workshops welcomed; recommendation to diversify advisory composition. - Controversial topics: - IEO brings alternative views challenging dominant positions; analyses must be fully evidence-based. - Recommendations — number, focus, evolution: - Need recommendations to be prioritized and focused. - Positive trends: number of recommendations has been reduced and less detailed over time. - Examples preserved: 2002 Prolonged Use of IMF Resources: 14 recommendations + 13 sub-recommendations; 2011 Run-Up to the Financial and Economic Crisis: 5 generic recommendations; 2012 International Reserves: 5 generic recommendations. - Shift observed toward process-focused recommendations since 2006: - PRE-2006 TOTAL: Substance 35.5; Process 17.5; Culture 1.0; Total 54 - PERCENT: 66% Substance, 32% Process, 2% Culture - POST-2006 TOTAL: Substance 15.4; Process 42.5; Culture 5.5; Total 63 - PERCENT: 24% Substance, 67% Process, 9% Culture - Note: recommendations/sub-recommendations split when falling into two categories, accounting for non-integers. - Interpretation: shift may reflect choice of topics; Panel urges reversal toward outcome-focused recommendations. - Comparative advantage and operationalization: - IEO should identify issues and desired outcomes; Management should propose operationalization via Management Implementation Plans (MIPs); Board to judge whether proposals deliver necessary changes. - Tone and relations: - IEO should be clear and, when necessary, blunt; balance oversight with enhancing learning culture. - Some defensive attitudes among parts of Staff, more prevalent among senior staff and reportedly increased since 2006; need rebuilding of mutual trust and constructive dialogue (SPR plays key role). C. Effects on the Fund - General: - IEO contributed to culture valuing alternative thinking, openness to policy alternatives, and accountability. - Specific contributions: - IEO evaluations influenced internal reviews (e.g., conditionality, 2011 Triennial Surveillance Review). - Staff survey indications: - Readership of reports limited; perception of IEO impact mixed. - Increase in proportion of Staff that view IEO recommendations as taken fairly or very seriously: - IMF management: 2006 = 53% ; 2012 = 56% - Department heads: 2006 = 54% ; 2012 = 58% - IMF Executive Board: 2006 = 50% ; 2012 = 56% - Only 17 percent of Staff think IEO is changing IMF’s culture; more than half responded “don’t know”: - Staff response to “Is the IEO changing the IMF’s culture?”: - 2006: Don’t know 40.5% ; No 43.5% ; Yes 16.0% - 2012: Don’t know 52.2% ; No 30.8% ; Yes 17.1% - Familiarity declines: “Not familiar at all” more than doubled since 2006; two lowest familiarity categories increased from 26 to 45 percent between 2006 and 2012. - Among staff with less than five years’ tenure: 45 percent are not familiar at all with the IEO. - Among staff with more than 10 years’ tenure: 31 percent are unfamiliar. - By grade in 2012: 59 percent of Staff at the A12 to A13 level, 36 percent at the A14 and A15 level, and 14 percent at the B level said they had no or very limited familiarity with the IEO. - Recommendations to increase traction: - Substantial increase in IEO ‘in-reach’ targeted at junior staff: seminars, participation in induction courses, informal contacts. - Management should emphasize IEO importance for institutional architecture and encourage familiarity. - Include in-reach to new Board members as part of induction. D. The follow-up process — core findings and problems (overview) - Follow-up instruments introduced January 2007: - Management Implementation Plans (MIPs) for Board-endorsed recommendations. - Periodic Monitoring Reports (PMRs) to track implementation. - Core flaws identified: - Lack of strong ownership by Board and Evaluation Committee (EVC). - Conflict of interest for Management (drafting Summing Up, preparing MIPs, monitoring implementation). - Conversion of IEO recommendations into specific actions tends to dilute substantive core. - No mechanism for following-up non-specific recommendations or broader lessons. - Process became bureaucratic (“box-ticking”) and lacks independent evaluation. - Significant time lags at stages: finalization to Board presentation; Board discussion to MIP preparation; MIP release to Board discussion/approval — leading to loss of timeliness, dilution, and undermined learning. - Examples and evidence: - Time lags between Board discussion and MIP release vary between 3 and 12 months; some processes took more than a year (e.g., MIP for Run-Up to the Financial and Economic Crisis issued to full Board on February 14, 2012 — over a year after report publication). - PMRs comprehensive but limited to Board-endorsed recommendations; broader concerns not endorsed may not be included; monitoring often discontinues after one-off actions implemented. Recommended structural reforms to the follow-up process (Panel proposals) - Drafting of record of Board discussion and recommendations: - Draft should be prepared by Chair of the Evaluation Committee for Board approval (instead of Secretary’s Department preparing it). - Deadlines for MIP preparation and approval: - Management should present MIP within one to three months following Board discussion. - Evaluation Committee should review MIP and ensure Board approval no later than two months after MIP release. - MIPs should include justification of actions in terms of delivering desired outcomes, timetable, and cost estimate. - Separation of implementation/monitoring functions: - Periodic Monitoring Reports should continue annually but be prepared by the Internal Audit Office (separating implementation and monitoring to avoid conflicts of interest). - Panel does not consider IEO appropriate to perform periodic monitoring of specific recommendations. - IEO biennial outcome reviews: - IEO to prepare for the Board, on a biennial basis, selective issues-oriented reviews of extent of implementation and effects; these reviews should concentrate on bigger issues and should not include further recommendations. - Interaction with IMFC: - IEO should submit a report to the IMFC once a year during Annual Meetings, presented by the Director, focusing on major policy issues from recent evaluations. - Roles and mandates of EVC: - EVC should have clearer mandates and play critical leadership: advise on Director appointment; report regularly to Board on IEO work plan and implementation; Chair to draft records of Board discussions and recommendations; review with Management proposed follow-up actions. - Recommendation to ensure continuity in EVC membership beyond regular two-year cycles and regularize meeting frequency. --- ### IEO staffing, budget, and career-return issues - Office composition: - IEO comprises 15 positions: four B-level (managerial) positions (including the Director), six economist positions, three researchers and two assistants. - More than half of IEO staff drawn from outside the Fund. - Current professional staff composition reported: nine external recruits and three recruited from the IMF (three additional Fund Staff in process of being hired). - Productivity: - On average approximately two evaluations per year, including subsidiary reports; substantial share of work in-house; consultants typically produce background papers. - Budget comparisons and recommendation: - OECD study (2009) shows IEO is smallest, absorbing 0.5 percent of the administrative budget of the organization versus average 1.4 percent. - Panel recommends increasing IEO budget to 0.6 percent of the operating costs of the IMF and retaining that level going forward. - Career-return and promotion concerns: - Survey: Would working for IEO advance career prospects — Total: No 44.0% ; Not sure 43.9% ; Yes 12.1% - A9‐A11: No 27.6% ; Not sure 48.0% ; Yes 24.5% - A12‐A13: No 35.0% ; Not sure 52.4% ; Yes 12.6% - A14‐A15: No 45.4% ; Not sure 43.5% ; Yes 11.1% - B1‐B5: No 63.1% ; Not sure 32.3% ; Yes 4.6% - Perception that IEO positions have limited career potential; some anecdotal discouragement from IMF managers. - Issue when IEO promotes internally: promotions not automatically guaranteed recognition by Fund upon return unless ratified by Review Committee; suggested option for IEO Director to seek ratification when making promotions. - Essential recommendation: correct perceived bias against staff moving to IEO; Management and senior Staff should signal IEO service is valued in career progression. --- ### Staff awareness, survey findings, and implications - Survey headline results: - 2012 staff survey: 59 percent view IEO fairly/very independent (2006 = 65 percent). IEO perceived as impartial by around 60%. - Awareness/familiarity declined: “not familiar at all” rose to 21%; two lowest familiarity categories increased from 26 to 45 percent between 2006 and 2012. - Among staff with <5 years tenure: 45 percent “not familiar at all”; tenure >10 years: 31 percent unfamiliar. - Usage of IEO reports declined from 54% to 46% since 2006. - Only 17% believe IEO is changing IMF’s culture; “don’t know” rose from 40% (2006) to 52% (2012). - Perceptions of IEO as place to work: - 39% would consider working for IEO (stable); only 12% see it as career advancement. - IEO viewed positively for “broader experience” 69% and “enhancing analytical skills” 46%. - Interaction and consultation: - Staff involvement in evaluation preparation declined; 29% report no personal interaction (i.e., 71% have had firsthand connection). - Perceived overlap with Office of Internal Audit increased from 12% to 18%. - Perceived priority effectiveness: - “Supporting institutional governance and oversight” — 40% deem IEO effective. - “Enhancing learning culture” — 14% deem effective. - “Strengthening external credibility” — 40% report effectiveness. - “Promote greater understanding of the Fund’s work” — less than 3% think this should be a priority. - Areas for improvement (first choice shares): - Implementation of recommendations by Management 20.0% - Staff’s buy-in of recommendations 19.8% - Quality of recommendations 16.0% - Analysis of the problem 14.4% - Greater objectivity 11.4% - Dissemination to wider public 9.0% - More consultation with stakeholders 7.8% - Writing style 1.8% --- ### Follow-up of IEO recommendations — Annex findings and specific flaws - Lissakers identified absence of mechanism to follow-up as undermining IEO mandate; recommended more systematic approach. - Current follow-up framework (approved January 2007): - Management provides MIP for Board-endorsed recommendations including timetable and cost estimate. - Management presents PMRs to Board on implementation status, indicating difficulties and remedial actions. - Process features: - Draft Summing Up prepared in advance by IMF staff (typically SPR) and modified during Board meeting with Secretary’s Department input; Management coordinates MIP preparation. - Management (coordinated by SPR) is main driver drafting/monitoring actions; Board consulted usually via EVC. - Summing Ups often general and “not clear about what the Board has endorsed,” complicating monitoring. - Monitoring implementation issues: - Time lags: between Board discussion and MIP release 3 to 12 months; delays sometimes >1 year. - PMRs limited to Board-endorsed recommendations; broader IEO concerns not explicitly endorsed may not be monitored. - Monitoring often discontinues after one-off actions implemented; policy-oriented outcomes not tracked unless Board agreed specific suggestion. - Core flaws and risks: - Lack of clear standards for monitoring and evaluating follow-up effectiveness. - Management’s agenda-setting dominance and time-lags undermine Board oversight. - Lack of single public location for MIPs and PMRs reduces external monitoring. - Recommendations reiterated: - Increase Board ownership by having Chair of EVC draft Summing Up; set limits on time lag between Board discussion and MIP finalization; create public link on IEO website to follow-up materials. --- ### Summary of Panel recommendations (key elements) - Mandates: - Retain core mandates to enhance learning culture and support governance/oversight; drop fourth mandate promoting greater understanding throughout membership. - Follow-up process — five key elements: - Draft records of Board discussions and recommendations prepared by Chair of EVC for Board approval. - Management to present MIP within one to three months following Board discussion. - EVC to review MIP and ensure Board approval no later than two months after MIP release. - Annual PMRs should continue but be prepared by the Internal Audit Office. - IEO to prepare biennial selective issues-oriented reviews of extent of implementation and effects (no new recommendations); IEO to present a report to IMFC during Annual Meetings focusing on major policy issues. - Make special effort to keep continuity in EVC membership beyond regular two-year cycle. - Relations with Management and Staff: - Enhance two-way dialogue and mutual trust; frequent formal/informal channels; protect IEO independence while increasing in-reach. - Focus and scope: - Two long-term cross-cutting reports per year appropriate; Board should define “current operations” as current lending programs; IEO Director’s main consideration: how IMF fulfills basic functions. - Increase systematic evaluation of select country programs via periodic evaluations (typically every two years) of samples of EPAs and EPEs; alternate with biennial implementation review. - Recommendations presentation: - IEO should present recommendations clearly and, when necessary, bluntly; focus on policy and culture rather than processes. - Awareness and outreach: - Significantly increase in-reach activities; increase outreach to country authorities and civil society. - Avoid bureaucratization: - Management should stress IEO importance; encourage more informal engagement; Board should use IEO to support oversight; IEO should engage more with IMFC. - Staffing and budget: - Change staffing rules to facilitate mobility to/from IEO. - Increase budget from 0.5 percent to 0.6 percent of IMF’s operational/administrative budget. --- ### Responses and next steps - Managing Director statement (Executive Board Meeting March 21, 2013): - Agrees IEO increased integrity/quality of Fund work and is regarded as fully independent. - Identified three key challenges: more interaction between staff and IEO; keep reports focused on long-term cross-cutting issues and avoid interfering with operations; refocus follow-up on broader policy objectives. - Asked staff for concrete measures to strengthen follow-up. - Staff response: - Views IEO continues to contribute to learning culture and governance oversight. - Support more two-way interactions and in-reach; suggest seminars during evaluation conduct and before finalization. - Support periodic evaluations of EPAs/EPEs; emphasize Management’s role in operationalization; support timely MIP presentation to EVC within three months. - On Summing Up: staff view chairing EVC drafting Summing Up inconsistent with governance; propose standard practice with authoring office draft and SEC finalization. - On PMRs and OIA involvement: caution and need for consultation with External Audit Committee. - IEO response (March 18, 2013): - Welcomes positive assessment; accepts need to improve collaboration, timing, follow-up, and awareness. - Supports biennial issues-oriented reviews; prepared to assess sample of EPAs/EPEs. - On Summing Up: concerned about proposed EVC chair role; suggests IEO prepare first draft and work with SEC or that any new process be reviewed by EVC after three Summings Up. - Ready to work with Board, Management and Staff to resolve practicalities. --- ### IMFC presentation and follow-up - IEO would welcome brief presentation to IMFC during Annual Meetings focusing on major policy issues from recent evaluations. - IEO intends to present follow-up information from external evaluation in future Annual Reports and would welcome a follow-up external evaluation in five years. - Public Information Notice No. 13/40 — March 29, 2013: Executive Board discussed the second External Evaluation of the IEO (Board discussion held March 21, 2013); Directors welcomed panel’s report and many recommendations; broad support for improving follow-up, topic selection clarity, and in-reach while preserving IEO independence. *Source: _010113 — External Evaluation of the Independent Evaluation Office (IMF, March 2013).* ### 1. INTRODUCTION ..................................................................................................... ### 1. INTRODUCTION ### Purpose and scope - The Executive Board of the International Monetary Fund requested an external review of the Independent Evaluation Office (IEO). - This is the second external evaluation in the IEO’s twelve year history; the first review (the “Lissakers Report”) was presented to the Board in 2006. - The report focuses on IEO activities since 2006 and evaluates how well the IEO has met its institutional mandates as set out in the Panel’s terms of reference (see Appendix I). ### Terms of reference highlights - The Panel was asked to “assess the IEO’s effectiveness along several dimensions, including: (i) the appropriateness of evaluation topics; (ii) the independence of the IEO; (iii) the cost-effectiveness of the IEO and its operations; and (iv) the appropriateness and adequacy of the evaluation process including, but not limited to, how IEO recommendations are endorsed by the Board and implemented.” ### Methodology and consultations - Consultations included representatives of many member countries; interviews with the current and three former Managing Directors of the IMF; current Deputy Managing Directors and two former ones; interaction with all members of the Executive Board; several discussions with the former Chair of the Evaluation Committee; the Chair and a member of the Lissakers Panel; the current and two former Directors of the IEO, and its current Deputy Director; Directors of several IMF Departments; the Director of the G-24; the Head of the World Bank’s Independent Evaluation Group. - Organized three consultations with experts in Washington D.C. (at the Brookings Institution), London (at Chatham House) and Buenos Aires (at Centro de Estudios de Estado y Sociedad, CEDES). - Two consultations with civil society in Washington D.C. (at New Rules for Global Finance) and London (at Chatham House). - Consultations conducted on a no-attribution basis (see Appendix II for list of interviewees). - Benefited from contributions to the conference organized by the IEO at the end of 2011 on the IEO’s first decade of work, and from Ruben Lamdany and Hali Edison (eds.), Independent Evaluation at the IMF: The First Decade, Washington D.C.: International Monetary Fund, 2012. - Conducted a survey of IMF staff on knowledge and views of the IEO and its reports, using a set of questions largely coinciding with those used for the Lissakers Report, with several new and reformulated questions. - The Panel had full access from the IMF and the IEO to all information deemed relevant. ### Report structure - The report is divided into seven parts: - Part 1: Introduction (this section) - Part 2: Overall evaluation of the IEO - Part 3: Review of implementation of Lissakers Report recommendations - Part 4: Assessment of IEO independence and mandates - Part 5: Detailed assessment of evaluation cycle (reports, follow-up, Evaluation Committee) - Part 6: Staffing and budgetary issues of the IEO - Part 7: Summary of main recommendations - The assessment and recommendations are fully shared by all members of the Panel. --- ### 2. OVERALL EVALUATION ### IEO mandates (as given at inception) - Enhance the learning culture of the Fund. - Support institutional governance and oversight. - Strengthen the Fund’s external credibility. - Promote greater understanding of the work of the Fund throughout the membership. ### Panel view on mandates - The Panel regards the first two mandates as the most important; the third is seen as a result of fulfilling the first two. - The Panel suggests the fourth mandate (“Promote greater understanding of the work of the Fund throughout the membership”) be dropped as it is no longer considered a crucial part of the IEO’s work. ### Broad assessment - The IEO has been a successful institution in improving governance and transparency of the IMF and in helping develop a learning culture within the Fund. - The IEO is widely considered to be the most independent of the evaluation offices of the international financial institutions. - A number of individuals who initially opposed the creation of the Office or doubted its independence changed their opinions based on the IEO’s record. - Strong consensus from IMF internal stakeholders, national governments, and external stakeholders that the IEO strengthened the IMF’s external credibility. ### Strengths of the IEO - Generally good topic selection and quality of reports, with focus on longer-term cross-cutting issues. - Use of solid evidence from a wide range of sources, including views of governments. - Full access to internal information (an issue raised by the Lissakers Report has been overcome). - IEO analyses provide alternative views to Management and Staff; these alternatives have sometimes been accepted and led to changes in IMF analyses, processes and operations. - Contribution to building a learning culture; the Lissakers Report conclusion that the IEO “deepened the Fund’s evaluation culture and helped make it more open to discuss policy alternatives” continues to be correct. ### Context and tensions - The IMF operates in a “risky and messy business” and faces pressures, including from its most influential members. - Some interviewees argued the IEO has not always recognized operational difficulties and constraints; critical views may at times have been seen as damaging the Fund’s external credibility. - The Panel judges that transparency and openness to evaluation have contributed to, rather than damaged, external credibility. ### Areas for improvement (identified by the Panel) - Follow-up process: - The post-Lissakers follow-up process lacks strong Board ownership, involves a conflict of interest for Management, is poorly suited for broader substantive recommendations, and has become very bureaucratic. - IEO recommendations tend to be watered down at each stage. - The Panel recommends overhauling the follow-up process, proposing involvement of the IEO in follow-up without weakening Board and Management responsibilities (see Section 5). - Attitudes within Management and Staff: - While relations are broadly positive, defensive or antagonistic attitudes in some parts of Management and Staff need to be addressed to derive full benefit from the IEO. - Topic selection and scope: - The principle that the IEO should not interfere in “current operations” remains valid, though the definition of “current operations” needs clarifying. - The main consideration in topic selection should be evaluating how the IMF is fulfilling its functions. - The IEO has not always chosen issues central to the IMF’s purposes; there is a strong case for revisiting key issues previously evaluated. - The IEO should focus recommendations more on policy issues for the Fund and its culture, rather than on processes (the latter are the responsibility and comparative advantage of Management). Since 2006, recommendations have tended to become more process-oriented. - Awareness and in-reach: - The Staff survey identified that awareness and knowledge of the IEO is surprisingly low within the Fund, and lower than in 2006. - Raising the IEO’s profile within the IMF is essential to increase traction of its reports. - The Panel proposes measures to increase ‘in-reach’ of the IEO and also to increase outreach to country authorities and external stakeholders, including civil society. - Risk of institutional routinization: - It is essential the IEO avoid becoming “bureaucratized, routinized and marginalized” (reiterating the Lissakers Report). - IMF Management must stress to Staff the importance of the IEO as an instrument for continuous improvement; Staff must engage continuously and positively with the IEO while respecting its independence. - The Executive Board must use the IEO as an essential instrument of its oversight. - The IEO should engage more with the International Monetary and Financial Committee (IMFC); after a good initial start, IEO input to the IMFC has become less important. ### Resource implications - Proposed reforms imply changes in staffing rules to facilitate mobility of IMF Staff to and from the IEO, and a small increase in the budget to allow the IEO to carry out suggested additional activities. --- ### 3. THE LISSAKERS REPORT ### Background and questions addressed by the Lissakers Panel - The Lissakers Panel assessed how well the IEO had carried out its four mandates during its first five years. - It also examined whether the IMF needed an evaluation office; whether the IEO was independent; whether it had addressed the right issues; and whether the reports were of high quality. ### Key Lissakers findings and the Panel’s assessment - Necessity of an evaluation office: - The Lissakers Panel concluded an evaluation unit was needed; independent evaluation aligned with international best practices of governance and accountability to shareholders. - The current Panel assesses the IEO remains highly relevant; the IMF has become more transparent and willing to engage with critics since 2006. - The IEO’s role shifted somewhat toward global systemic issues following the global economic and financial crisis, while controversies in operations continue, necessitating ongoing learning. - Independence and safeguards: - Lissakers concluded safeguards ensured the IEO was – and was seen to be – independent, but warned that (a) IEO staff should not be dominated by IMF “insiders”, (b) the IEO needed full access to internal documents, and (c) it had to guard against undue influence by Management and Staff. - The current Panel finds the IEO remains fully independent, and finds no evidence the three concerns are still relevant. - The IEO’s staff mix is now more skewed to “outsiders”; recruitment of high quality staff from the IMF has been difficult. The Panel believes a mix of “insiders” and “outsiders” remains appropriate. - No evidence the IEO encountered difficulties accessing information or that judgments were influenced by pressure from IMF Staff or Management. - Topic selection: - Lissakers found general satisfaction with IEO topic choices, urged focus on controversial issues and scarce resources, and recommended evaluating more individual country programs including “explosive cases like Argentina and Indonesia”. - The current Panel agrees IEO should be able to evaluate controversial issues and should choose topics carefully. It maintains the principle that the IEO should not evaluate “current operations”, but that the definition of “current operations” needs clarification. - The Panel views the IEO’s comparative advantage as lying more with cross-country comparisons to draw lessons of wider relevance. - Report quality: - Lissakers judged IEO largely avoided major omissions or errors but fell short in revealing “deeper truths about the quality of the Fund’s performance”, focused on process rather than substantive outcomes, and made “insufficient use of counterfactuals”. - Lissakers recommended regular use of peer review by outside experts and “thought-provoking” recommendations focused on most important issues. - The current Panel finds IEO reports generally of high quality and that the IEO now regularly uses external expertise. - However, recommendations since 2006 have become increasingly process-oriented; the Panel argues recommendations should aim at outcomes to be achieved by the IMF, leaving Management and the Board to design actions. - Follow-up and impact on learning culture: - Lissakers highlighted enhancing the IMF’s learning culture as central and found some evidence lessons were being incorporated, but called for “more systematic monitoring of follow-up to IEO reports”. - The current Panel agrees the IEO continues to have a positive impact but finds follow-up by the IMF to IEO recommendations remains patchy and incomplete. - A formal follow-up mechanism was put in place after the Lissakers Report, but the Panel judges it has not been working properly and must be revamped. - Role in Board oversight: - Lissakers was critical of the Board and Evaluation Committee, concluding “the Board needs to be more assertive with regard to evaluations” and noting “no formal mechanism for the Board to follow-up specific recommendations made by the IEO”, singling out deficiencies in the ‘summing up’. - The current Panel finds the Board continues to make less use of IEO reports in oversight than it should and proposes concrete steps to allow the Board to play a more proactive governance role. - Outreach: - Lissakers judged IEO outreach to external stakeholders and civil society “inadequate”, that reports were “too long, verbose and dense”, and outreach “concentrated on the richest member countries”, recommending a “complete overhaul”. - The current Panel finds evidence the IEO has responded and devoted more effort to outreach and making reports more accessible; the Panel supports these changes. - In contrast to Lissakers, the current Panel considers ‘in-reach’ activities (internal outreach within the IMF) have become an increasing weakness and that improvements in in-reach are essential to strengthen the Fund’s learning culture. *Source: _010113 - 1. INTRODUCTION* ### 4. INDEPENDENCE AND MANDATES ### 4. INDEPENDENCE AND MANDATES ### Independence assessment - Ensuring the IEO’s unqualified independence — both actual and perceived — is central to effective discharge of all its core mandates. - The IEO must exercise independence in multiple dimensions: independent staffing of the Office, independence in topic selection, and independence in the conduct of evaluations. - Finding: The IEO retains both actual and perceived independence in its work, as reflected in the views of a wide range of stakeholders – country authorities, Board members, Management, Staff, academia and civil society. - Evidence of retained independence: - Developed and maintained an independent capacity to conduct evaluations and to discharge its mandates free from external influence. - Retained internal capacity to conduct effective evaluation and to disseminate findings, founded on independence in staff selection and recruitment, in choice of external consultants and advisory panelists, and in independent outreach efforts. - Continues to select and pursue evaluation topics and to conduct evaluations free from external influence. - Has access to all available information required. - Continues to freely comment on any aspect of Fund policy and process, and of Fund governance, delivering analysis and recommendations free from interference. ### Perceptions and survey results - Perceptions of IEO independence among staff remain reasonably robust. - 2012 staff survey: 59 percent of survey respondents considered the IEO to be either independent or very independent. - Comparison: 2006 survey result was 65 percent. - Only one in ten respondents considered that the IEO was not independent. ### Exercise of independence and relationships with stakeholders - The Lissakers report found the institution had secured and maintained its independence in early years and urged continued guarding of independence by addressing significant policy issues and exercising firm and bold judgment. - Finding: The IEO has remained resolute in maintaining an arms-length relationship with the Board, Management and Staff, reflected in the confidence and firmness of its evaluations and recommendations. - Emerging problem: As the IEO matured, concern with perceived independence increased, leading to relations that are more tense, formalized, process-focused, and dependent on quality of inter-personal dynamics. - Assessment: This trend has been detrimental to the IEO’s effectiveness and should be reversed; greater collaboration should be restored. - Identified tensions and areas needing attention: - Prioritization of IEO recommendations. - The follow-up process to IEO recommendations. - Management signaling to Staff the relevance of IEO analyses and recommendations. - The Board’s role in taking stronger ownership of the IEO’s evaluations and recommendations. - Panel view: Need to better codify processes governing these areas. - Equally important: restore informal links between the IEO and the Fund and encourage interaction among the IEO, Board and Management; this view is shared by many stakeholders. ### Mandates and policy recommendation - Broad agreement that the first two of the IEO’s mandates continue to be relevant. Underlying rationale: increase external credibility of the Fund and its operations through objective, transparent and independent evaluations; independence is key. - The Panel identifies two key mandates: - Enhance the learning culture of the Fund, thus ensuring that it draws lessons from past experiences to improve its future performance. - Support institutional governance and oversight, working as a basic instrument for the Board to effectively carry out its governance and oversight responsibilities of the Fund. - The third function, strengthening the Fund’s external credibility, is largely the result of exercising the first two mandates. - Fourth mandate assessment: - The fourth mandate, promoting greater understanding of the work of the Fund throughout the membership, was important in the early years. - Given the significant increase in transparency of the Fund’s policies and operations in recent years, this mandate has become much less important and may even be inconsistent with IEO independence and oversight functions. - External stakeholders widely share this view; Staff survey shows Staff do not perceive this to be a major objective of the IEO. - Recommendation: The IEO’s fourth mandate should be dropped. *Source: _010113 - 4. INDEPENDENCE AND MANDATES* ### 5. A DETAILED ASSESSMENT OF THE EVALUATION CYCLE ### 5. A DETAILED ASSESSMENT OF THE EVALUATION CYCLE ### A. Nature of the Reports and Topic Selection - Over its twelve year history the IEO has presented to the Board nineteen reports — slightly less than two per year. - Most reports have dealt with major long-term cross-cutting IMF issues; two reports dealt with relations between the IMF and individual countries (Argentina and Jordan) and were completed during the first five-year period of the Office (outside the period covered by this evaluation). - Topic selection process: - Topics are chosen by the Director after open consultation with country authorities, Board members, IMF Management and Staff, and external stakeholders. - Suggested topics are posted on the Office’s website and open for comment. - The latest consultation on topics for FY 2013 (starting in May 2013) is based on a document produced by the IEO in September 2012. - The Panel considers this a well-functioning process and recommends the Director continue to have full freedom to choose IEO issues. - Definition of “current operations”: - Existing rules constrain topics so they do not interfere in the ‘current operations’ of the Fund. - Major Panel concern: lack of a clear definition of ‘current operations’. - Recommendation: Board should define ‘current operations’ in a narrow sense, as current lending programs, enabling IEO review of recent or current activities that do not involve lending programs, and past lending programs even if countries have new programs in place. - Distinction urged between ‘current operations’, ‘current issues’, and ‘recurrent activities’. - Views on number, length and depth of reports: - General agreement that topics selected since 2006 have been broadly appropriate, though some reports were not viewed as central to the Fund’s mandate (examples noted: Jordan; IMF Involvement in International Trade Policy Issues). - Comparative staff readership of IEO reports (Figure 1): - IMF Performance in the Run-up to the Financial and Economic Crisis: 28.4 percent of Staff - Research at the IMF: 18.90 percent - Structural Conditionality: 16.4 percent - Interactions with Member Countries: 13.4 percent - Exchange Rate Policy Advice: 12.3 percent - Multilateral Surveillance: 12.1 percent - Governance of the IMF: 11.0 percent - Financial Sector Assessment Program: 10.0 percent - IMF and Aid to Sub-Saharan Africa: 9.0 percent - International Trade Policy: 4.3 percent - Note: 17% of Staff indicated they had not read any of the IEO reports listed. - Recommendation: primary consideration for topic choice should be centrality to the Fund’s mandates; IEO should be free to revisit past-evaluated issues (Director has decided to update two 2002 reports). - Number and format of reports: - Panel concurs that two full-scale evaluation topics per year is broadly appropriate given current Office size and Board/Management capacity. - Many interviewees favored addition of shorter, more timely reports; Board could consider requesting arms-length advisory reports but Director retains discretion. - Panel does NOT favor expanding to a larger set of shorter reports for fear of jeopardizing rigor and reputation. - Country-specific evaluations: - Merits: reveal important insights about program design and implementation. - Risks: with only two substantial evaluations per year, risk that country-specific reports lack broad applicability. - Majority view: cross-cutting issues are more appropriate, though “paradigmatic cases” (e.g., Argentina; mentions of Iceland, Latvia, Greece) can be appropriate. - Recommendation: IEO should evaluate country programs more systematically but focus on strategic and generic issues and not duplicate internal reviews. - Specific recommendation: IEO should undertake periodic evaluations, typically every two years, of a sample of internal ex-post assessments (EPAs) and ex-post evaluations (EPEs) of country programs. - Note: EPAs review programs for countries with long-term access to Fund financing; EPEs assess programs involving exceptional access. - If the Fund were to undertake ex-post evaluations of all programs, IEO would evaluate a sample of all IMF programs. - Access to information and report length: - Problems highlighted in the Lissakers Report appear overcome; IEO feels it has full access to needed information. - Report length is now appropriate; technical aspects covered in appendices/addenda. - Executive summaries could be improved to better convey main analysis and recommendations; this view is shared particularly by country authorities. ### B. Quality of the Reports and Nature of their Recommendations - Overall quality: - Evaluation of eleven reports presented to the Board since 2006: generally of high quality; this view was broadly and overwhelmingly shared in consultations. - Methodology and consultation: - Use of Advisory Groups and workshops with policymakers and analysts is welcomed and encouraged. - Recommendation: further diversify composition of Advisory Groups and workshops to include a broader set of external views. - Controversial nature of topics: - IEO evaluations often controversial; responsibility to bring alternative views that challenge dominant positions (including those of large shareholders, Fund Management, and Staff). - High requirement for analyses to be fully evidence-based and meet high quality standards. - Recommendations: number, focus and evolution - Need for recommendations to be prioritized and focused; Lissakers report criticized earlier IEO focus on “process rather than on the substantive issues underlying the process” and excessive number of recommendations. - Positive trend: IEO has reduced the number of recommendations and they have become less detailed over time. - Example: 2002 Prolonged Use of IMF Resources: 14 recommendations + 13 sub-recommendations. - 2011 Run-Up to the Financial and Economic Crisis: 5 generic recommendations. - 2012 International Reserves: 5 generic recommendations. - However, Panel finds shift toward process-focused recommendations since 2006: - Pre-2006 eight reports: two-thirds of recommendations were substantive versus process. - Post-2006 eleven reports: around two-thirds of recommendations were process-focused. - Table 1 summarized (exact extracted totals and percents preserved): - PRE-2006 TOTAL: Substance 35.5; Process 17.5; Culture 1.0; Total 54 - PERCENT: 66% Substance, 32% Process, 2% Culture - POST-2006 TOTAL: Substance 15.4; Process 42.5; Culture 5.5; Total 63 - PERCENT: 24% Substance, 67% Process, 9% Culture - Note: Recommendations and sub-recommendations that fell into two separate categories were split, accounting for non-integers. - Interpretation: part of the shift may reflect choice of topics (earlier evaluations focused on programs or surveillance; later on functional work), but implications exist for topic choice and recommendation focus. - Comparative advantage and operationalization: - IEO comparative advantage: identify issues requiring changes in existing practices and the outcomes expected from such changes. - Management and Staff have comparative advantage in proposing how to operationalize solutions. - Therefore Management should propose operationalization through Management Implementation Plans (MIPs); Board should judge whether proposals will deliver necessary changes. - This implies significant reforms to the follow-up process for Board-endorsed recommendations (see section D). - Tone and balance: - IEO should express recommendations clearly and, when necessary, bluntly. - Tension exists between oversight and enhancing the Fund’s learning culture; IEO must remain balanced, objective, and constructive, focusing on areas where the Fund could do better while accounting for conditions at decision times. - Perception: IEO has generally achieved a reasonable balance; sometimes criticized externally for not being hard-hitting enough, sometimes internally for being too critical. - IEO seen as bringing new ideas and alternative views to the Fund. - Relations with Staff and Management: - Some defensive or antagonistic attitudes observed among parts of IMF Staff toward the IEO, more prevalent among senior staff and reportedly increased since 2006. - These attitudes can be counter-productive and require constructive dialogue and rebuilding of mutual trust; Strategy and Policy Review (SPR) Department plays a key role in Management-IEO communication. - Examples of positive reception: Run-Up to the Crisis report—critical yet widely read and generally perceived as identifying real problems needing address. - Many Management and Staff responses were positive about IEO analysis and recommendations, often asking for greater specificity. ### C. Effects on the Fund - General assessment: - Perception (broadly shared) that the Fund has undergone significant positive changes in recent years; IEO has made an important contribution, primarily by promoting a culture valuing alternative thinking and encouraging openness to policy alternatives (helped break down “group think”). - IEO has contributed to developing a culture of accountability. - Specific contributions: - Identification of problems in individual reports has led to changes in specific areas, sometimes with significant lags. - Examples: 2012 internal reviews on conditionality and the 2011 Triennial Surveillance Review drew in part from IEO evaluations identifying excessive structural conditionality and silo behavior in surveillance. - Panel concludes IEO is largely fulfilling its function of enhancing the Fund’s learning culture. - Less positive trends from staff survey: - Limited number of Staff who have read or consulted IEO reports. - Relatively poor perception of IEO impact: - Increase in proportion of Staff that view IEO recommendations as being taken fairly or very seriously: - IMF management view of IEO recommendations: 2006 = 53% ; 2012 = 56% - Department head view of IEO recommendations: 2006 = 54% ; 2012 = 58% - IMF Executive Board view of IEO recommendations: 2006 = 50% ; 2012 = 56% - However, only 17 percent of Staff think the IEO is changing the IMF’s culture; more than half responded “don’t know”. - Staff response to “Is the IEO changing the IMF’s culture?”: - 2006: Don’t know 40.5% ; No 43.5% ; Yes 16.0% - 2012: Don’t know 52.2% ; No 30.8% ; Yes 17.1% - Limited knowledge/familiarity with the IEO among Staff: - Staff familiarity categories (Figure 3): exact category shares not tabulated here except the following points preserved: - “Not familiar at all” has more than doubled since 2006. - The two lowest categories of familiarity increased from 26 to 45 percent between 2006 and 2012. - Among staff with less than five years’ tenure: 45 percent are not familiar at all with the IEO. - Among staff with more than 10 years’ tenure: 31 percent are unfamiliar with the IEO. - By grade familiarity in 2012: 59 percent of Staff at the A12 to A13 level, 36 percent at the A14 and A15 level, and 14 percent at the B level said they had no or very limited familiarity with the IEO — all significant increases vis-à-vis 2006. - Recommendations to increase traction and awareness: - Substantial increase in IEO ‘in-reach’ activities is needed, particularly targeted at junior staff. - Suggested measures: seminars on evaluation reports; IEO participation in staff induction courses; more informal contact methods. - Management should proactively emphasize to Staff that the IEO is an important part of institutional architecture and encourage familiarity with IEO analyses and recommendations. - Part of IEO in-reach should be directed at new Board members as part of induction. ### D. The Follow-up Process - Background: - Current follow-up process introduced to meet a Lissakers Report recommendation (2006). - Two instruments created: - Management Implementation Plans (MIPs) for recommendations endorsed by the Board. - Periodic Monitoring Reports (PMRs) to track implementation. - Broad perception and performance: - Broad-based perception (including many Board members and external analysts) that the follow-up process is not working well. - Staff survey highlights implementation of IEO recommendations as the area most needing improvement. - Panel concurs that the follow-up process must be revamped. - Core issues identified with current process: - IEO recommendations are converted into specific actions that tend to dilute the substantive core of recommendations. - No monitoring of broad policy conclusions and concerns unless specific recommendations have been explicitly endorsed by the Board. - Consequence: recommendations deemed met or on track are sometimes raised again in subsequent IEO reports because broader underlying issues were not monitored. - Example: 2007 Structural Conditionality evaluation recommended “the Fund develop a monitoring and evaluation framework linking conditions in each program to reforms and specified goals.” Board endorsed the recommendation and an interim step of improving and disclosing MONA (Monitoring of Fund Arrangements). After MONA implementation the Fund considered the issue addressed, but there was no subsequent monitoring of the broader recommendation to link conditions to reforms. - Panel identifies four major problems with the process (text ends here as source content ends). *Source: _010113 - 5. A DETAILED ASSESSMENT OF THE EVALUATION CYCLE* ### 1. It lacks strong ownership by the Board and, particularly, its Evaluation Committee ### 1. It lacks strong ownership by the Board and, particularly, its Evaluation Committee (EVC). ### Major findings: weaknesses in the follow-up process - The follow-up process: - Lacks strong ownership by the Board and, particularly, its Evaluation Committee (EVC). - Involves conflicts of interest for Management, which has the triple responsibility of overseeing the summing up of the Board discussion, preparing the subsequent implementation plan, and monitoring its application. - Does not take into account the diverse nature of IEO recommendations; there is no mechanism for following up on non-specific recommendations and lessons that have broader policy and/or cultural implications. - Has become a very bureaucratic (“box-ticking”) exercise and lacks any independent evaluation. - The summing up is defined as a record of the recommendations that are endorsed by the Board. - The IEO’s comparative advantage is to identify policy issues and desirable outcomes rather than to design and implement the detailed actions; responsibility for selecting and designing specific actions should lie with Management and be expressed through Management Implementation Plans (MIPs). - Significant delays (lags) exist at three stages: - From finalization of IEO evaluations to the presentation of the reports to the Board. - From Board discussion to preparation of Management Implementation Plans. - From release of MIPs to Board discussion/approval of MIPs. - Examples of consequences of long lags: - Recommendations lose timeliness and ability to influence contemporary programs and policies. - Disproportionate dilution of recommendations; loss of institutional memory amid frequent turnover of Board and Evaluation Committee membership. - Undermining of the IEO’s ability to influence the internal learning culture. - Evidence on lags: although several cases finished in 5 to 7 months, in three cases the whole process took more than a year (delays in Board approval of MIPs often reflect disagreements between the Board and Management). ### Recommended structural reforms to the follow-up process - Drafting of the record of Board discussion and recommendations: - The draft record of the Board discussion and of recommendations that are endorsed by the Board should be prepared by the Chair of the Evaluation Committee for approval by the Board (instead of being prepared by the Secretary’s Department). - Deadlines for MIP preparation and approval: - Deadlines should be set for preparation of the MIP and its approval by the Board. - A period of between one to three months following the Board discussion is considered sufficient to prepare the necessary Management Implementation Plan. - Another month or two should be allowed for the MIP to be approved by the Board. - Management Implementation Plans should include: a justification of actions in terms of delivering the desired outcomes, a timetable, and an estimate of their cost. - The Evaluation Committee should initially review the Management Implementation Plan on behalf of the Board to reduce further delays. - Separation of implementation/monitoring functions: - Periodic Monitoring Reports should continue to be prepared annually, but under the responsibility of the Internal Audit Office (separating Management’s implementation and monitoring functions to avoid conflicts of interest and to promote internal lesson-learning). - The Panel does not consider it appropriate for periodic monitoring of specific recommendations to reside with the IEO. - IEO biennial outcome reviews: - The IEO should prepare for the Board, on a biennial basis, an issues-oriented review of the extent to which its recommendations have been implemented and their effects. - These reviews should be selective, concentrate on ‘bigger issues’ relevant at a point in time, and should not include further recommendations; the Board would decide whether further actions are needed in light of these reviews. - Interaction with the IMFC: - The IEO should submit a report to the IMFC once a year, during the Annual Meetings, which would be presented in the meeting by the Director. - This report should be substantive in focus, concentrating on the major policy issues arising from recent evaluation reports. ### Roles and mandates of the Evaluation Committee (EVC) - The Evaluation Committee is the main instrument of the Board in relation to the evaluation process; full ownership and leadership by the Committee is critical for success. - The Committee plays a crucial role in interacting with Management and Staff on implementation of Board-endorsed recommendations. - Concerns about continuity: - Relatively frequent rotation in the Committee is a concern; special effort should be made to keep continuity in membership beyond the regular two-year cycle, and to regularize the frequency of its meetings. - Meeting frequency note: Three meetings a year have been typically held since 2006, except in 2010 (when only one was held) and 2011 (four). Meetings were more frequent in 2004 (six) and 2005 (five). - The Committee’s current mandate is very general: “To follow closely the evaluation function in the Fund and advise the Executive Board on matters relating to evaluations, including those of the IEO.” - The Panel identifies four specific roles for the Committee with regard to the IEO: - To advise the Board on the appointment of the IEO Director. - To report regularly to the Board on the IEO’s work plan, ongoing evaluations, and implementation of IEO recommendations endorsed by the Board (using feedback from monitoring reports). - Through the Chair of the Committee, to draft records of Board discussions and of recommendations endorsed by the Board. - To review with Management the proposed actions to follow up on Board-endorsed recommendations. - Recommendation: the mandates of the Evaluation Committee should be more clearly drafted. ### Two types of monitoring reports (recommended) - Type 1 — Monitoring of MIP actions (periodic, specific): - Periodic Monitoring Reports currently fulfill this role. - Recommended responsibility: Internal Audit Office should prepare these reports (annual). - Type 2 — Monitoring of broader, generic issues and cultural change (biennial, selective): - IEO should carry out biennial selective reviews on major generic issues (e.g., embedding cultural changes recommended in the IEO report on IMF Performance in the Run-up to the Financial and Economic Crisis; whether silos have been broken down). - These reviews should employ IEO evaluation techniques, be selective, focus on bigger issues, and not include new recommendations. --- ### Staff and budgetary issues - IEO staffing and output: - The Office comprises 15 positions: four B-level (i.e., managerial) positions (including the Director), six economist positions, three researchers and two assistants. - More than half of the IEO’s staff is drawn from outside the Fund. - The IEO remains productive, conducting on average approximately two evaluations per year, including a range of subsidiary reports for each evaluation. - A substantial share of IEO work is conducted in-house; consultants are used typically to produce background papers accompanying main evaluation reports. - Since 2006 the IEO has increased use of external advisors and commentators to solicit a wide range of opinions and perspectives. - Current professional staff composition reported: nine external recruits and three recruited from the IMF (three additional Fund Staff are in the process of being hired). - Budget scale and recommendation: - An OECD study provides comparative data of the IEO’s budget and those of five multilateral development banks in 2009. - Results indicate the IEO is the smallest, absorbing 0.5 percent of the administrative budget of the organization versus an average of 1.4 percent. - The Panel considers the budget appropriate to date but recommends a modest increase to meet additional work from recommendations: - Recommend that the IEO’s budget be increased to 0.6 percent of the operating costs of the IMF, and that this level be retained going forward. - Career-return and promotion concerns for Fund staff serving in the IEO: - Survey results (Figure 5) — staff response to whether working for the IEO would advance career prospects: - Total: No 44.0% ; Not sure 43.9% ; Yes 12.1% - A9‐A11: No 27.6% ; Not sure 48.0% ; Yes 24.5% - A12‐A13: No 35.0% ; Not sure 52.4% ; Yes 12.6% - A14‐A15: No 45.4% ; Not sure 43.5% ; Yes 11.1% - B1‐B5: No 63.1% ; Not sure 32.3% ; Yes 4.6% - Perception exists that a position at the IEO has limited career potential, particularly for those planning to return to the Fund. - Some anecdotal evidence that some IMF managers have discouraged Staff from applying for IEO jobs. - Issue with returning staff promoted within the IEO: - Economist and Managerial-level personnel have the right to return to their former or equivalent positions in the same department at the Fund, with their period of service at the IEO counting towards grade requirements for promotion. - Difficulty arises with IEO staff promoted internally by the IEO Director: without ratification by a relevant IMF Review Committee there is no guarantee the promotion will be recognized by the Fund at return. - IMF Human Resources makes best efforts to find a suitable position at that grade, but no automatic guarantee exists. - Suggested options to address return/recognition problem: - The IEO Director could seek ratification by the Review Committee when promotions at higher levels are made; if ratified, staff could retain that grade on return; if not ratified, the Director could still make the promotion on the understanding it was not necessarily recognized by the Fund. - Essential recommendation: correct perceived bias against Staff moving to the IEO; Management and senior Staff should consistently signal that service at the IEO is welcomed, valued in career progression, and regarded as enhancing skills and independent thinking. *Source: _010113 - 1. It lacks strong ownership by the Board and, particularly, its Evaluation Committee* ### 7. SUMMARY OF RECOMMENDATIONS ### 7. SUMMARY OF RECOMMENDATIONS ### Overall assessment - The IEO has been a successful institution that has helped improve transparency and oversight and enhanced the Fund’s learning culture. - The IEO is regarded by all stakeholders as fully independent, likely the most independent of the evaluation offices of the international financial institutions. - The IEO has generally chosen good topics and produced evaluations of high quality. - There is broad consensus that the IEO has strengthened the IMF’s external credibility. ### Mandates and recommended change - The IEO’s core mandates: - Enhance the Fund’s learning culture. - Support institutional governance and oversight as an instrument of the Board. - The third mandate—strengthening the Fund’s external credibility—is an outcome of the core mandates. - Recommendation: Drop the fourth mandate—promoting greater understanding of the work of the Fund throughout the membership—because it has become less important with increased Fund transparency and may be inconsistent with IEO independence and oversight functions. ### Follow-up process: identified problems - The current follow-up process for Board-approved recommendations: - Is undermined by lack of strong ownership by the Board and its Evaluation Committee. - Involves a significant conflict of interest for Management. - Does not sufficiently monitor broader, more policy-oriented recommendations. - Has become very bureaucratic. ### Follow-up process: proposed new system (five key elements) - Draft records of Board discussions on IEO evaluations and of recommendations endorsed by the Board would be prepared by the Chair of the Evaluation Committee, for approval by the Board. - Management should present the Management Implementation Plan within one to three months following the Board discussion. - The Evaluation Committee should review the Management Implementation Plan on behalf of the Board and ensure that it is approved no later than two months after its release. - Periodic Monitoring Reports should continue to be prepared annually, but by the Internal Audit Office. - The IEO should prepare for the Board, on a biennial basis, an issues-oriented review of the extent to which its recommendations have been implemented; this report should be selective and focus on major generic issues rather than an exhaustive review of specific actions. - The IEO should present a report to the IMFC during the Annual meetings, concentrating on the major policy issues arising from recent reports. - The new system increases responsibilities for the Evaluation Committee and its Chair; recommendation to make a special effort to keep continuity in the Committee membership beyond the regular two-year cycle. ### Relations with Management and Staff - While relations are generally positive, defensive and in some cases antagonistic attitudes exist between some parts of Management and Staff and the IEO. - Recommendation: Address these through enhanced two-way dialogue and building mutual trust, particularly between the Strategy and Policy Review Department and the IEO. - Dialogue should be frequent, include formal and informal channels, and fully respect the IEO’s independence to ensure IEO reports gain internal traction. ### Focus of reports and definition of scope - The IEO’s focus on long-term cross-cutting issues is appropriate; two such reports each year seems broadly right. - The Panel recommends the Board define “current operations” as current lending programs, implying the IEO should be free to review any other recent, current, or recurrent Fund activities. - Topic selection principle: the IEO Director’s main consideration should be how the IMF is fulfilling its basic functions. - Suggested future topics: functioning of the international monetary system and global macroeconomic cooperation; revisiting key issues from past evaluations. - Recommendation: Increase systematic evaluation of select country programs: - Periodic evaluations (typically every two years) of a sample of internal ex-post assessments (EPAs) and ex-post evaluations (EPEs) of country programs. - If the Fund carries out ex-post evaluations of all its programs (recommended), the IEO would evaluate a sample of all programs. - Alternate these evaluations every other year with the biennial issues-oriented implementation review. ### Presentation of recommendations - The IEO should present its recommendations clearly and, when necessary, bluntly. - The IEO should focus recommendations more on policy and the culture of the Fund rather than on processes, which are Management’s responsibility. - Observation: Recent trend has been toward more process-oriented recommendations; recommendation to reverse this trend. ### Awareness, outreach, and “in-reach” - Staff survey shows surprisingly low awareness and knowledge of the IEO’s work among Staff. - Recommendation: Significantly increase ‘in-reach’ activities to raise the IEO’s profile within the IMF. - Recommendation: Increase outreach to country authorities and external stakeholders, including civil society. ### Avoiding bureaucratization and enhancing use of IEO - Essential that the IEO avoid becoming “bureaucratized, routinized and marginalized” as it matures. - IMF Management should more actively and regularly stress to Staff the importance of the IEO for enhancing the Fund’s credibility and learning culture. - Consider measures to facilitate more frequent and informal engagement between Staff and the IEO while fully respecting IEO independence. - The Board should more effectively use the IEO to support its oversight function. - The IEO should engage more frequently with the International Monetary and Financial Committee. ### Staffing rules and budget - Recommend changes in staffing rules to facilitate mobility of IMF staff to and from the IEO. - Recommend a small increase in the budget from 0.5 percent to 0.6 percent of the IMF’s operational budget to allow the IEO to undertake the additional activities proposed. *Source: 7. SUMMARY OF RECOMMENDATIONS, _010113 - 7. SUMMARY OF RECOMMENDATIONS* ### ANNEX III: ASSESSMENT OF FOLLOW-UP OF IEO RECOMMENDATIONS ### ANNEX III: ASSESSMENT OF FOLLOW-UP OF IEO RECOMMENDATIONS ### 1. Weaknesses identified by the Lissakers Report - Absence of a mechanism to follow-up on IEO recommendations endorsed by the Executive Board undermines the IEO’s mandate to enhance the learning culture: “The real test of whether the IEO is having an impact on the learning culture of the IMF is whether or not lessons learned are incorporated into IMF practices” (p.23). - The Lissakers Report conclusion: “a more systematic approach is needed to follow-up on the recommendations of the IEO and monitor their implementation” (p.29). - Two follow-up options suggested in the Lissakers Report: - The IEO could periodically issue an evaluation of the follow-up and implementation of IEO recommendations endorsed by the Board. - The Evaluation Committee could play a more active role by scheduling meetings with Management or relevant Department Heads, together with the IEO team leader, to discuss follow-up plans and monitor progress. ### 2. The current follow-up mechanism (MIPs and PMRs) - Framework approved January 2007 requires: - Management provides the Board with a forward-looking implementation plan (MIP) for Board-endorsed recommendations, including a timetable of action and an estimate of their costs. - Management presents periodic monitoring reports (PMRs) to the Board on the state of implementation of actions in the MIPs; PMRs should indicate implementation difficulties and propose remedial or substitute actions. - Process description: - Draft Summing Up of Board discussions is prepared in advance by IMF staff (typically SPR), modified during Board meetings with Secretary’s Department input. - With SPR coordination, Management prepares the MIP based on the Summing Up; Management then prepares PMRs assessing implementation status of outstanding recommendations from MIPs. - Two salient features of the process: - Management (coordinated by SPR) is the main driver of drafting and monitoring proposed actions, though the Board is consulted (usually via the EVC). - The formulation of Board-endorsed recommendations influences follow-up: vague or policy-oriented recommendations are difficult to monitor; Summing Ups are often general and may be “not clear about what the Board has endorsed,” leading staff and Management to address recommendations only as part of on-going or planned processes. Proposed benchmarks sometimes consist of activities (e.g., preparing a policy paper) without explaining how they achieve requested policy or operational change. ### 3. Monitoring Implementation Plans (MIPs) - MIPs are prepared by Management and presented to the Board after the Board discussion of each IEO report. - Timing issues: - Time lag between Board discussion of an IEO report and the associated MIP release varies between 3 and 12 months. - Time lag between IEO report release and associated MIP discussion/approval by the Board can be even longer. - Example delays: - MIP on “IMF interaction with Member Countries” withdrawn so EVC could first meet to review it. - Second version of the MIP on the “Run-Up to the Financial and Economic Crisis” postponed until after the 2011 Triennial Surveillance Review; EVC required revisions and additional information; the MIP was issued to the full Board on February 14, 2012 — over a year after the report’s formal publication. - EVC role in MIP handling: - EVC discusses the first draft and decides whether the MIP needs revision and whether a full Board discussion is required or approval can occur on a lapse-of-time (LOT) basis. ### 4. Periodic Monitoring Reports (PMRs) - PMRs report on implementation status of Board-endorsed recommendations included in MIPs and typically include follow-up of recommendations “not deemed completed” from previous PMRs. - Characteristics and limitations: - PMRs are comprehensive but limited to specific Board-endorsed IEO recommendations; concerns raised by the IEO that are not explicitly endorsed by the Board may not be included in MIPs or PMRs. - Monitoring often discontinues after one-off actions are implemented, especially for policy recommendations where broader policy objectives are not tracked unless the Board agreed to a specific suggestion. - Evolution in coverage: - After EVC requested broader updates, Staff agreed to include comprehensive analysis of all Board-endorsed IEO recommendations in future PMRs. - The fifth PMR includes a consolidated picture of recent progress on all Board-endorsed recommendations made since the first PMR in 2007 and reviews most IEO evaluations since 2007. ### 5. Brief assessment — core flaws and risks - Lack of clear standards and effectiveness evaluation: - Monitoring and evaluation of implementation lack clear standards; there are currently no standards by which to monitor and improve follow-up. - Follow-up is limited to specific Board-endorsed recommendations and rarely evaluates effectiveness of the follow-up process. - Vague, policy-oriented recommendations that lack implementation guidance tend to be ignored; process-oriented recommendations risk becoming a tick-box exercise. - Including monitoring of policy-oriented outcomes (not just specific tasks) would reduce bureaucratic tick-boxing. - Management’s agenda-setting dominance and Board oversight constraints: - Management defines what issues are followed-up and how; the incentive problem is exacerbated by high turnover of Executive Directors and EVC members and significant time-lags between report discussion and MIP finalization. - The Lissakers Report noted that because Management drafts the Summing Up, it “may deem it premature or undesirable to lay a path for implementation of IEO recommendations, particularly if it disagrees” (p.26). - Recommendation to increase Board ownership: give responsibility for drafting the Summing Up to the Chair of the EVC — this should not add significant workload but could help ‘legitimize’ the follow-up process. - Recommendation to set a limit on the time lag between when a report is discussed by the Board and when the associated MIP is finally discussed or approved. - Transparency and external monitoring: - Lack of a single public location for MIPs and PMRs makes outside monitoring difficult. - Recommendation to create a link on the IEO’s website to all material related to follow-up of recommendations, including Summing Ups, to facilitate external monitoring. ### 6. Staff awareness and survey findings (summary highlights from Annex IV) - Standout survey result: significant decline in awareness and familiarity of IMF staff with the IEO and its work; there was an overwhelming volume of “don’t know” responses and growth in passive responses since the 2006 study. - Specific perceptions: - Around 60% believe the IEO is an impartial body. - Perceptions of independence have dropped slightly; perception of the IEO as a career-advancement mechanism has fallen. - Management implementation is regarded as weak and has deteriorated, with lower levels of interdepartmental collaboration. - Executive Board seen as most influential over IEO work; Senior Management perceived as having the least sway. - Staff ratings from 2006 to 2012 show declines in consultation and involvement with the IEO’s work (topic selection, review of initial analysis, general awareness of reports). - Perceived duplication and overlap with other functions has declined marginally overall; perception of overlap with the Office of Internal Audit has increased markedly. - Overall perception of recent change is mildly positive for the IEO on issues under its control; slight improvement in the IEO creating space to discuss policy alternatives, but no change in perceptions of organizational change and room for dissent. - Familiarity levels detail: - Staff reporting “not familiar at all” more than doubled to 21%. - Junior staff (A12-A13 grades) show the lowest familiarity and the largest decline between survey periods. - Senior staff (B1-B5) show a modest dip in familiarity. - By tenure: nearly half of staff with tenure of less than 5 years report being “not familiar at all” with the IEO; among staff with tenure between 5 and 10 years, more than 40% report some level of unfamiliarity. *Source: ANNEX III: ASSESSMENT OF FOLLOW-UP OF IEO RECOMMENDATIONS* ### 1.2 Staff Awareness ### 1.2 Staff Awareness ### Awareness and Familiarity - Both the “don’t know” and “intermediate” responses have shown marked increases compared to the 2006 study. - Very high levels of respondents recorded “don’t know” throughout the survey questionnaire, indicating lowered familiarity and low awareness of the activities and impact of the IEO. - High levels of claimed nescience may also indicate respondent apathy. - High incidence of “intermediate” responses (selecting the mid-point on a five-point Likert scale) is significantly higher than most human capital studies and may point to low staff commitment levels and diminished enthusiasm and/or morale. ### Perceptions of the IEO — Independence - The IEO is perceived to be fairly independent, with 59% of staff favoring that option. - Perception of independence marginally declined compared to 2006 (down from 65%). - Perception of independence increases with “familiarity” with the IEO: 74% of those familiar view the IEO as independent. - Staff reporting the IEO to be “very independent” remained unchanged at 16%. ### Perceptions of the IEO — IEO as a Place to Work - Those who would consider working for the IEO remained stable at 39%. - By grade: A9-A11 staff 42%; B1-B5 staff 29%. - Only 12% of the sample view working for the IEO as an opportunity to advance careers. - By grade for career advancement: junior staff 25%; senior staff 5%. - Broad consensus that the IEO provides challenging work and is advantageous in: - “broader experience” 69% - “enhancing analytical skills” 46% ### Perceptions of the IEO — Interactions with Other Departments - Marginal rise in those viewing their Department’s interaction with the IEO as “defensive” compared to 2006. - SPR reports the highest “defensive” levels at 29% (bottom 2-boxes) and 35% (top 2-boxes) view the relationship as collaborative. - Support/Special Services Group: 41% report a collaborative relationship and 12% a defensive relationship. - Junior staff view the relationship most collaboratively; senior staff view it least collaboratively. - The most dramatic and statistically significant change since 2006 is the increase in “intermediate” responses, growing from 16% to 41%. ### Perceptions of the IEO — Significance of Recommendations - Overall improvement in the seriousness with which IEO recommendations are received. - Senior staff most likely to report that Management takes IEO recommendations seriously. - SPR and Support/Special Service Departments report their Department Heads take IEO recommendations most seriously. ### Perceptions of the IEO — Culture and Implementation of Recommendations - Among all staff, 31% report that Management has been “effective” in implementing recommendations. - SPR reports highest levels of effectiveness in implementation at 49% (top 2-boxes). - The most common response was “moderate effectiveness” at 51%. - Only 17% believe the IEO is changing the IMF’s culture (marginally up from 16% in 2006). - Decline in those who think the IEO is not changing the IMF’s culture: from 44% in 2006 to 31% in 2012. - Increase in “don’t know” on culture change: from 40% in 2006 to 52% in 2012. ### Perceptions of the IEO — Website - Visitors to the IEO website fell marginally from 48% to 44%. - Those describing the website as useful dipped slightly. - Junior staff are least likely to have consulted the site; over 50% of B1-B5s have visited the site. ### Evaluation Process — Topic Selection - Perceived influence over IEO topic selection (ranked 1st by respondents): - IEO office 32% ranking 1st - senior Management 27% ranking 1st - Borrowing Members 17% ranking 1st - SPR judged to have the least influence over topic selection. ### Evaluation Process — Staff Consultation and Involvement - 29% report no “personal interaction”; therefore 71% of staff has had a firsthand connection with the IEO’s work (an increase from 61% in 2006). - Staff involvement in preparation of evaluations has declined in all areas, with largest declines in “reading early drafts” and “helping to prepare staff or Management responses.” - Perception of staff consultation on IEO evaluations dropped compared to 2006. ### Evaluation Process — IEO Report Utility - Usage of IEO reports declined from 54% to 46% since 2006. - Senior staff report heaviest use at 41%; junior staff the least at 31%. - Departmental use (heaviest consumers): SPR 71%; the Secretaries, OMD and DMD 63%. - Functional Departments are least likely to use the reports. - Around 67% of staff report having discussed IEO reports with colleagues (slightly down from 2006). - Discussion outside the Fund is flat at 22%. - Sharing of reports outside the Fund grew only with academia and non-borrowing members; all other external constituents show declines compared to 2006. ### Evaluation Process — Duplication of Work - Perceived overlap between IEO and SPR has marginally declined. - Perceived overlap with the Office of Internal Audit increased from 12% to 18%. ### Impact of IEO Evaluation — Effective Delivery of Priorities - Staff ranks priorities for the IEO and reports effectiveness: - “supporting institutional governance and oversight of the IMF’s role” — 40% deem IEO effective in fulfilling this role. - “enhancing a learning culture” — only 14% believe the IEO is effective in this role. - “strengthening the IMF’s external credibility” — 40% report effectiveness. - “promote greater understanding of the work of the IMF” — less than 3% believe this should be a priority; 23% report the IEO is effective in this area. ### Impact of IEO Evaluation — Publications and Influence - Significant increase from 2006 in those who believe IEO findings have encouraged policy discussion. - Dramatic fall in those who think findings have created room for dissent. - Most read/consulted specific report: IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004-07 (2011) at 28%. - Lowest readership: IMF Involvement in International Trade Policy Issues (2009) at 4%. - Nearly one-in-five claimed not to have read or consulted any of the reports listed. - Overall ratings of report quality showed marginal deterioration since 2006; “Topic Selection” largest decline from 64% to 52% (top-2 boxes). ### Impact of IEO Evaluation — Report Improvement and Influence - Largest area for improvement (ordinal ranking) is Management’s implementation of recommendations; it grew in prominence compared to 2006. - New in 2012: “staff buy-in of recommendations” identified as an area for potential improvement. - “Quality of recommendations” remains a key area for potential improvement. - Areas Where IEO Reports Could Be Most Improved — First choice: - Implementation of recommendations by Management 20.0% - Staff’s buy‐in of recommendations 19.8% - Quality of recommendations 16.0% - Analysis of the problem 14.4% - Greater objectivity 11.4% - Dissemination to wider public 9.0% - More consultation with stakeholders 7.8% - Writing style 1.8% - Total 100% - Perceived specific positive effects of IEO recommendations on IMF engagement: - 21% think recommendations significantly altered the effectiveness of IMF surveillance and the conditions attached by the IMF to future lending. - 20% report an effect on the content of IMF policy advice. - 12% estimate there has been an effect on IMF technical assistance. - Including those who see “some” effect, perception of contribution ranges from 54% to 68%. ### Impact of IEO Evaluation — Work Areas and Volume Preferences - 65% believe the IEO should increase investigation of “Best Practice.” - Views on current volume (About the Same / Less / More): - Country case studies: 54.0% / 14.5% / 31.5% - Investigation of best practices: 28.9% / 6.5% / 64.6% - Policy recommendations: 54.7% / 13.6% / 31.8% - Specific operational recommendations: 46.1% / 18.1% / 35.8% - Direct follow-up of Board-endorsed recommendations: 43.2% / 18.6% / 38.2% - Impact on individuals’ specific work: - Less than 10% think the IEO has significantly influenced their work (down from 13% in 2006). - 61% report a small or moderate effect. - Over 29% report “no influence at all.” ### IEO Change and Evolution - New battery of eight questions comparing 2012 to 2006. - Positive evolutions reported: - 25% have seen positive evolution in topic selection and overall quality. - 20% report IEO’s independence has seen a positive evolution; almost half “don’t know.” - 7% believe overall quality has deteriorated. - 17% have seen positive evolution in report accuracy. - Similar number view improvements in “technically sound analysis.” - How the IEO’s work evolved over the last 5 years (summary percentages): - Topic selection: Positive (4+5) 25.5% / Mixed (3) 21.3% / Negative (1+2) 6.4% / Don’t know 46.8% - Independence: Positive (4+5) 20.5% / Mixed (3) 22.7% / Negative (1+2) 6.8% / Don’t know 50.0% - Accuracy: Positive (4+5) 16.5% / Mixed (3) 25.3% / Negative (1+2) 6.7% / Don’t know 51.5% - Technically sound analysis of problems: Positive (4+5) 17.9% / Mixed (3) 25.8% / Negative (1+2) 8.1% / Don’t know 48.2% - Overall quality: Positive (4+5) 25.1% / Mixed (3) 23.0% / Negative (1+2) 6.8% / Don’t know 45.1% - No clear conclusion on IEO’s general impact on organizational change and room for dissent, though there is moderate positive assessment of contribution to discussion of policy alternatives. *Source: _010113 - 1.2 Staff Awareness* ### 27.1 Which report(s) was/were most useful to you? ### 27.1 Which report(s) was/were most useful to you? ### Survey instrument: questions on IEO reports, follow-up, and influence - Question 27.1: "Which report(s) was/were most useful to you?" - Question 28: Rate IEO reports in terms of: - Topic Selection; Independence; Accuracy; Technically sound analysis of problems; Feasibility of recommendations; Overall quality. - Response scale: Poor 1 2 3 4 Excellent 5; Don’t know/NA. - Question 29: Rank the three areas in which the IEO reports could be most improved (1 = First choice, 2 = Second choice, 3 = Third choice). Options: - Analysis of the problem; Greater objectivity; More consultation with stakeholders; Writing style; Quality of recommendations; Dissemination to wider public; Implementation of recommendations by management; Staffs Buy in of Recommendations. - Question 30 (New for 2012): For each of the following please rate whether the IEO should do more, do about the same or do less: - Country case studies; Investigation of best practices; Policy recommendations; Specific operational recommendations; Direct follow-up of Board-endorsed recommendations; Other (please specify below). - Question 30.1: Space to specify "Other." - Question 31: How would you rate IEO work in promoting the understanding of the Fund’s work: - Inside the Fund; Outside the Fund. - Response scale (implied): No change in understanding 1 ... Significantly increased understanding 5; Don’t Know/NA. - Question 32: To what extent have IEO findings and recommendations altered: - The content of IMF policy advice; The effectiveness of IMF surveillance; The effectiveness of IMF technical assistance; The conditions attached by the Fund to lending; The size of loans by the Fund. - Response scale: No Effect at All 1 ... Major Effect 5; Don’t Know/NA. - Question 33: How much has the work of the IEO office influenced your work? - Select one: 1 No Influence At All ... 5 Significant Influence. - Question 34: How has the work of the IEO affected your work? - Select one: 1 Significant Negative Influence (Answer question 34.2.) ... 5 Significant Positive Influence (Answer question 34.1.) - Question 34.1: How has your work been positively affected by the work of the IEO office? (free text) - Question 34.2: How has your work been negatively affected by the work of the IEO office? (free text) - Question 35 (New for 2012): How has the IEO’s work evolved in the following areas since 2006? - Topic Selection; Independence; Accuracy; Technically sound analysis of problems; Overall quality. - Response scale: Negative Evolution 1 ... Positive Evolution 5; Don’t Know/NA. - Question 36 (New for 2012): How has the IEO’s work affected the following areas since 2006? - IMFs organizational change; Space to discuss policy alternatives within the Fund; Room for dissent within the Fund. - Response scale: Negative Change 1 ... Positive Change 5; Don’t Know/NA. - Question 37: "Overall, in your opinion, how could the IEO be improved?" (free text) - Page markers in source: 70, 71, 72 (appear in the questionnaire pages). ### Observations about instrument content and structure - The questionnaire covers both quantitative rating scales and qualitative free-text questions. - Multiple items are flagged "*New for 2012*" (Questions 30, 35, 36). - The survey explicitly separates perceptions "Inside the Fund" and "Outside the Fund." - The follow-up and operational emphasis appears in questions about direct follow-up of Board-endorsed recommendations and on operational recommendations versus higher-level policy recommendations. ### Key numeric labels preserved from source - Page identifiers: 70; 71; 72. --- ### Statement by the Managing Director on the External Evaluation of the Independent Evaluation Office (Executive Board Meeting March 21, 2013) - Acknowledges extensive consultations underpinning the panel’s recommendations. - Agrees that the IEO has increased the integrity and quality of Fund work and is regarded as fully independent. - Accepts that tensions between Fund staff and the IEO are to some extent inevitable and that the relationship needs improvement. - Identifies three key challenges to uphold and promote IEO mandates (enhance learning culture; support institutional governance and oversight): - Encourage more interaction between Fund staff and the IEO: - Invite externally-recruited IEO staff to visit departments as observers. - Consider additional measures to facilitate mobility of “high performing” IMF staff to and from the IEO. - Keep IEO reports focused on long-term cross-cutting issues and ensure topic choice does not interfere with the Fund’s operational activities. - Refocus the follow-up process to Board-endorsed IEO recommendations on the broader policy objectives: - Endorses the summing up process as well-established practice. - Has asked staff for concrete measures to strengthen other areas of the follow-up process. - Concludes agreement with the panel’s broad conclusions and commitment to improve IEO effectiveness by using its recommendations. ### Staff Response to the External Evaluation of the Independent Evaluation Office (Executive Board Meeting March 21, 2013) - Staff view: IEO continues to make important contributions to the Fund’s learning culture and governance oversight. - Notes that IEO recommendations are highlighted in staff policy reviews (examples cited: recent Triennial Surveillance Review and the Review of Conditionality). - Survey results show staff perceive IEO recommendations are taken seriously—especially by Department Heads—and perception has improved since 2006. - On relations between Fund staff and IEO: - Accept recommendation to improve relations while noting some tension is natural and healthy. - Support more two-way interactions and in-reach from the IEO; suggest seminars during evaluation conduct and before report finalization. - Note such interactions were more prevalent initially but have fallen into disuse. - On topic selection: - Terms of reference give IEO a broad mandate but require avoiding interference with operational activities, including programs. - Staff view that limitation should continue to apply to avoid adversely affecting ongoing operational activities. - On focus and types of reviews: - Support keeping IEO reports on long-term cross-cutting issues; selective deeper case reviews acceptable if not interfering with operations. - Endorse periodic evaluations of a sample of internal ex post assessments (EPAs) and ex post evaluations (EPEs). - Emphasize IEO’s comparative advantage in identifying issues requiring changes to existing practices and that Management should propose how to operationalize solutions. - Warn against too many process-driven recommendations that lead to box-ticking follow-up. - On improving the follow-up process: - Support timely response to Board-endorsed recommendations and propose Management present the Management Implementation Plan (MIP) to the Evaluation Committee within three months following the Board discussion. - Support a biennial issues-oriented IEO review of the extent to which previous recommendations have been implemented, subject to IEO work program constraints. - Agree IEO presentations at the Annual Meetings have merit; an appropriate forum should be found. - On the Summing Up of Board discussions: - Staff view: preparation by the chair of the Evaluation Committee would be inconsistent with Fund governance where Management is chair of the Board. - Propose following standard practice: authoring office prepares first draft of the Summing Up; Secretary’s Department (SEC) would amend to reflect Directors’ statements and Board discussion; Executive Directors can request modifications grounded in the meeting record; SEC retains responsibility under Management’s direction. - On Periodic Monitoring Reports and involvement of OIA: - Continue annual preparation of Periodic Monitoring Reports. - Involvement of the Office of Internal Audit (OIA) requires careful consideration and consultation with the External Audit Committee; OIA mandate must be respected. - Note: factual comments will be conveyed separately. ### Independent Evaluation Office Response to the Report of the External Evaluation Team (March 18, 2013) - IEO welcomes the report’s clear and concise positive assessment. - Notes stakeholders view IEO reports as highly relevant and high quality; IEO has strengthened effectiveness, learning culture, external credibility and accountability of the IMF. - Committed to following through on recommendations and acknowledges areas for improvement: - Collaboration and engagement with IMF Management and Staff; timing of evaluations; follow-up process. - On awareness within the Fund: - Agrees there is room for greater awareness and knowledge of IEO findings and recommendations inside the Fund. - Plans to increase ‘in-reach’ activities among Board members and work with Management to reach Staff. - On access to information: - Confirms that during the review period the IEO was able to gain access to requested documents and information from Staff; looks forward to continued cooperative interactions. - On timing of evaluations: - Shares panel’s concern that the current constraint (avoid interfering in ‘current operations’) is not sufficiently clear. - Believes it's important to evaluate major issues while still relatively ‘fresh’ to produce lessons for future operations, balanced against not jeopardizing program success. - Supports constraints proposed by the external evaluation report (i.e., not to evaluate current lending programs) as adequate. - On follow-up process: - Agrees current follow-up has not been working properly and needs change. - Sees value in preparing a biennial “issues-oriented review” of progress on main reforms recommended in evaluation reports. - On the Summing Up of Board discussions: - Notes the Summing Up is a critical element and that the panel’s proposal that the EVC chair prepares it is not compatible with current practice. - Expresses concern that Staff’s proposal (IEO not preparing the Summing Up) may also pose practical difficulties and risk IEO being identified with a Summing Up that does not conform to its view. - Offers two suggestions to move forward: - Any new process should be reviewed by the Evaluation Committee after three Summings Up to assess effectiveness and propose corrective measures. - IEO could prepare the first draft of the Summing Up, but would work with SEC in preparing the final version after the Board discussion. - On EPAs and EPEs: - Prepared to produce a periodic assessment of a sample of EPAs and EPEs focusing on strategic and cross-cutting issues to improve internal self-assessments and program design. - On recommendation types: - Affirms IEO’s comparative advantage is to distill outcome-focused recommendations, leaving operationalization to Management and the Board. - Notes Executive Directors often ask for more specific recommendations; IEO seeks Directors’ views on balancing high-level vs. more specific recommendations. - On staff recruitment to IEO: - External evaluation notes difficulties attracting suitable IMF Staff for temporary assignments, largely due to IEO’s position and mandate. - IEO appreciates suggestions to create incentives and looks forward to Management and Staff ideas to implement such measures. - IEO expresses readiness to work with Board, Management and Staff to resolve practical difficulties in implementing external evaluation proposals, especially regarding follow-up. *Source: _010113 - 27.1 Which report(s) was/were most useful to you? (PDF).* ### 14. Finally, the IEO would welcome the opportunity to make a brief presentation to the ### _010113 - 14. Finally, the IEO would welcome the opportunity to make a brief presentation to the ### IEO presentation to IMFC / Annual Meetings - The IEO would welcome the opportunity to make a brief presentation to the IMFC during the Annual Meetings, as was the practice in the past. - Such a presentation could focus on major policy issues arising from recent evaluations that would be of concern to Ministers, and not necessarily cover the full gamut of IEO findings and recommendations. - Most Directors were open to considering an appropriate forum for the IEO to present its recent work during the Fund’s Annual Meetings; a number favored the idea of the IEO presenting a report in a meeting of the IMFC, while a number of others doubted its usefulness. ### Acknowledgements, follow-up, and periodic external evaluation - The IEO expressed appreciation to José Antonio Ocampo, Stephen Pickford, and Cyrus Rustomjee for their report, which will help guide the IEO going forward. - The external evaluation provided the IEO with an opportunity to reflect on its work and suggestions to enhance its effectiveness. - The IEO intends to present information about follow-up on key findings and recommendations of the external evaluation in future Annual Reports. - The IEO finds the process of periodic external evaluations extremely useful and would welcome a follow-up exercise in five years. ### Public Information Notice (PIN) and timing - PIN No. 13/40 - FOR IMMEDIATE RELEASE - March 29, 2013 - On March 21, 2013, the Executive Board of the International Monetary Fund (IMF) discussed the second External Evaluation of the Independent Evaluation Office (IEO). ### Background on the External Evaluation and IEO mandate - The IEO provides objective and independent evaluation on issues related to the IMF and operates independently of IMF management and the Executive Board. - The IEO was set up with four mandates: - to serve as a means to enhance the learning culture within the Fund; - to strengthen the Fund's external credibility; - to promote greater understanding of the work of the Fund throughout its membership; and - to support the Executive Board's institutional governance and oversight responsibilities. - The purpose of the evaluation was to assess the effectiveness of the IEO and to consider possible improvements to its structure, mandate, operational modalities, or terms of reference. - The independent team of experts was chaired by José Antonio Ocampo and included Stephen Pickford and Cyrus Rustomjee. - This was the second external evaluation of the IEO; the first, chaired by Karin Lissakers, was published in 2006. ### Executive Board Assessment — key findings and views - Directors welcomed the Panel’s report assessing the IEO’s effectiveness in the seven years since the last assessment. - Directors acknowledged the Panel’s extensive consultations, interviews, and interactions underpinning its analysis and recommendations. - Directors welcomed many recommendations aimed at enhancing the IEO’s effectiveness, including recommendations on: - evaluation topics; - follow-up processes; and - interactions with the Board, management, and staff. - Directors welcomed the Panel’s assessment that: - the IEO has played an important role in supporting the Fund’s governance and transparency and enhancing its learning culture; and - the IEO’s independence has been widely recognized and has strengthened the external perception of the Fund. - Most Directors concurred that the IEO’s objective of promoting greater understanding of the Fund’s work throughout the membership has become less important with the increased transparency of the Fund over the past decade, and therefore could be dropped as an element of the IEO’s mandate. ### Scope, topic selection, and focus of IEO reports - Directors agreed that IEO reports should focus on long-term cross-cutting issues and drawing out lessons of wider relevance and applicability for the advance of Fund policy and culture. - The IEO Director should continue to have full freedom in choosing subjects for evaluation, consistent with the IEO’s Terms of Reference. - Most Directors considered the current process for selecting evaluation topics appropriate. - Views on topic constraints: - A number of Directors saw scope for relaxing somewhat the current constraint that limits the choice of topics, and a number favored the Panel’s proposal to define that boundary as “current lending programs.” - A few Directors preferred not to modify the current Terms of Reference, which state that the IEO should avoid interfering with operational activities, including programs. - Directors generally agreed that, in framing recommendations, the IEO should focus on policy issues for the Fund rather than on processes, which are the responsibility and comparative advantage of management, though acknowledging practical difficulties in separating substance from process. - Many Directors considered it useful for the IEO to undertake, subject to resource availability, periodic evaluations of ex post assessments and ex post evaluations of selected country programs. ### Follow-up process and monitoring of recommendations - Directors agreed on the need to improve the follow-up process to Board-endorsed IEO recommendations. - They underscored the importance of strong ownership and active engagement by the Board, especially through its Evaluation Committee. - Roles and proposals: - The Evaluation Committee could review and monitor Management Implementation Plans (MIPs) and ensure their timeliness, including by setting time limits for preparation and submission of the MIPs. - Most Directors saw merit in regular IEO reviews of implementation of previous Board-endorsed recommendations, possibly every two years; a few others found such reviews inappropriate and the responsibility of the Board. - Many Directors were open to the Office of Internal Audit preparing reports on periodic monitoring of IEO recommendations, conditional on confirmation by the External Audit Committee; other Directors were not in favor. - Interaction between the IEO and the membership is essential for follow-up; most Directors were open to considering an appropriate forum for the IEO to present recent work during Annual Meetings. ### Governance, Board discussions, and summings up - With regard to Board discussions of IEO reports, most Directors did not see a need for a major change in the governance structure where management functions as chair of the Board and the Secretary’s Department finalizes the record of Board meetings. - Most Directors did not favor the Panel’s suggestion that the Chair of the Evaluation Committee be responsible for drafting the record of Board discussions of IEO reports, though some saw merit. - Many Directors supported, or were open to, the IEO preparing draft summings up for Board discussions of its reports and working with the Secretary’s Department in preparing the final version, in line with standard procedures for all other summings up. - Many Directors noted a lack of understanding on how Directors’ silence on specific IEO recommendations is interpreted in recording the outcome of the Board discussion, suggesting that the same treatment of silence should apply as in other Board meetings. ### Raising IEO profile, staff mobility, and budget considerations - Directors noted Panel recommendations for raising the profile of the IEO within the Fund to increase effectiveness. - They emphasized the need for enhanced dialogue between the IEO and Fund staff without compromising IEO independence, including through ‘in-reach’ activities such as internal seminars and discussions of recommendations. - Directors welcomed management’s intention to consider measures to facilitate mobility of high-performing staff to and from the IEO. - A number of Directors were open to considering an increase in the budget for the IEO to take on additional activities as proposed; a number of others did not see a clear case for a budget increase. ### Follow-up responsibilities and next steps - Recommendations that received broad support and outstanding issues that warrant further consideration will be followed up by the appropriate parties—the Evaluation Committee, the IEO, staff, and management. - Directors would have further opportunities to discuss concrete proposals in the coming months. *International Monetary Fund — Public Information Notice No. 13/40, March 29, 2013. (Executive Board discussion held March 21, 2013.)* --- _Source: https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/np/pp/eng/2013/_010113.pdf_