## MRLIC programme review — By March 2018

## Source details

**Canonical URL:** [MRLIC programme review — By March 2018](https://www.imf.org/-/media/files/topics/lics/macro-research-for-development/documents/june2018.pdf)

## Other formats

- [Markdown version](/-/media/files/topics/lics/macro-research-for-development/documents/june2018.pdf.md)
- [Structured JSON version](/-/media/files/topics/lics/macro-research-for-development/documents/june2018.pdf.json)

---

### Research production — Working papers and publications
- Working papers (Indicator 1.1)
  - Targets by March 2018: H (90 papers), M (86 papers), L (82 papers), A++
  - Achievement:
    - 19 produced in 2017/18
    - 94 cumulative (108% of high target)
  - Core research areas covered:
    - Monetary and exchange rate policies; Public investment, growth, and debt sustainability; Macroeconomic management of natural resources; Macroeconomic policies and income distribution; Financial deepening for macroeconomic stability and sustained growth; Growth through diversification; Gender and macroeconomics; Capital flows.
  - Selected completed working papers (2017/18):
    - Choi, S. et al. (2017), Oil Prices and Inflation Dynamics: Evidence from Advanced and Developing Economies, IMF Working Paper 17/196
    - Choi, S., Furceri, D., and Tovar Jalles, J. (2017), Fiscal Stabilization and Growth: Evidence from Industry-level Data for Advanced and Developing Economies, IMF Working Paper 17/198
    - Eicher, T. et al. (2018), Forecasts in Times of Crises, IMF Working Paper 18/48
    - Melina, G. and Portillo, R. A. (2018), Economic Fluctuations in Sub-Saharan Africa, IMF Working Paper 18/40
    - Furceri, D. et al. (2018), The Distributional Effects of Government Spending Shocks in Developing Economies, IMF Working Paper 18/57

- Peer-reviewed publications (Indicator 1.2)
  - Targets by March 2018: H (44 paper), M (42 papers), L (40 papers), A++
  - Achievement:
    - 11 published in 2017/18
    - 48 published in total (104% of high target)
  - Notable peer-reviewed outputs:
    - Shimeles, A., Zerfu Gurara, D. and Woldeyes, F. (2017) Taxman's Dilemma: Coercion or Persuasion? Evidence from a Randomized Field Experiment in Ethiopia, American Economic Review, vol.107(5), pp. 420-24 (open access)
    - Furceri, D., Loungani, P. and Zdzienicka, A. (2018), The effects of monetary policy shocks on inequality, Journal of International Money and Finance, vol. 85, pp. 168-186
    - Choi, S. et al., (2018). Oil prices and inflation dynamics: Evidence from advanced and developing economies, Journal of International Money and Finance, vol. 82(C), pages 71-96
    - Eicher, T. et al. (2018), Forecasts in Times of Crises, International Journal of Forecasting, forthcoming

- Freely available books (Indicator 1.3)
  - Targets by March 2018: H (2 books), M (2 books), L (2 books), A
  - Achievement:
    - No book published in 2017/18
    - 2 published in total (100% of target)
  - Book chapters in 2017/18 included:
    - Melina, G. and Portillo, R. (2018), Economic Fluctuations in Sub-Saharan Africa, in Berg, A. and Portillo, R. (eds), Monetary Policy in Sub-Saharan Africa, Oxford University Press, Oxford
    - Duttagupta, R. et al. Globalisation and Inclusive Growth: Can They Go Hand in Hand in Developing Countries?, XXIX Villa Mondragone International Economic Seminar proceeding, forthcoming.

### Uptake and application of research (Output 2)
- Overall output score: A++ (Impact weighting 30%)
- Indicator 2.1 — Application and use by IMF country teams
  - Targets by March 2018: H (77), M (73), L (70), A++
  - Achievement:
    - 21 applications in 2017/18
    - Cumulatively 85 (110% of the high target)
  - Uses and applications:
    - Support to development of targeted FPAS models, Article IV missions, Selected Issues Papers
    - Application of LIC-specific DIG model and DIGNAR model (Natural Resources) in policy design and World Economic Outlook simulations
    - Diversification toolkit applied to Sub-Saharan Africa, Guinea-Bissau, and Vanuatu
    - DIG model applied in Solomon Islands, Madagascar, Democratic Republic Timor-Leste, Lesotho, Maldives, and in the World Economic Outlook (October 2017)
    - Income and gender inequality toolkits applied to Benin, Dominica, Myanmar; gender budgeting toolkit applied in Nigeria, Sri Lanka, Rwanda

- Indicator 2.2 — Application and use by country authorities
  - Targets by March 2018: H (19), M (18), L (17), A++
  - Achievement:
    - 13 applications in 2017/18
    - Cumulatively 29 (153% of the high target)
  - Examples:
    - Development and bilateral support for country FPAS models in Tanzania, Mozambique, Uganda, Kenya, Ghana, Rwanda
    - Participation in EAC Technical Working Group on FPAS (December 2017) and engagement with the EAC Monetary Affairs Committee

- Indicator 2.2.1 — Courses offered to country authorities
  - Targets by March 2018: H (3), M (2), L (1), A++
  - Achievement:
    - 9 courses in 2017/18
    - Cumulatively 9 (300% of the high target)
  - Examples:
    - Peer Learning Conference on Gender Equality in Kigali (November 2017): 150 representatives from 22 sub-Saharan African nations attended
    - Support to Bank of Uganda modelling team, Article IV applications in Vanuatu, customised training missions to Malawi and Mozambique
    - Nine workshops/courses on toolkits and applications held in various countries and settings

### High-level conferences and outreach (Section 3.1)
- Conference activity and metrics
  - Eight high-level conferences in 2017/18; cumulative total 30 (115% of the high target of 26)
  - Two conferences focused on gender budgeting and gender inequality: World Government Summit Session on Gender Budgeting (co-organised with the UAE Gender Balance Council) and Gender Equality: From Theory to Practice Conference (co-organised with Rwanda’s Ministry of Gender and Family Promotion)
  - Conference topics also included financial inclusion, capital flows, forecasting issues, inclusive growth, and future challenges for LICs
  - “Towards 2030: Trends, Opportunities, Challenges, and Policies for Inclusive Growth” at IMF Annual Meeting 2017 presented research on monetary policy and credit dynamics in LICs and on gender budgeting; attended by senior IMF staff including Tao Zhang; keynote speakers Dani Rodrik and Ricardo Hausmann
  - One-day conference “Financial Inclusion: Drivers and Real Effects” at IMF Headquarters chaired by Rachel Glennerster with participation from Asli Demirguc-Kunt, Patrick Njoroge, Rohini Pande, Ratna Sahay, and opening remarks by Maurice Obstfeld

- Attendance and presentations
  - 35 presentations at high-level policy conferences in 2017/18; cumulative presentations 86 (141% of the high target)
  - Four presentations at the Centre for the Study of African Economies (CSAE) conference at Oxford University

- IMF board and policy papers
  - MRLIC research utilised in five IMF Board Papers in 2017/18 covering anti-corruption, gender, climate change, inequality and macroeconomic trends; cumulative total 16 (127% of the high target)
  - MRLIC research or analysis featured in five IMF policy papers in 2017/18 (Fiscal Monitor, World Economic Outlook, Africa Regional Economic Outlook, Asia-Pacific Regional Economic Outlook); cumulative total 24 (104% of the high target)
  - IMF Board Meeting on Emerging Issues (November) featured presentation of two pilots (Malawi and Ethiopia) as good practice; meeting chaired by Christine Lagarde

- Recommendations linked to outreach
  - DFID should evaluate the possibility of DFID representatives attending IMF bi-annual meetings hosting seminars showcasing MRLIC research
  - Proactively present MRLIC research to DFID senior staff, leveraging MRLIC staff travel schedules to facilitate briefings

### Dissemination channels, metrics and lessons (Section 4.3)
- Dissemination channels used
  - IMF newsletters and magazines (Finance & Development, IMF Research Bulletin)
  - IMF Blog and IMF intranet
  - Presentations and TED-style talks (IMF Analytical Corner at IMF Spring Meetings; talk on gender budgeting during IMF 2018 Spring Meetings)
  - Media coverage in The Wall Street Journal, Bloomberg, Deutsche Welle, Human Rights Watch, CNBC videos
  - Articles on specialised websites such as VoXEU

- Lessons and action points on dissemination
  - Maintain strong links with prominent external researchers to ensure quality and outreach, while recognising commissioning challenges when external incentives differ from IMF staff priorities
  - Reassess commissioning process and correct logframe target for commissioned research; deadline: by December 2018
  - Track indicators for dissemination products (views and downloads of newsletters, blogs and videos) by end December 2018 to quantify reach and impact

### Theory of Change, progress toward outcomes, and logframe changes
- Implicit Theory of Change logical framework:
  - OUTPUT 1: High quality, policy relevant research on macroeconomic issues affecting LICs produced
  - OUTPUT 2: IMF research products produced under this project used by IMF country teams and partner authorities
  - OUTPUT 3: Engagement by senior IMF policymakers on LIC issues strengthened
  - OUTPUT 4: IMF strengthens research capacity by expanding the network of researchers and policymakers working on LIC macroeconomics
  - OUTCOME: Better engagement by IMF policymakers on LIC-specific macroeconomic issues, leading to improved policy-making in project thematic areas
  - IMPACT: Better macroeconomic policymaking in LICs leading to faster economic growth, job creation and poverty reduction
- Status and decisions:
  - ToC remains valid; decision to partner directly with the IMF to maximise early engagement and shape research agenda by demand for policy advice
  - DFID and MRLIC will set out a more detailed ToC by end October 2018
- Progress and uptake highlights:
  - Direct uptake by IMF country teams and country authorities more than doubled in the year
  - MRLIC models and analyses used in a number of IMF Article IV country reports
  - FPAS model adopted in a co-ordinated fashion by member states of the East African Community
  - Numbers of papers presented to the IMF Board doubled in the year
  - Suggested: DFID and MRLIC should reflect on channels more and less effective in generating policy impact as part of drafting the new ToC

- Logframe changes and planned adjustments:
  - Phase 3: targets adjusted upwards based on years 1-5 average, then adjusted down pro-rata for budget cut to set “medium” targets
  - New Phase 3 logframe targets added: 1.3 (Number of freely available books), 2.2.1 (Courses offered to country authorities), 4.4.1 (Number of data toolkits)
  - Planned changes: reset targets under output 2 in line with observed performance; increase targets for 2019 and 2020 for outputs 2.2 and 2.2.1; effort needed to reach 2019 targets on outputs 1, 3 and 4
  - Recommendation: produce figures on views, downloads and citations of MRLIC papers and dissemination products by end December 2018; DFID to consider adding such figures as a new logframe target

### Value for Money (VfM), costs and financial performance
- Unit cost performance and interpretation
  - Programme achieving a unit cost of just over $87,000 for working papers, substantially lower than Business Case expectations
  - Business Case unit cost per research paper: $193,600 for the IMF programme and $288,000 for the next best option
  - Interpretation: lag in working paper acceptance implies cost per published paper expected to decrease over time

- Cost allocation and unit costs (Years 1-6 / life-to-date)
  - Total Staff/Contractual/VS Costs $10,916,414
    - Research Paper Costs $8,187,311
    - Country Applications Costs $2,729,104
  - Working Papers Produced 94
  - Cost Per Working Paper $87,099
  - Published Papers Produced 48
  - Cost per Published Paper $170,569
  - Assumption: staff and visiting scholars spend 75 percent of their time on research papers and 25 percent on country applications, yielding approximate per-paper costs of $87,000 (£66,000) and $170,000 (£130,000)

- Project expenditures Years 1-6 (Activity Total, Years 1-6)
  - HQ led missions including entire mission team $6,529,504
  - Short-term Advisors - CD delivery $477,102
  - Research HQ based / Visiting Scholars $3,909,808
  - Seminars & Study Tours $1,510,683
  - Project Backstopping $2,250,403
  - Project Management $161,009
  - Exceptional Expenses $146,730
  - Trust Fund Fee $1,048,967
  - Total $16,034,206
  - As of April 2018, $16.0 million has been drawn down from the subaccount; figures reflect a lag of approximately two to three months to verify expenses before charging donor subaccounts

- VfM rationale
  - Economy: programme management costs approximately 8% of total spend (1% direct management costs + 7% IMF trust fund management fee)
  - Efficiency: outputs produced under firm timeframes; 48 publications in peer-reviewed journals
  - Effectiveness: MRLIC team primarily IMF staff can communicate research findings quickly to IMF country staff
  - Sustainability: research uptake budget reduced by 50% this year while IMF departments subsidise overseas missions; fewer visiting researchers needed due to established network

### Risk overview and mitigation
- Overall risk rating: Minor (each risk category also rated Minor)
- Risk categories and notes:
  - External context: most research carried out at IMF headquarters using secondary datasets, limited exposure to LIC contexts
  - Delivery: established delivery systems, demonstrated high delivery; demand remains high
  - Operational: experienced management teams at DFID and IMF
  - Fiduciary: all funds spent directly by the IMF and subject to extensive IMF financial management controls
  - Reputational: little or no reputational risk
  - Innovation: not subject to substantial innovation risks
  - Safeguarding: activities rarely involve contact with vulnerable persons; external researchers managed per IMF procedures
- New potential risks from change in delivery model:
  - Sourcing funding from other IMF departments and cutting funds for senior researchers alters operating model
  - Actions:
    - DFID and MRLIC to update the risk matrix by end September 2018; risk rating expected to remain Minor
    - MRLIC programme team to develop a risk register by end July 2018 in line with DFID smart rules

### Monitoring, evidence, learning and governance
- Monitoring and reporting
  - Programme monitors numbers of papers and conference presentations through staff reports
  - Policy uptake monitored through MRLIC staff engagement on IMF country missions and Board engagement via internal management communications
  - Programme meetings monthly; logframe target updates compiled every four months and shared with DFID
  - DFID monitors progress through bi-annual meetings and informal contact between DFID SRO and IMF programme staff
  - Recommendation: generate statistics on numbers of views, downloads and citations of MRLIC research products, and track feedback on training events; MRLIC/DFID to produce such statistics by end December 2018

- Evidence and learning
  - Fundamental evidence for need of programme unchanged from original business case; evaluation not applicable
  - MRLIC operates monthly meetings to take stock and address delays or blockages; no formal system for logging lessons learned—recommend management team consider whether such a system would add value

- Progress on prior recommendations
  - From 2017 Annual Review:
    - Logframe revised at beginning of Phase 3 with targets raised (progress)
    - MRLIC began developing evidence products that distil key lessons; “How to” notes published for gender and inequality in IMF work; two more notes planned for 2019 (progress)
    - No progress on establishing a programme steering committee; suggestion to consider a panel of senior academics and policymakers for oversight

### Selected lessons identified and policy recommendations (selected)
- Research production and dissemination
  - Maintain numbers of junior or mid-ranking researchers (assistants and associates) while cutting funds for hiring visiting senior researchers; still allocate some resources for hosting external researchers and maintaining collaborations with key experts to support research quality and programme profile
  - Produce policy briefs and "How-to" notes; MRLIC produced longer “How-to” notes and two policy briefs/synthesis papers on gender issues and income inequality published on the IMF website in June 2018; MRLIC should continue to publish policy briefs and how-to notes for dissemination to policy-makers
  - Keep a record of the decision process on topics for research and publications to facilitate hand-over, increase transparency, and build know-how; discuss the role of a steering committee at the next bi-annual DFID-IMF management meeting
  - Monitor views, downloads and citations for publicly available papers through access logs and online tools to quantify impact and visibility; monitoring should be in place by end December 2018

- Uptake and operational model
  - MRLIC cut funding for country missions by 50% and replaced funds with direct funding from IMF country teams; DFID and MRLIC should continue to monitor effects of this operating model change
  - Where MRLIC models have been used for policy advice, underlying analysis was often delivered by an MRLIC staff member—this ensures quality control but constrains breadth of uptake; MRLIC and DFID should consider the long-term plan for relinquishing direct control over use of tools
  - DFID should initiate a dialogue with the IMF to benefit from MRLIC’s knowledge and experience on knowledge sharing and uptake of research by country teams
  - Strengthen tracking of feedback from participants in training and use it to improve delivery; compile such information by end December 2018, leveraging IMF Institute for Capacity Development (ICD) feedback systems

- Governance and annual review
  - Annual review conducted by a DFID A2 Economic Adviser and B1 Assistant Economist independent of the programme, supported by programme manager and Senior Responsible Owner; quality assurance by the A1 manager of DFID Growth Research Team

*Source: IMF/DFID MRLIC annual review content unit "4.3 Project Outputs disseminated" from June 2018 materials.*

### 1.1 Number of priority research papers produced, on the

### 1.1 Number of priority research papers produced, on the following research topics

### Research production — Working papers (Indicator 1.1)
- Targets by March 2018: H (90 papers), M (86 papers), L (82 papers), A++
- Achievement:
  - 19 produced in 2017/18
  - 94 cumulative (108% of high target)
- Additional detail:
  - "MRLIC researchers completed 19 working papers this year, bringing the total to 94 over the life of the programme to date. This exceeds the high target for this year of 90."
  - Working papers cover the core areas: Monetary and exchange rate policies; Public investment, growth, and debt sustainability; Macroeconomic management of natural resources; Macroeconomic policies and income distribution; Financial deepening for macroeconomic stability and sustained growth; Growth through diversification; Gender and macroeconomics; Capital flows.
- Examples of working papers completed in 2017/18 (selected from the programme list):
  - Choi, S. et al. (2017), Oil Prices and Inflation Dynamics: Evidence from Advanced and Developing Economies, IMF Working Paper 17/196
  - Choi, S., Furceri, D., and Tovar Jalles, J. (2017), Fiscal Stabilization and Growth: Evidence from Industry-level Data for Advanced and Developing Economies, IMF Working Paper 17/198
  - Eicher, T. et al. (2018), Forecasts in Times of Crises, IMF Working Paper 18/48
  - Melina, G. and Portillo, R. A. (2018), Economic Fluctuations in Sub-Saharan Africa, IMF Working Paper 18/40
  - Furceri, D. et al. (2018), The Distributional Effects of Government Spending Shocks in Developing Economies, IMF Working Paper 18/57

### Peer-reviewed publications (Indicator 1.2)
- Targets by March 2018: H (44 paper), M (42 papers), L (40 papers), A++
- Achievement:
  - 11 published in 2017/18
  - 48 published in total (104% of high target)
- Additional detail:
  - "This year MRLIC published 9 papers in peer-reviewed journals, plus 2 chapters in books following a similar peer-review process (11 in total for 2017/18). This brings the total to 48 over the life of the programme to date. This exceeds the high target of 44."
  - Notable journals include the Journal of Development Economics, Economica, Journal of International Money and Finance, International Journal of Forecasting, and the American Economic Review (AER).
  - Specific peer-reviewed outputs listed:
    - Shimeles, A., Zerfu Gurara, D. and Woldeyes, F. (2017) Taxman's Dilemma: Coercion or Persuasion? Evidence from a Randomized Field Experiment in Ethiopia, American Economic Review, vol.107(5), pp. 420-24 (open access)
    - Furceri, D., Loungani, P. and Zdzienicka, A. (2018), The effects of monetary policy shocks on inequality, Journal of International Money and Finance, vol. 85, pp. 168-186
    - Choi, S. et al., (2018). Oil prices and inflation dynamics: Evidence from advanced and developing economies, Journal of International Money and Finance, vol. 82(C), pages 71-96
    - Eicher, T. et al. (2018), Forecasts in Times of Crises, International Journal of Forecasting, forthcoming

### Freely available books (Indicator 1.3)
- Targets by March 2018: H (2 books), M (2 books), L (2 books), A
- Achievement:
  - No book published in 2017/18
  - 2 published in total (100% of target)
- Additional detail:
  - "No books were published this year. However, the programme has already published two freely available books, so is already meeting expectations for this stage in the programme."
  - Two book chapters were published in 2017/18 and are included under peer-reviewed outputs:
    - Melina, G. and Portillo, R. (2018), Economic Fluctuations in Sub-Saharan Africa, in Berg, A. and Portillo, R. (eds), Monetary Policy in Sub-Saharan Africa, Oxford University Press, Oxford
    - Duttagupta, R. et al. Globalisation and Inclusive Growth: Can They Go Hand in Hand in Developing Countries?, XXIX Villa Mondragone International Economic Seminar proceeding, forthcoming.

### Uptake and application of research (Output 2)
- Output score: A++ (Impact weighting 30%)
- Indicator 2.1 — Application and use by country teams:
  - Targets by March 2018: H (77), M (73), L (70), A++
  - Achievement:
    - 21 applications in 2017/18
    - Cumulatively 85 (110% of the high target)
  - Examples:
    - Applications covered four broad topics: Monetary and Exchange Rate Policies; Public investment and debt sustainability; Growth through Diversification; Income and Gender Inequality.
    - Use cases include support to development of targeted FPAS models, Article IV missions, Selected Issues Papers, and application of the LIC-specific DIG model and DIGNAR model (Natural Resources) in policy design and World Economic Outlook simulations.
    - Diversification toolkit applied to Sub-Saharan Africa, Guinea-Bissau, and Vanuatu.
    - DIG model applied in Solomon Islands, Madagascar, Democratic Republic Timor-Leste, Lesotho, Maldives, and in the World Economic Outlook (October 2017).
    - Income and gender inequality toolkits applied to Benin, Dominica, Myanmar; gender budgeting toolkit applied in Nigeria, Sri Lanka, Rwanda.

- Indicator 2.2 — Application and use by country authorities:
  - Targets by March 2018: H (19), M (18), L (17), A++
  - Achievement:
    - 13 applications in 2017/18
    - Cumulatively 29 (153% of the high target)
  - Examples:
    - Development and bilateral support for country FPAS models in Tanzania, Mozambique, Uganda, Kenya, Ghana, Rwanda.
    - Participation in EAC Technical Working Group on FPAS (December 2017) and engagement with the EAC Monetary Affairs Committee.

- Indicator 2.2.1 — Courses offered to country authorities:
  - Targets by March 2018: H (3), M (2), L (1), A++
  - Achievement:
    - 9 courses in 2017/18
    - Cumulatively 9 (300% of the high target)
  - Examples:
    - Co-organisation of a Peer Learning Conference on Gender Equality in Kigali (November 2017): 150 representatives from 22 sub-Saharan African nations attended.
    - Support to Bank of Uganda modelling team, Article IV applications in Vanuatu, customised training missions to Malawi and Mozambique.
    - Nine workshops/courses on toolkits and applications held in various countries and settings.

### Lessons identified and recommendations (selected)
- Research production and dissemination
  - Recommendation 1: Maintain numbers of junior or mid-ranking researchers (assistants and associates) while cutting funds for hiring visiting senior researchers; still allocate some resources for hosting external researchers and maintaining collaborations with key experts to support research quality and programme profile.
  - Recommendation 2: Produce policy briefs and "How-to" notes; MRLIC produced longer “How-to” notes and two policy briefs/synthesis papers on gender issues and income inequality published on the IMF website in June 2018; MRLIC should continue to publish policy briefs and how-to notes for dissemination to policy-makers.
  - Recommendation 3: Keep a record of the decision process on topics for research and publications to facilitate hand-over, increase transparency, and build know-how; discuss the role of a steering committee at the next bi-annual DFID-IMF management meeting.
  - Recommendation 4: Monitor views, downloads and citations for publicly available papers through access logs and online tools to quantify impact and visibility; monitoring should be in place by end December 2018.

- Uptake and operational model
  - Recommendation 1: MRLIC cut funding for country missions by 50% and replaced funds with direct funding from IMF country teams; DFID and MRLIC should continue to monitor effects of this operating model change.
  - Recommendation 2: Where MRLIC models have been used for policy advice, underlying analysis was often delivered by an MRLIC staff member—this ensures quality control but constrains breadth of uptake; MRLIC and DFID should consider the long-term plan for relinquishing direct control over use of tools.
  - Recommendation 3: DFID should initiate a dialogue with the IMF to benefit from MRLIC’s knowledge and experience on knowledge sharing and uptake of research by country teams.
  - Recommendation 4: Strengthen tracking of feedback from participants in training and use it to improve delivery; compile such information by end December 2018, leveraging IMF Institute for Capacity Development (ICD) feedback systems.

*MRLIC programme review, By March 2018.*

### 3.1 High level policy conferences

### 3.1 High level policy conferences

### High-level conferences and outreach
- Research produced within the programme was presented and discussed at eight high-level conferences in 2017/18, bringing the overall total to 30. This exceeds the high target of 26.
- Two conferences focused on gender budgeting and gender inequality:
  - World Government Summit Session on Gender Budgeting, co-organised with the UAE Gender Balance Council.
  - Gender Equality: From Theory to Practice Conference, co-organised with Rwanda’s Ministry of Gender and Family Promotion.
- Other conference themes included financial inclusion, capital flows, forecasting issues, inclusive growth, and future challenges for LICs.
- “Towards 2030: Trends, Opportunities, Challenges, and Policies for Inclusive Growth” (part of the IMF Annual Meeting 2017) presented research on monetary policy and credit dynamics in LICs and on gender budgeting; attended by senior IMF staff, including IMF Deputy Managing Director Tao Zhang; featured keynote speeches from Dani Rodrik and Ricardo Hausmann.
- A one-day conference “Financial Inclusion: Drivers and Real Effects” at IMF Headquarters brought together scholars and a policy panel chaired by Rachel Glennerster (DFID Chief Economist) with participation from Asli Demirguc-Kunt (World Bank Director of Research), Patrick Njoroge (Governor, Central Bank of Kenya), Rohini Pande (Professor of Public Policy), Ratna Sahay (Deputy Director, IMF), and opening remarks by Maurice Obstfeld (Director, IMF Research Department).

### Metrics and scores for conference-related outputs
- By March 2018: H (26) M (25) L (24) — score A++.
- 8 conferences in 2017/18.
- Cumulative total: 30 (115% of the high target).

### Attendance of external researchers and policy makers at high-level conferences
- The programme produced 35 presentations at high-level policy conferences in 2017/18.
- Four presentations were at the Centre for the Study of African Economies (CSAE) conference at Oxford University.
- Cumulative presentations: 86 (141% of the high target).
- Indicator milestones: By March 2018, H (61) M (58) L (56) — score A++.

### Results reflected in IMF board discussions
- Results of MRLIC research were utilised in five IMF Board Papers in 2017/18, covering anti-corruption, gender, climate change, inequality and macroeconomic trends.
- By March 2018: H (13) M (12) L (12) — score A++.
- 5 pieces in 2017/18.
- Cumulative total: 16 (127% of the high target).
- IMF senior management (Christine Lagarde, Tao Zhang, Maurice Obstfeld) provided positive feedback and support for the programme.

### Results reflected in IMF policy papers
- MRLIC research, or analysis from MRLIC models, featured in five IMF policy papers in 2017/18, including the Fiscal Monitor, the World Economic Outlook, the Africa Regional Economic Outlook, and the Asia-Pacific Regional Economic Outlook.
- The IMF Board Meeting on Emerging Issues in November featured presentation of the two pilots (Malawi and Ethiopia) as good practice; the meeting was chaired by Christine Lagarde.
- By March 2018: H (23) M (22) L (21) — score A++.
- 5 pieces in 2017/18.
- Cumulative total: 24 (104% of the high target).

### Lessons identified and policy recommendations
- Recommendation 1: DFID should evaluate the possibility of DFID representatives attending the IMF bi-annual meetings where seminars showcasing the research are hosted.
- Recommendation 1 (continued): The review team suggests actively looking for opportunities to present MRLIC research to DFID senior staff (e.g., by taking advantage of MRLIC staff travel schedules to bring them to DFID offices in UK and elsewhere).

### Related output metrics in the IMF strengthens research capacity output
- Output number per LF: 4. Output Score: A+. Impact weighting: 15% (revised since last AR? Yes, increased 5%).
- 4.1 Number of commissioned research papers from new researchers:
  - By March 2018, H(24) M(23) L (22) — score A++.
  - 5 produced in 2017/18.
  - Cumulative: 25 (104% of the high target).
  - Commissioned papers in 2017/18:
    - Iyer, T. (2017), Exchange Rate Choices with Inflexible Markets and Costly Price Adjustments, IMF Working Paper 17/154
    - Stuart, E., Samman, E., and Hunt, A. (2017), Informal is the new normal: Improving the lives of workers at risk of being left behind, ODI Working Paper 530
    - Eicher, T. et al. (2018), Forecasts in Times of Crises, IMF Working Papers 18/48
    - Jedwab, R., Loungani, P. and Yezer, A. (2017), Modern Urban Technology and the Future of the New Urban Giants, mimeo
    - Atolia, M. et al. (2017), Premature Deindustrialization, Structural Transformation, and Economic Development: Review and Policy Analysis, mimeo
- 4.1.1 Data toolkits:
  - By March 2018, H (6) M (6) L (5) — score A.
  - 1 toolkits produced in 2017/18, 1 updated.
  - Cumulative: 6 (equal to the high target).
  - The programme released two toolkits in 2017/18: the inequality toolkit (targeted at LICs and applied to the 2017 Article IV Consultation for Benin) and an updated version of the diversification toolkit (the update is discounted for scoring because the original version was already counted).
- 4.2 Attendance of external researchers and policy makers at high-level policy conferences:
  - By March 2018, H (61) M (58) L (56) — score A++.
  - 35 presentations at conferences in 2017/18.
  - Cumulative: 86 (141% of the high target).
- 4.3 Project outputs disseminated in e-newsletter and updated web-page and R4D:
  - By March 2018, H (22) M (21) L (20) — score A+.
  - 4 updates in 2017/18.
  - Cumulative: 22 (equal to the high target).

*The output scoring conclusion: MRLIC substantially exceeded two targets, somewhat exceeded the third for the 3.1–3.3 cluster; overall MRLIC performance on the research-capacity output is assessed A+.* 

*Source: IMF MRLIC programme reporting.*

### 4.3 Project Outputs disseminated

### 4.3 Project Outputs disseminated

### Dissemination channels and coverage
- Further dissemination of the research was achieved through:
  - IMF newsletters and magazines, such as Finance & Development and the IMF Research Bulletin
  - blogs, such as a blog on infrastructure on the IMF Blog and the IMF intranet
  - presentations and TED-style talks, such as the IMF Analytical Corner at the IMF Spring Meetings, and a talk on gender budgeting during the IMF 2018 Spring Meetings
  - Media coverage and news articles, with articles featured on The Wall Street Journal, Bloomberg, Deutsche Welle, Human Rights Watch, CNBC videos, and others
  - Articles on specialised websites such as VoXEU

### Lessons identified and recommendations linked to dissemination outputs
- Recommendation 1:
  - Importance: Good working links with external researchers who are prominent in the field of macroeconomics of LICs act as a marker of quality, a source of new ideas, and an important channel of outreach.
  - Challenge: Ensuring that work commissioned solely from external sources closely aligns with MRLIC intended outputs is difficult because external researchers may have different understandings, interests or incentives to IMF staff.
  - Implication: Obtaining high quality research through commissions often requires a very close relationship with the IMF; this issue has impacted the number of papers produced this year.
  - Action: Reassess the commissioning process to better assess how and when papers are commissioned, and the correct target for this logframe indicator. Deadline: by December 2018.
- Recommendation 2:
  - MRLIC should, where possible, track indicators for its dissemination products (i.e. views and downloads of newsletters, blogs and videos) by end December 2018.
  - Purpose: Provide an indication of the reach these products have and their potential impact.

### Theory of Change (ToC) and progress towards outcomes
- Implicit ToC summarized in the logical framework:
  - OUTPUT 1 High quality, policy relevant research on macroeconomic issues affecting LICs produced.
  - OUTPUT 2 IMF research products (policy analysis, practical and operational tools and frameworks) produced under this project used by IMF country teams and partner authorities.
  - OUTPUT 3 Engagement by senior IMF policymakers on issues affecting LICs strengthened through this project.
  - OUTPUT 4 IMF strengthens research capacity, by expanding the network of researchers and policy makers working on LIC macroeconomics
  - OUTCOME Better engagement by IMF policy-makers on LIC-specific macroeconomic issues, leading to improved policy-making in specific project thematic areas.
  - IMPACT Better macroeconomic policymaking in LICs leading to faster economic growth, job creation and poverty reduction.
- Status:
  - The ToC remains valid; no major changes over the past year.
  - Decision taken to partner directly with the IMF to maximise early engagement of potential users and shape the research agenda by demand for policy advice.
  - DFID and MRLIC will set out a more detailed ToC by end October 2018.

### Progress to outcomes and policy uptake
- Program is on track to contribute to desired outcome and impact.
- Achievements and uptake:
  - Direct uptake by IMF country teams and by country authorities has more than doubled this year.
  - MRLIC models and analyses used directly in a number of IMF Article IV country reports this year, feeding into IMF policy recommendations.
  - A substantial number of country authorities have been trained on the use of MRLIC models.
  - Forecasting and Policy Analysis System (FPAS) model: being adopted in a co-ordinated fashion by member states of the East African Community.
  - Numbers of papers presented to the IMF Board has doubled this year.
  - Topics covered include inequality, gender, corruption and macro-economic trends.
  - Quoted: IMF’s Deputy Managing Director noted that “The DFID-IMF collaboration has been successful in strengthening the IMF’s capacity to assist our member countries”.
  - Individual IMF staff reported MRLIC output has been catalytic in improving IMF engagement on these topics.
- Suggested reflection: DFID and MRLIC should reflect together on channels more and less effective in generating policy impact as part of drafting the new ToC.

### Logframe changes and planned adjustments
- Changes made at beginning of Phase 3:
  - Targets adjusted upwards based on average performance over first five years, then adjusted downwards pro-rata to account for budget cut to set “medium” targets (equivalent to a DFID score of “A” or “expectations met”).
  - “High” targets set equal to average level of delivery of years 1-5; “Low” targets set as an equal distance below “medium”.
- New logframe targets added under Phase 3:
  - 1.3: Number of freely available books
  - 2.2.1: Courses offered to country authorities
  - 4.4.1: Number of data toolkits (A data toolkit is a new dataset that is put in a format that is accessible and applicable to the work of IMF country teams and country authorities).
- Planned changes as a result of this review:
  - Reset targets under output 2 (“IMF research products produced under this project used by IMF country teams and partner authorities”) in line with performance observed this year.
  - Figures for outputs 2.2 and 2.2.1 have already exceeded targets for 2019; increase targets for 2019 and 2020 is appropriate.
  - No other target changes required at this time, but effort will be needed to reach 2019 targets on indicators under outputs 1, 3 and 4.
  - Recommendation: Produce figures related to number of views, downloads and citations of MRLIC papers and dissemination products by end December 2018; DFID should consider adding such figures as a new target to the logframe or replacing an existing target.

### Value for Money (VfM) assessment and unit costs
- Unit cost performance:
  - Programme achieving a unit cost of just over $87,000 for working papers, substantially lower than planned in the Business Case.
  - Business Case envisaged a unit cost per research paper of $193,600 for the IMF programme and $288,000 for the next best option.
- Cost allocation and estimates:
  - Total Staff/Contractual/VS Costs $10,916,414
    - Research Paper Costs $8,187,311
    - Country Applications Costs $2,729,104
  - Working Papers Produced 94
  - Cost Per Working Paper $87,099
  - Published Papers Produced 48
  - Cost per Published Paper $170,569
  - On the basis that staff and visiting scholars spend 75 percent of their time on research papers and 25 percent on country applications, cost per working paper is approximately $87,000 (£66,000), cost per published paper is approximately $170,000 (£130,000).
  - Decrease in per paper costs from last year noted.
- Interpretation:
  - Lag in getting working papers accepted in peer-reviewed journals means cost per published paper expected to decrease over time as more working papers are submitted and accepted.
  - Conclusion: Programme represents excellent value for money and should continue based on merits and wider portfolio context.
- Reasons for VfM:
  - Economy: Programme management costs low at approximately 8% of total spend (1% direct management costs + a standard 7% IMF trust fund management fee).
  - Efficiency: Outputs produced under firm timeframes; 48 publications in peer-reviewed journals.
  - Effectiveness: MRLIC team composed primarily of IMF staff can communicate research findings quickly to IMF country staff to achieve policy impact.
  - Sustainability: Programme increasingly sustainable with less burden on DFID; research uptake budget reduced by 50% this year while IMF departments subsidise overseas missions; fewer visiting researchers needed due to network.

### Risk overview and mitigation
- Overall risk rating: Minor (each risk category also rated Minor).
- Risk categories and notes:
  - External context: Most research carried out at IMF headquarters using secondary datasets, limited exposure to LIC contexts.
  - Delivery: Established delivery systems, demonstrated high delivery; demand remains high.
  - Operational: Experienced management teams at DFID and IMF.
  - Fiduciary: All funds spent directly by the IMF and subject to extensive IMF financial management controls.
  - Reputational: Little or no reputational risk.
  - Innovation: Not subject to substantial innovation risks.
  - Safeguarding: Activities rarely involve contact with vulnerable persons; external researchers managed per IMF procedures.
- New potential risks from change in delivery model:
  - Sourcing funding from other IMF departments and cutting funds for senior researchers alters operating model.
  - Action: DFID and MRLIC to update the risk matrix by end September 2018; risk rating expected to remain Minor.
  - Action: MRLIC programme team should develop a risk register by end July 2018 in line with DFID smart rules.

### Delivery, commercial and financial performance
- Phase 3 operated under a lower budget compared to previous phases but met or exceeded “high” targets on almost all outputs.
- Cost management strategies:
  - Shifted to use of research assistants (RAs) rather than costly visiting academics to cut salary costs.
  - Initial investment in visiting academics yielded a network to draw on.
  - Increased cost sharing with other IMF departments subsidising travel and conferences.
  - Efficient use of internal economists.
- Project expenditures Years 1-6 (Activity Total, Years 1-6):
  - HQ led missions including entire mission team $6,529,504
  - Short-term Advisors - CD delivery $477,102
  - Research HQ based / Visiting Scholars $3,909,808
  - Seminars & Study Tours $1,510,683
  - Project Backstopping $2,250,403
  - Project Management $161,009
  - Exceptional Expenses $146,730
  - Trust Fund Fee $1,048,967
  - Total $16,034,206
- As of April 2018, $16.0 million has been drawn down from the subaccount. Figures reflect a lag of approximately two to three months to verify expenses before charging donor subaccounts.
- Financial controls and continuity:
  - IMF accurate and reliable in cost estimates; predictable salary increases incorporated without additional resources required.
  - MRLIC programme director involved since inception and remains main point of contact between IMF and DFID.
- Dates:
  - Date of last narrative financial report(s) May 2018
  - Date of last audited annual statement(s) 31 July 2017

### Monitoring, evidence and learning
- Monitoring:
  - Program monitors numbers of papers and conference presentations through staff reports.
  - Policy uptake monitored through MRLIC staff engagement on IMF country missions and Board engagement via internal management communications.
  - Programme meetings monthly; updates and forward looks on each logframe target compiled every four months and shared with DFID.
  - DFID monitors progress through bi-annual meetings and frequent informal contact between DFID SRO and IMF programme staff.
  - Recommendation: Generate statistics on numbers of views, downloads and citations of MRLIC research products, and track feedback on training events; MRLIC/DFID to produce such statistics by end December 2018.
- Evidence:
  - Fundamental evidence for need of programme unchanged from original business case; evaluation not applicable.
- Learning:
  - MRLIC operates monthly meetings to take stock and address delays or blockages; issues raised regularly with DFID.
  - No formal system for logging lessons learned; recommendation that management team consider whether such a system would add value.

### Progress on recommendations from previous reviews
- From 2017 Annual Review:
  - Progress made on two of three items:
    - Logframe revised at beginning of Phase 3 with targets raised.
    - MRLIC began developing evidence products that distil key lessons; “How to” notes published for gender and inequality in IMF work; two more notes planned for 2019. DFID welcomes additional notes on specified topics.
  - No progress on programme steering committee:
    - DFID and MRLIC staff engage regularly via bi-annual meetings and informally; MRLIC workplans signed off by senior IMF Research Department members.
    - Suggestion: Consider a panel containing senior academics and policymakers to add oversight; DFID and IMF should consider.

### Annual review process
- Conducted by a DFID A2 Economic Adviser and B1 Assistant Economist independent of the programme, supported by programme manager and Senior Responsible Owner.
- Quality assurance provided by the A1 manager of DFID Growth Research Team.
- Review included DFID and IMF documents and in-depth discussions with IMF and DFID teams.

---


_Source: https://www.imf.org/-/media/files/topics/lics/macro-research-for-development/documents/june2018.pdf_