## USE OF THIRD PARTY INDICATORS IN FUND REPORTS

## Source details

**Canonical URL:** [USE OF THIRD PARTY INDICATORS IN FUND REPORTS](https://www.imf.org/-/media/files/publications/pp/2017/pp101217use-of-third-party-indicators-in-fund-reports.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/pp/2017/pp101217use-of-third-party-indicators-in-fund-reports.pdf.md)
- [Structured JSON version](/-/media/files/publications/pp/2017/pp101217use-of-third-party-indicators-in-fund-reports.pdf.json)

---

### EXECUTIVE SUMMARY — Background and scope
- Fund staff use indicators developed by other organizations (third-party indicators, TPIs) as input into analysis in surveillance and, to a lesser extent, in program work.
- TPIs measure concepts such as business environment, competitiveness, and quality of governance.
- TPIs are drawn from numerous compilers including international institutions and multilateral development banks, survey institutes, think tanks, non-governmental organizations, and private sector firms.
- It is anticipated that staff will continue to draw on other institutions’ expertise and estimates, consistent with the Executive Board’s guidance where internal expertise is lacking or limited.
- The framework in this paper will apply to all documents subject to the Fund’s Transparency Policy; staff are encouraged to follow similar guidelines for other Fund documents.

### EXECUTIVE SUMMARY — Challenges and risks with TPIs
- TPIs vary in quality: some are based on hard data, others are perception-based, composites, or qualitative expert assessments that are sometimes scored.
- Reliability concerns have been raised for perception-based indicators relating to governance and political risk, including perceived bias in institutions generating the indicators and opacity of sources and methodologies.
- Not understanding TPIs’ limitations can lead to flawed analysis and presentation, potentially undermining the Fund’s credibility.
- Close attention to indicators from private third-party providers may be warranted, including those based on hard data, especially in bilateral and multilateral financial sector surveillance.

### EXECUTIVE SUMMARY — Findings on current use in Fund reports
- Text mining exercise covering 184 recent bilateral and multilateral surveillance reports, 113 Selected Issues Papers, and 82 UFR reports over the period 2016 to mid-2017 finds:
  - More than three-quarters of the sampled Article IV reports included terms associated with selected TPIs.
  - Only one fifth of program reports included terms associated with selected TPIs.
  - Approximately half of the multilateral surveillance reports contained terms associated with these TPIs.
- Use by topic:
  - TPIs are more prevalent in discussion of business climate issues: slightly more than one-third of the sampled Article IV and program staff reports that discussed business climate issues used terms associated with the selected TPIs.
  - Structural reform or political risk: about 20 percent of the Article IV staff reports and only 4 percent of program reports discussing these issues used terms associated with TPIs from this set.
  - Governance and corruption TPIs do not appear to be used regularly in surveillance and programs.
- Purposes of use:
  - Nearly 40 percent of the sampled Article IV staff reports rely on TPIs to show the country’s ranking at a given time or changes of the ranking over time, most often for business environment discussions.
  - Common approaches include: (i) a country’s overall rank at a point in time (often with subcomponents), (ii) evolution or change over time of rank, and (iii) comparing rank or change to peer/benchmark countries.
  - For cross-country comparison, a common approach relies on distance to frontier (DTF) scores; Only 18 percent of the sampled Article IV reports make use of TPIs for this purpose. Most of these reports relied on the Doing Business Indicators.
  - In program documents TPIs are used much less: Twelve percent of the sampled program staff reports used TPIs for ranking purposes, and just 4 percent used them for cross-country comparison purposes. TPIs are not used in specification of conditionality.
- Most frequently referenced TPIs in the sampled staff reports include Doing Business Indicators, Global Competitiveness Index, Consensus Forecasts, and Corruption Perceptions Index.

### FRAMEWORK AND PROPOSED TOOLS FOR BEST PRACTICE
- The framework has three elements:
  1. A principles-based Guidance Note (forthcoming) to raise awareness and promote best practice.
  2. An “Indicators Digest” (a database for internal use only) that compiles selected indicators’ characteristics, including strengths and weaknesses, based on an adapted data quality assessment framework (DQAF).
  3. A review process to ensure best practice in Fund reports.
- The Indicators Digest will compile selected indicators’ characteristics to inform staff judgment and will be based on the adapted DQAF.
- The structured methodology to assess data quality and its application to TPIs is outlined in the paper; the Indicators Digest is presented in full in Background Note I.

### PRINCIPLES EMPHASIZED IN THE FORTHCOMING GUIDANCE NOTE
- Transparency when selecting indicators and interpreting results.
- Robustness: use indicators as one of many inputs; use multiple indicators and sources to measure similar concepts.
- Reflection of stakeholders’ views.
- In applying these principles, staff would consider factors including assurances of integrity, methodological soundness, accuracy, reliability, accessibility of TPIs, and compiler’s characteristics (e.g., whether it is an international organization with a broad membership).

### GOOD PRACTICE AND LESSONS FROM OTHER IOs
- Other IOs (EBRD, OECD, World Bank) use TPIs extensively and share overlapping indicators with the Fund.
- Common good practices across IOs include:
  - Use of own data sources where possible.
  - Reliance on internal expert judgment and internal review processes when integrating TPIs into staff assessments.
  - Clearly distinguishing use of TPIs from institution’s primary sources and staff judgment.
  - Taking stakeholders’ views into account as part of regular consultations.
- Where private sources are used repeatedly, methods have been examined more closely (e.g., methodological seminars at OECD) to gain insight into TPIs’ quality standards.

### RECOMMENDATIONS FOR STAFF PRACTICE
- Provide specific information about TPIs used to guide interpretation and inform overall assessment (indicators’ characteristics, methodological shortcomings, measurement uncertainties).
- Use a range of indicators where appropriate to ensure a more robust measure for a country.
- Adopt the forthcoming Guidance Note’s principles and make use of the Indicators Digest and review process to support rigorous and transparent use of TPIs in Fund reports.

### BOX 1 — Use of TPIs in Selected OECD Country Economic Surveys: 2014–17 — Main findings
- Analysis of 35 OECD country reports during 2014–17 shows:
  - A majority of these reports (63 percent) use at least one TPI as an input along with others for diagnostic and analytical purposes.
  - The most frequently used TPIs are the Doing Business indicators and Global Competitiveness Index. These are complemented by the OECD’s own indicators of Product Market Regulation.
  - Governance-related TPIs are used less often. When they are used, they are often complemented by the OECD’s civic engagement and governance index, a subcomponent of the Better Life Index.
  - From a good-practice perspective, complementing TPIs with internally produced indicators, where available, provides a useful way to check for robustness.

### BOX 1 — Lessons learned and high-level principles
- TPIs are useful for:
  - Facilitating cross-country comparisons.
  - Identifying concrete evidence to inform analysis in areas where an institution may not gather its own primary data.
  - Informing and complementing staff’s own assessments.
- Challenges and limitations:
  - Users may have more limited insight into how TPIs are compiled and how their quality is ensured, compared to statistics compiled internally.
- Good-practice elements observed across IOs:
  - Emphasis on staff judgment.
  - Use of internal review to assess strengths and weaknesses and ensure proper use.
  - Consultation with country authorities and other stakeholders.
  - Being clear about what is—and is not—known about the TPI and its source, its benefits and potential shortcomings, and its relevance to staff’s analysis.
  - Using overlap of TPIs from different sources measuring common concepts to draw insights and improve robustness.
  - Staff judgment remains central; seek authorities’ views and additional contextual background as relevant.
- Key guiding principles highlighted:
  - Transparency: discuss key characteristics (objective measurement, expert qualitative assessment, or perception-based); acknowledge measurement uncertainties and, where possible, standard errors.
  - Robustness: use multiple inputs; avoid reliance on a single indicator; draw on multiple indicators/sources measuring similar concepts.
  - Reflect stakeholders’ views: present staff’s independent views while dialoguing with authorities; for multilateral publications document data sources, add caveats, and conduct robustness tests.

### THE INDICATORS DIGEST — Purpose, scope, and coverage
- Purpose:
  - The Indicators Digest is a central, internal database to help inform staff’s judgment on selected TPIs by providing information on statistical characteristics, strengths, and weaknesses.
  - It is a companion to the forthcoming Guidance Note.
  - The Digest is not a validation exercise and is not intended to present an ex-ante positive or negative list of acceptable indicators.
- Scope and coverage:
  - For the initial version, STA reviewed 13 TPIs that are widely used by the Fund.
  - The initial set of indicators covers predominantly business climate, governance, and political risk.
  - The Digest is a living database but not exhaustive; further expansion will be demand-driven and reflect increasing use in emerging macro-critical areas.
  - Staff and Board members will have continuing access to the expanded Digest.
- Interactions informing the Digest:
  - Staff interacted with selected compilers (including the EBRD, EIU, Maplecroft, PRS Group, Transparency International, and the World Bank) to understand methodology, data collection, processing, and dissemination practices.
  - Assessments in the Digest are based on information provided to the public by compilers; STA staff were not always able to obtain full access to compilers or complete sources and methods.
- Initial set of TPIs reviewed in the Digest (Table 1):
  1. International Country Risk Guide — Political Risk Services (PRS) Group
  2. Political Instability Rankings — The Economist Intelligence Unit (EIU)
  3. Political Stability Risk Index — The Economist Intelligence Unit
  4. Political Risk Component of the EIU Country Risk Model — The Economist Intelligence Unit
  5. Corruption Perception Index — Transparency International
  6. Worldwide Governance Indicators — Daniel Kaufmann (Natural Resource Governance Institute and Brookings Institution) and Aart Kraay (World Bank)
  7. Corruption Index — Verisk Maplecroft
  8. Index of Economic Freedom — Heritage Foundation
  9. Global Competitiveness Index — World Economic Forum (WEF)
  10. Transition Indicators — European Bank for Reconstruction and Development (EBRD)
  11. Doing Business Indicators — World Bank
  12. Indicators of Central Bank Independence and Transparency — Dincer and Eichengreen (2008, 2010, 2014)
  13. Consensus Forecasts — Consensus Economics

### METHODOLOGY — Adapted DQAF and assessment template
- DQAF context:
  - DQAF recognizes five dimensions of data quality for official statistics: assurances of integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.
  - Ratings used in DQAF assessments: observed (O), largely observed (LO), largely non-observed (LNO), non-observed (NO).
- For TPIs, STA used an adapted DQAF focusing on four relevant areas:
  - Assurances of integrity: Is the TPI produced on an impartial and transparent basis?
  - Methodological soundness: Does the compilation follow internationally accepted statistical standards, guidelines, or good practices?
  - Accuracy and reliability: Are the data sources accurate, reliable, and timely?
  - Accessibility: Do users have easy access to the data, metadata, and relevant contact staff?
- The adapted DQAF template:
  - Guides assessment of strengths and weaknesses along the four dimensions and provides ratings on observance of best practices or feasibility of assessment.
  - Includes textual assessments detailing compiler institutional background, known affiliations and sources of funding, and notions of professional independence and impartiality.
  - Describes indicator data sources and collection systems, representativeness of survey samples, compilation techniques, and methodology for composite indicators.
  - Provides an overall assessment summarizing principal elements to assist users in forming judgments on indicator strengths and weaknesses.

### EXAMPLES OF GOOD PRACTICES (selected)
- Hungary 2017 Article IV:
  - Qualifies TPIs as measuring “perceptions” of Hungary’s investment climate.
  - Uses a scatterplot to present countries’ performance across indicators focused on similar issues from different sources.
  - Provides country context and authorities’ views on factors impacting business climate indicators.
- Ukraine 2016 Article IV:
  - SIP on “Corruption and Growth” uses four different corruption indicators for robustness.
  - Describes alternative indicators’ key characteristics upfront.
  - Shows latest data points and evolution over time compared to peers.
  - Presents regional-level corruption indicators where available and discusses country reforms and implementation of anti-corruption measures.
- Kingdom of Swaziland 2015 Article IV:
  - Clarifies survey-based indicators reflect investors’ perceptions and includes a cautionary footnote on limited respondents and coverage.
- Myanmar 2015 Article IV:
  - SIP charts performance relative to peers and includes cautionary footnote on respondents and coverage.
- Georgia 2017 Request for an arrangement Under the Extended Fund Facility:
  - Annex uses official, empirical data and TPIs from different sources; links TPI performance to bottlenecks and opportunities; charts use DTF scores and peer comparators.
- Belgium 2016 Article IV:
  - SIP uses several sources for TPIs and qualifies survey-based indicators with “perceived” in text and chart titles.
- Union of Comoros 2014 Article IV:
  - Clearly indicates assessments are based on survey-based indicators and discusses authorities’ views on rankings.
- Kyrgyz Republic 2017 Third Review under ECF:
  - Uses governance and corruption indicators combined with detailed country context and sectoral developments when developing policy advice.

### CONCLUSIONS AND KEY FINDINGS
- Role and uses of TPIs:
  - TPIs are widely used at the Fund, other IOs, and in the private sector for various purposes.
  - Institutions have relied on TPIs for cross-country comparisons, to identify concrete evidence to inform analysis where primary data are not gathered, and to inform and complement staff assessments.
  - At the Fund, TPIs have provided valuable inputs in surveillance and other contexts, especially for macro-critical surveillance priorities; the most commonly used TPIs are those related to business climate issues.
- Challenges and concerns:
  - General concerns center on the quality of several TPIs.
  - Opacity of methodologies used by private entities and reliance on value judgment exacerbate concerns.
  - Lack of systematic staff guidance, particularly related to the use and presentation of perception-based indicators, raises additional concerns.
  - Not understanding TPIs’ strengths and weaknesses could lead to flawed presentation and analysis with negative effects on country work and reputational risks for the Fund.
- Proposed framework and tools:
  - A framework proposed in this paper addresses these concerns via three best practice principles accompanied by a central database—the Indicators Digest.
  - The principles cover transparency in compilation and use, robustness checks, and presenting authorities’ and other stakeholders’ views to provide contextual value.
  - The Indicators Digest provides information on selected indicators’ characteristics to help inform staff judgment.
  - The assessment of TPIs in the Digest is based on relevant criteria used in the Data Quality Assessment Framework endorsed by the Board in 2003.
  - The Digest is not intended as an ex-ante positive or negative list of acceptable indicators and will remain a living, demand-driven database; it will not be exhaustive of all possible TPIs staff may use.
- Scope and implementation:
  - The proposed approach will apply to all country documents, policy documents, and multi-country documents subject to the Fund’s Transparency Policy.
  - A staff guidance note will be issued and the review process will be used to help ensure quality standards and proper use.
  - Specific good practice examples will help apply elements in practice; staff are encouraged to follow similar guidelines for other Fund documents not subject to the Transparency Policy.
  - The proposed approach should help promote learning, support best practice, and better traction with the membership.

### ISSUES FOR DISCUSSION (Questions posed to Directors)
- Do Directors agree with the proposed approach, which consists of three best practice principles accompanied by a central database (the Indicators Digest), to help promote best practice in Fund reports?
- Do Directors agree with the proposed scope and coverage of the Indicators Digest, specifically that future expansion of the Digest will be demand-driven, focusing on emerging macro-critical areas in surveillance?

### ANNEX I — Continuum of Data Types Underpinning TPIs
- Continuum categories (from more subjective to more objective):
  - Perceptions: Includes surveys of experts or general public to obtain their opinions.
  - Experiences: Includes surveys of experts or general public to obtain insight into their experiences or knowledge, which may be supplemented by interviews.
  - External Assessments: Includes assessments conducted by experts involving ratings or scoring, possibly based on case studies, audit reports, agency statistics, etc.
  - Official or Administrative: Reflects records, laws, reports, or other official observations and may consist of statistics or data generated by governments or agencies.

### ANNEX II — Third-Party Indicators Text Mining Exercise: Sample and Scope
- Sample composition (total number of reports in sample):
  - Article IV reports (main papers) 173
  - Selected issues papers 113
  - UFR reports 82
  - Multilateral surveillance products 11
- Geographical coverage of the sample (by product):
  - Article IV reports (main papers): AFR 24, APD 41, EUR 49, MCD 20, WHD 39
  - Selected issues papers: AFR 21, APD 15, EUR 41, MCD 12, WHD 24
  - UFR reports: AFR 33, APD 4, EUR 16, MCD 21, WHD 8
- Core list of 14 TPIs and their compilers used in the search:
  - International Country Risk Guide, Political Risk Services Group
  - Political Instability Rankings, Economist Intelligence Unit
  - Political Stability Risk Index, Economist Intelligence Unit
  - Corruption Perception Index, Transparency International
  - Worldwide Governance Indicators, Kaufmann (Natural Resource Governance Research Institute and Brookings Institution) and Kraay (World Bank)
  - Corruption Index, Verisk Maplecroft
  - Index of Economic Freedom, Heritage Foundation
  - Global Competitiveness Index, World Economic Forum
  - Transition Indicators, European Bank for Reconstruction and Development
  - Doing Business Indicators, World Bank
  - Indicators of Product Market Regulation, OECD
  - Indicators of Employment Protection, OECD
  - Central Bank Independence and Transparency Indicators, Dincer and Eichengreen
  - Consensus Forecasts, Consensus Economics
- Six questions explored and search approaches:
  - Question 1: To what extent are these third-party indicators used across Fund products?
    - Search approach: Identify reports containing at least one of the TPI core search terms.
  - Question 2: To what extent are selected TPIs used when discussing business climate issues?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to business climate (e.g., business climate, investment climate, business environment).
  - Question 3: To what extent are selected TPIs used when discussing structural reform or political issues?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to structural reform issues (e.g., product market, labor market, political risk).
  - Question 4: To what extent are selected TPIs used for ranking purposes?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to ranking exercises (e.g., increase, decrease, better, worse, relative to, unchanged).
  - Question 5: To what extent are selected TPIs used for cross-country comparisons?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to comparisons across countries (e.g., better, worse, benchmark, peer, regional, average).
  - Question 6: To what extent are selected TPIs accompanied by a discussion of their characteristics or caveats?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to indicator characteristics or caveats (e.g., coverage, limitation, methodology, perception, assumption, interpretation).

*Source: EXECUTIVE SUMMARY, Use of Third-Party Indicators in Fund Reports, October 12, 2017.*

### EXECUTIVE SUMMARY

### EXECUTIVE SUMMARY

### Background and scope
- Fund staff use indicators developed by other organizations (third-party indicators, TPIs) as input into analysis in surveillance and, to a lesser extent, in program work.
- TPIs measure concepts such as business environment, competitiveness, and quality of governance.
- TPIs are drawn from numerous compilers including international institutions and multilateral development banks, survey institutes, think tanks, non-governmental organizations, and private sector firms.
- It is anticipated that staff will continue to draw on other institutions’ expertise and estimates, consistent with the Executive Board’s guidance where internal expertise is lacking or limited.
- The framework in this paper will apply to all documents subject to the Fund’s Transparency Policy; staff are encouraged to follow similar guidelines for other Fund documents.

### Challenges and risks with TPIs
- TPIs vary in quality: some are based on hard data, others are perception-based, composites, or qualitative expert assessments that are sometimes scored.
- Reliability concerns have been raised for perception-based indicators relating to governance and political risk, including perceived bias in institutions generating the indicators and opacity of sources and methodologies.
- Not understanding TPIs’ limitations can lead to flawed analysis and presentation, potentially undermining the Fund’s credibility.
- Close attention to indicators from private third-party providers may be warranted, including those based on hard data, especially in bilateral and multilateral financial sector surveillance.

### Findings on current use in Fund reports
- A text mining exercise covering 184 recent bilateral and multilateral surveillance reports, 113 Selected Issues Papers, and 82 UFR reports over the period 2016 to mid-2017 finds:
  - More than three-quarters of the sampled Article IV reports included terms associated with selected TPIs.
  - Only one fifth of program reports included terms associated with selected TPIs.
  - Approximately half of the multilateral surveillance reports contained terms associated with these TPIs.
- Use by topic:
  - TPIs are more prevalent in discussion of business climate issues: slightly more than one-third of the sampled Article IV and program staff reports that discussed business climate issues used terms associated with the selected TPIs.
  - Structural reform or political risk: about 20 percent of the Article IV staff reports and only 4 percent of program reports discussing these issues used terms associated with TPIs from this set.
  - Governance and corruption TPIs do not appear to be used regularly in surveillance and programs.
- Purposes of use:
  - Nearly 40 percent of the sampled Article IV staff reports rely on TPIs to show the country’s ranking at a given time or changes of the ranking over time, most often for business environment discussions.
  - Common approaches include: (i) a country’s overall rank at a point in time (often with subcomponents), (ii) evolution or change over time of rank, and (iii) comparing rank or change to peer/benchmark countries.
  - For cross-country comparison, a common approach relies on distance to frontier (DTF) scores; Only 18 percent of the sampled Article IV reports make use of TPIs for this purpose. Most of these reports relied on the Doing Business Indicators.
  - In program documents TPIs are used much less: Twelve percent of the sampled program staff reports used TPIs for ranking purposes, and just 4 percent used them for cross-country comparison purposes. TPIs are not used in specification of conditionality.
- Most frequently referenced TPIs in the sampled staff reports include Doing Business Indicators, Global Competitiveness Index, Consensus Forecasts, and Corruption Perceptions Index.

### Framework and proposed tools for best practice
- The framework has three elements:
  1. A principles-based Guidance Note (forthcoming) to raise awareness and promote best practice.
  2. An “Indicators Digest” (a database for internal use only) that compiles selected indicators’ characteristics, including strengths and weaknesses, based on an adapted data quality assessment framework (DQAF).
  3. A review process to ensure best practice in Fund reports.
- The Indicators Digest will compile selected indicators’ characteristics to inform staff judgment and will be based on the adapted DQAF.
- The structured methodology to assess data quality and its application to TPIs is outlined in the paper; the Indicators Digest is presented in full in Background Note I.

### Principles emphasized in the forthcoming Guidance Note
- Transparency when selecting indicators and interpreting results.
- Robustness: use indicators as one of many inputs; use multiple indicators and sources to measure similar concepts.
- Reflection of stakeholders’ views.
- In applying these principles, staff would consider factors including assurances of integrity, methodological soundness, accuracy, reliability, accessibility of TPIs, and compiler’s characteristics (e.g., whether it is an international organization with a broad membership).

### Good practice and lessons from other IOs
- Other IOs (EBRD, OECD, World Bank) use TPIs extensively and share overlapping indicators with the Fund.
- Common good practices across IOs include:
  - Use of own data sources where possible.
  - Reliance on internal expert judgment and internal review processes when integrating TPIs into staff assessments.
  - Clearly distinguishing use of TPIs from institution’s primary sources and staff judgment.
  - Taking stakeholders’ views into account as part of regular consultations.
- Examples: OECD complements TPIs with internal indicators and detailed country context; EBRD’s assessments are rooted in several TPIs complemented by internal expert judgment; World Bank consults TPIs in assembling the CPIA and emphasizes staff judgment.
- Where private sources are used repeatedly, methods have been examined more closely (e.g., methodological seminars at OECD) to gain insight into TPIs’ quality standards.

### Recommendations for staff practice
- Provide specific information about TPIs used to guide interpretation and inform overall assessment (indicators’ characteristics, methodological shortcomings, measurement uncertainties).
- Use a range of indicators where appropriate to ensure a more robust measure for a country.
- Adopt the forthcoming Guidance Note’s principles and make use of the Indicators Digest and review process to support rigorous and transparent use of TPIs in Fund reports.

*Source: EXECUTIVE SUMMARY, Use of Third-Party Indicators in Fund Reports, October 12, 2017.*

### Box 1. Use of TPIs in Selected OECD Country Economic Surveys: 2014–17

### Box 1. Use of TPIs in Selected OECD Country Economic Surveys: 2014–17

### Main findings
- Analysis of 35 OECD country reports during 2014–17 shows:
  - A majority of these reports (63 percent) use at least one TPI as an input along with others for diagnostic and analytical purposes.
  - The most frequently used TPIs are the Doing Business indicators and Global Competitiveness Index. These are complemented by the OECD’s own indicators of Product Market Regulation.
  - Governance-related TPIs are used less often. When they are used, they are often complemented by the OECD’s civic engagement and governance index, a subcomponent of the Better Life Index.
  - From a good-practice perspective, complementing TPIs with internally produced indicators, where available, provides a useful way to check for robustness.

### Lessons learned
- TPIs are useful for:
  - Facilitating cross-country comparisons.
  - Identifying concrete evidence to inform analysis in areas where an institution may not gather its own primary data.
  - Informing and complementing staff’s own assessments.
- Challenges and limitations:
  - Users may have more limited insight into how TPIs are compiled and how their quality is ensured, compared to statistics compiled internally.
- Good-practice elements observed across IOs:
  - Emphasis on staff judgment.
  - Use of internal review to assess strengths and weaknesses and ensure proper use.
  - Consultation with country authorities and other stakeholders.
  - Being clear about what is—and is not—known about the TPI and its source, its benefits and potential shortcomings, and its relevance to staff’s analysis.
  - Using overlap of TPIs from different sources measuring common concepts to draw insights and improve robustness.
  - Staff judgment remains central; seek authorities’ views and additional contextual background as relevant.

### Principles for best practice (high-level)
- The principles-based approach preserves staff flexibility while avoiding flawed analysis and reputational risks.
- The Indicators Digest serves as a centralized quality assessment database for commonly used TPIs and as a companion to the Guidance Note.
- Internal review processes support observance of these principles.

Key guiding principles:
- Transparency
  - Briefly discuss the key characteristics of the indicator, including whether it represents an objective measurement, an expert qualitative assessment, or stakeholders’ views (perception-based).
  - Acknowledge measurement uncertainties; where possible, discuss standard errors and take these into account before using indicators for comparisons across time or across countries.
- Robustness
  - Use indicators as one of many inputs and a complement to qualitative discussion.
  - Avoid relying on a single indicator or source to reach conclusions.
  - Draw on multiple indicators and sources that measure similar concepts to provide robustness checks and reveal potential discrepancies.
  - When multiple indicators/sources are not available, supplement with other facts or qualitative discussion of country-specific factors.
- Reflect stakeholders’ views
  - Present staff’s independent and candid views while dialoguing with authorities and other stakeholders when they offer different interpretations or additional background.
  - For Multilateral Policy Issues Documents (e.g., World Economic Outlook, Global Financial Stability Report, Fiscal Monitor), where stakeholder feedback prior to Board issuance may be impractical, staff should document data sources, add caveats, and conduct robustness tests. Staff may revise publications to address membership comments expressed at Executive Board meetings per the Fund’s Transparency Policy.

### Other specific guidance (selection)
- Be aware of changes in data sources and methodology over time when comparing indicator scores across time.
- For cross-country comparisons, be aware of sample sizes, representativeness, and country-specific quality concerns.
- Recognize degree of uncertainty (confidence intervals and/or standard errors) around reported point estimates, especially for perception-based indicators.
  - When confidence intervals overlap, avoid focusing on rankings; consider presenting country scores relative to a range or an average of peers.
- Staff approaches by indicator type (summary from Figure 2):
  - Official or administrative data
    - Comparison across time: Be aware of changes in data sources and methodology over time.
    - Cross-country comparison: Be aware of sample sizes, representativeness, and methodologies.
    - Quantitative presentation: Index/score may facilitate identification of drivers over time; rankings reflect relative performance; distance-to-frontier may reflect change relative to frontier.
    - Qualitative discussion: Provides flexibility to capture country context and stakeholder views.
  - Perception-based data
    - Comparison across time: Be aware of changes in data sources and methodology over time and degree of uncertainty; consider using historical averages if volatile.
    - Cross-country comparison: Use with caution; be aware of uncertainty around point estimates.
    - Quantitative presentation: Ensure clarity on definition and limits; be cautious with indicators that don’t produce standard errors or confidence intervals; consider presenting relative to peers.
    - Qualitative discussion: Can complement official data and explain decisions and behaviors; ensure clarity on limits of perception-based indicators.

### The Indicators Digest: purpose, scope, and coverage
- Purpose:
  - The Indicators Digest is a central, internal database to help inform staff’s judgment on selected TPIs by providing information on statistical characteristics, strengths, and weaknesses.
  - It is a companion to the forthcoming Guidance Note.
  - The Digest is not a validation exercise and is not intended to present an ex-ante positive or negative list of acceptable indicators.
- Scope and coverage:
  - For the initial version, STA reviewed 13 TPIs that are widely used by the Fund.
  - The initial set of indicators covers predominantly business climate, governance, and political risk.
  - The Digest is a living database but not exhaustive; further expansion will be demand-driven and reflect increasing use in emerging macro-critical areas.
  - Staff and Board members will have continuing access to the expanded Digest.
- Interactions informing the Digest:
  - Staff interacted with selected compilers (including the EBRD, EIU, Maplecroft, PRS Group, Transparency International, and the World Bank) to understand methodology, data collection, processing, and dissemination practices.
  - Assessments in the Digest are based on information provided to the public by compilers; STA staff were not always able to obtain full access to compilers or complete sources and methods.

- Initial set of TPIs reviewed in the Digest (Table 1):
  1. International Country Risk Guide — Political Risk Services (PRS) Group
  2. Political Instability Rankings — The Economist Intelligence Unit (EIU)
  3. Political Stability Risk Index — The Economist Intelligence Unit
  4. Political Risk Component of the EIU Country Risk Model — The Economist Intelligence Unit
  5. Corruption Perception Index — Transparency International
  6. Worldwide Governance Indicators — Daniel Kaufmann (Natural Resource Governance Institute and Brookings Institution) and Aart Kraay (World Bank)
  7. Corruption Index — Verisk Maplecroft
  8. Index of Economic Freedom — Heritage Foundation
  9. Global Competitiveness Index — World Economic Forum (WEF)
  10. Transition Indicators — European Bank for Reconstruction and Development (EBRD)
  11. Doing Business Indicators — World Bank
  12. Indicators of Central Bank Independence and Transparency — Dincer and Eichengreen (2008, 2010, 2014)
  13. Consensus Forecasts — Consensus Economics

### Methodology: adapted DQAF and assessment template
- The Fund’s Data Quality Assessment Framework (DQAF) provides a comprehensive assessment of countries’ data quality; for TPIs the DQAF can be adapted to inform staff judgment.
  - DQAF recognizes five dimensions of data quality for official statistics: assurances of integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.
  - Ratings used in DQAF assessments: observed (O), largely observed (LO), largely non-observed (LNO), non-observed (NO).
- For TPIs, STA used an adapted DQAF focusing on four relevant areas:
  - Assurances of integrity: Is the TPI produced on an impartial and transparent basis?
  - Methodological soundness: Does the compilation follow internationally accepted statistical standards, guidelines, or good practices?
  - Accuracy and reliability: Are the data sources accurate, reliable, and timely?
  - Accessibility: Do users have easy access to the data, metadata, and relevant contact staff?
- The adapted DQAF template:
  - Guides assessment of strengths and weaknesses along the four dimensions and provides ratings on observance of best practices or feasibility of assessment.
  - Includes textual assessments detailing compiler institutional background (e.g., international organizations, nongovernmental or private entities, foundations), known affiliations and sources of funding, and notions of professional independence and impartiality.
  - Describes indicator data sources and collection systems (national data, international organizations, proprietary surveys, other TPIs), representativeness of survey samples, compilation techniques, and methodology for composite indicators.
  - Provides an overall assessment summarizing principal elements to assist users in forming judgments on indicator strengths and weaknesses.

*Source: Box 1. Use of TPIs in Selected OECD Country Economic Surveys: 2014–17.*

### 25.      The approach aims to strike the right balance by considering the coverage of

### USE OF THIRD PARTY INDICATORS IN FUND REPORTS

### Examples of Good Practices
- Hungary 2017 Article IV
  - Explicitly qualifies third party indicators as measuring “perceptions” of Hungary’s investment climate.
  - Uses a scatterplot to present countries’ performance across indicators focused on similar issues from different sources.
  - Provides country context by discussing underlying factors that may influence Hungary’s performance on business climate indicators, and presents the authorities’ views on factors that may be impacting indicators.
- Ukraine 2016 Article IV
  - SIP on “Corruption and Growth” uses four different corruption indicators for robustness in analyzing the impact of corruption on economic growth.
  - Provides a description of the alternative indicators’ key characteristics upfront, in line with the transparency principle.
  - Shows latest data points as well as evolution over time when comparing Ukraine to its peers.
  - Shows regional level corruption indicators, where the data is available, to complement aggregated data.
  - Provides country context on reforms and implementation of anti-corruption measures to complement information from indicators.
- Kingdom of Swaziland 2015 Article IV
  - Relies on several third-party indicators; clarifies that survey-based indicators reflect investors’ perceptions.
  - Includes a footnote encouraging caution due to the limited number of respondents, geographical coverage, and standardized assumptions.
- Myanmar 2015 Article IV
  - SIP on developing a competitive export sector charts performance relative to peers.
  - Includes a footnote encouraging caution when interpreting indicators due to limited respondents, geographical coverage, and standardized assumptions.
- Georgia 2017 Request for an arrangement Under the Extended Fund Facility
  - Annex draws on official and empirical data, as well as TPIs measuring similar concepts from different sources.
  - Analysis reflects knowledge of the country context and the authorities’ reform agenda and links TPI performance to specific bottlenecks and opportunities.
  - Charts reflect use of DTF scores and peer comparators.
- Belgium 2016 Article IV
  - SIP on “Making Public Expenditure More Efficient” relies on several sources for TPIs and uses the qualifier “perceived” ahead of survey-based indicator names in text and chart titles.
- Union of Comoros 2014 Article IV
  - Section categorically indicates assessment is based on survey-based indicators pointing to issues related to infrastructure and institutional and governance.
  - Write-up discusses the authorities’ view on these rankings.
- Kyrgyz Republic 2017 Third Review under ECF
  - Draws on governance and corruption indicators to highlight scope for reform in the context of an ECF review.
  - Provides detailed country context, including specific examples, to explain trends in governance and corruption indicators.
  - Uses indicators as one input combined with knowledge of sectoral level developments when developing policy advice.

### Conclusions and Key Findings
- Role and uses of TPIs
  - TPIs are widely used at the Fund, other IOs, and in the private sector for various purposes.
  - Institutions have relied on TPIs for cross-country comparisons, to identify concrete evidence to inform analysis where primary data are not gathered, and to inform and complement staff assessments.
  - At the Fund, TPIs have provided valuable inputs in surveillance and other contexts, especially for macro-critical surveillance priorities; the most commonly used TPIs are those related to business climate issues.
- Challenges and concerns
  - General concerns center on the quality of several TPIs.
  - Opacity of methodologies used by private entities and reliance on value judgment exacerbate concerns.
  - Lack of systematic staff guidance, particularly related to the use and presentation of perception-based indicators, raises additional concerns.
  - Not understanding TPIs’ strengths and weaknesses could lead to flawed presentation and analysis with negative effects on country work and reputational risks for the Fund.
- Proposed framework and tools
  - A framework proposed in this paper addresses these concerns and challenges via three best practice principles accompanied by a central database—the Indicators Digest.
  - The principles cover transparency in compilation and use, robustness checks, and presenting authorities’ and other stakeholders’ views to provide contextual value.
  - The Indicators Digest provides information on selected indicators’ characteristics to help inform staff judgment.
  - The assessment of TPIs in the Digest is based on relevant criteria used in the Data Quality Assessment Framework endorsed by the Board in 2003.
  - The Digest is not intended as an ex-ante positive or negative list of acceptable indicators and will remain a living, demand-driven database; it will not be exhaustive of all possible TPIs staff may use.
- Scope and implementation
  - The proposed approach will apply to all country documents, policy documents, and multi-country documents subject to the Fund’s Transparency Policy.
  - A staff guidance note will be issued and the review process will be used to help ensure quality standards and proper use.
  - Specific good practice examples will help apply elements in practice; staff are encouraged to follow similar guidelines for other Fund documents not subject to the Transparency Policy.
  - The proposed approach should help promote learning, support best practice, and better traction with the membership.

### Issues for Discussion (Questions posed to Directors)
- Do Directors agree with the proposed approach, which consists of three best practice principles accompanied by a central database (the Indicators Digest), to help promote best practice in Fund reports?
- Do Directors agree with the proposed scope and coverage of the Indicators Digest, specifically that future expansion of the Digest will be demand-driven, focusing on emerging macro-critical areas in surveillance?

### Annex I — Continuum of Data Types Underpinning TPIs
- Continuum categories (from more subjective to more objective):
  - Perceptions
    - Includes surveys of experts or general public to obtain their opinions.
  - Experiences
    - Includes surveys of experts or general public to obtain insight into their experiences or knowledge, which may be supplemented by interviews.
  - External Assessments
    - Includes assessments conducted by experts involving ratings or scoring, possibly based on case studies, audit reports, agency statistics, etc.
  - Official or Administrative
    - Reflects records, laws, reports, or other official observations and may consist of statistics or data generated by governments or agencies.

### Annex II — Third-Party Indicators Text Mining Exercise: Sample and Scope
- Sample composition (total number of reports in sample):
  - Article IV reports (main papers) 173
  - Selected issues papers 113
  - UFR reports 82
  - Multilateral surveillance products 11
- Geographical coverage of the sample (by product):
  - Article IV reports (main papers): AFR 24, APD 41, EUR 49, MCD 20, WHD 39
  - Selected issues papers: AFR 21, APD 15, EUR 41, MCD 12, WHD 24
  - UFR reports: AFR 33, APD 4, EUR 16, MCD 21, WHD 8
- Core list of 14 TPIs and their compilers used in the search:
  - International Country Risk Guide, Political Risk Services Group
  - Political Instability Rankings, Economist Intelligence Unit
  - Political Stability Risk Index, Economist Intelligence Unit
  - Corruption Perception Index, Transparency International
  - Worldwide Governance Indicators, Kaufmann (Natural Resource Governance Research Institute and Brookings Institution) and Kraay (World Bank)
  - Corruption Index, Verisk Maplecroft
  - Index of Economic Freedom, Heritage Foundation
  - Global Competitiveness Index, World Economic Forum
  - Transition Indicators, European Bank for Reconstruction and Development
  - Doing Business Indicators, World Bank
  - Indicators of Product Market Regulation, OECD
  - Indicators of Employment Protection, OECD
  - Central Bank Independence and Transparency Indicators, Dincer and Eichengreen
  - Consensus Forecasts, Consensus Economics
- Six questions explored and search approaches:
  - Question 1: To what extent are these third-party indicators used across Fund products?
    - Search approach: Identify reports containing at least one of the TPI core search terms.
  - Question 2: To what extent are selected TPIs used when discussing business climate issues?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to business climate (e.g., business climate, investment climate, business environment).
  - Question 3: To what extent are selected TPIs used when discussing structural reform or political issues?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to structural reform issues (e.g., product market, labor market, political risk).
  - Question 4: To what extent are selected TPIs used for ranking purposes?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to ranking exercises (e.g., increase, decrease, better, worse, relative to, unchanged).
  - Question 5: To what extent are selected TPIs used for cross-country comparisons?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to comparisons across countries (e.g., better, worse, benchmark, peer, regional, average).
  - Question 6: To what extent are selected TPIs accompanied by a discussion of their characteristics or caveats?
    - Search approach: Identify reports containing—in the same paragraph—at least one of the TPI core search terms and one of a set of search terms related to indicator characteristics or caveats (e.g., coverage, limitation, methodology, perception, assumption, interpretation).

*USE OF THIRD PARTY INDICATORS IN FUND REPORTS*

---


_Source: https://www.imf.org/-/media/files/publications/pp/2017/pp101217use-of-third-party-indicators-in-fund-reports.pdf_