## GUIDANCE NOTE FOR THE USE OF THIRD-PARTY INDICATORS

## Source details

**Canonical URL:** [GUIDANCE NOTE FOR THE USE OF THIRD-PARTY INDICATORS](https://www.imf.org/-/media/files/publications/pp/2018/pp071718guidance-note-for-the-use-of-third-party-indicators-in-fund-reports.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/pp/2018/pp071718guidance-note-for-the-use-of-third-party-indicators-in-fund-reports.pdf.md)
- [Structured JSON version](/-/media/files/publications/pp/2018/pp071718guidance-note-for-the-use-of-third-party-indicators-in-fund-reports.pdf.json)

---

### Executive summary — Purpose, scope, and principles
- Purpose and scope:
  - Provides guidance on use of third-party indicators (TPIs) in Fund reports, in line with the policy endorsed by the Board in November 2017.
  - Applies to all documents subject to the Fund’s Transparency Policy (i.e., Board documents); Executive Board asked staff to follow similar guidelines for other Fund publications (Regional Economic Outlooks, Staff Discussion Notes).
  - Defines a TPI as an indicator compiled by other organizations (other than the Fund or member country authorities): numerical values (e.g., an index) or categorical values that can be ranked (e.g., a credit rating).
  - Note date: July 17, 2018.
- Principles for best practice (TPI framework):
  - Transparency
  - Robustness
  - Reflection of stakeholders’ views
- Rationale:
  - Continue using other institutions’ expertise where internal expertise is lacking or limited.
  - A principles-based approach preserves staff flexibility while limiting flawed analysis and reputational risks.
  - Positive/negative lists of TPIs are neither feasible nor desirable; staff judgment, stakeholder engagement, and review mechanisms are essential.

### Practical considerations when working with TPIs
- General guidance:
  - No indicator is banned from Fund reports if used appropriately.
  - TPIs should supplement—not replace—an open, candid, robust, and well-documented discussion with the authorities.
- Potential pitfalls to consider:
  - Opacity of sources and methodologies.
  - Indicators based on perception or expert judgment.
  - Misattribution of compilers (example: Worldwide Governance Indicators attribution).
- Transparency actions staff should consider (apply with judgment to keep reports streamlined):
  - Explain whether and why TPI use is needed.
  - Clearly identify TPIs and their compilers; be precise to avoid misattribution.
  - Justify indicator choice and explain how it informed assessment.
  - Discuss key characteristics (survey of perceptions vs. hard data vs. expert judgment), timeliness, country/time coverage, and sample representativeness.
  - Acknowledge measurement uncertainties; report confidence intervals and/or standard errors when provided by the compiler.
  - Be aware of factors impacting perceptions of objectivity (mission, funding, governance).
- Robustness-promoting actions:
  - Use TPIs as one input among many (official sources, qualitative input, staff judgment).
  - Supplement with qualitative input and analysis rooted in country context and dialogue with authorities.
  - Refer to multiple indicators from different, independent sources where feasible; recognize potential correlation among TPIs that rely on similar inputs.
  - When widely accepted TPIs are used, explicit robustness discussion may not be necessary.
- Stakeholders’ views—process varies by product:
  - Article IV and UFR reports: engage stakeholders on or before mission; reflect authorities’ views in reports issued to the Board.
  - Multilateral Policy Issues Documents (WEO, GFSR, Fiscal Monitor): pre-Board incorporation of stakeholders’ views may be infeasible; post-Board modifications possible as allowed by the Transparency Policy.
  - Policy Papers: cross-country analysis may not permit pre-Board stakeholder incorporation; revisions prior to publication can reflect views modified in light of Board discussion (provided Management shares those views).
  - Regional Economic Outlooks: staff could seek authorities’ views during drafting by consulting relevant Executive Director(s).
  - Staff Discussion Notes and Working Papers: stakeholder views can be obtained when shared with relevant Executive Director(s).

### Expectations for review and departmental responsibilities
- Review process:
  - Undertaken by authoring departments, SPR, and STA to ensure principles are observed operationally.
  - Directors emphasized judicious and evenhanded use of TPIs based on macro-criticality and relevance, including country-specific context.
- Departmental responsibilities:
  - Authoring departments: primary responsibility for applying the framework; consult STA for indicators not in the Indicators Digest.
  - SPR: verify adherence in staff reports, consulting STA as needed while keeping reports streamlined.
  - Area departments: ensure consistency with the TPI framework in other publications (Selected Issues Papers, REOs, Staff Discussion Notes, Working Papers) and encourage similar guidelines for non-Transparency Policy Fund documents.

### The Third-Party Indicators Digest — purpose and scope
- Serves as a centralized internal database highlighting strengths and weaknesses of many commonly used TPIs at the Fund.
- Contents and uses:
  - Lists types of source data used, compiler funding source, methodology summary, and research references.
  - Presents an overall assessment intended to assist staff judgment; staff can tailor assessments to country circumstances.
  - Groups commonly used indicators by concepts they measure and summarizes strengths/weaknesses using an adapted Data Quality Assessment Framework (DQAF), with coverage across countries and time.
- Key facts about the Digest:
  - Originally prepared in November 2017 and updated in May 2018.
  - Provides information on 13 TPIs commonly used by Fund staff in political risk, governance and corruption, and business climate.
  - Assessment is for internal use only; staff should not refer to it in external publications.
  - The assessment is not a validation exercise and is not intended as an ex-ante positive or negative list of indicators acceptable for use.
  - Inclusion in the Digest does not imply endorsement by the Fund, nor does exclusion imply lower reliability.

### Types of source data and implications (Figure 1 / Figure 2 references)
- Spectrum of four data types (more objective → more subjective):
  - Official Data: official records, laws, reports, or observations based on government statistics.
  - Survey of Hard Data: surveys involving hard data, official records, or specific quantitative outcomes.
  - Expert Judgment: assessments by experts involving ratings/scoring, case studies, audit reports, or models.
  - Survey of Perceptions: surveys of experts, business executives, firms, households, or the public to obtain perceptions/opinions/experiences.
- Some TPIs are composite indicators that rely on a variety of inputs, including other TPIs.
- Implications:
  - The type of source data guides presentation, interpretation, and rigor required.

### Perception-based TPIs: value, measurement uncertainty, and presentation
- Value and pitfalls:
  - Perceptions can reflect gaps between reality and written laws/policies and matter for economic decisions.
  - Individual perceptions can change year-to-year without reflecting underlying reality.
  - Cross-country/time-trend analysis may be unreliable if survey samples are inconsistent or change.
- Measurement uncertainty:
  - Recognize uncertainty around point estimates; incorporate standard errors or confidence intervals when provided.
  - Example: For WGI-CCI, the 90 percent confidence interval can be calculated as the score +/- 1.645 times the standard error.
  - Guidance: When confidence intervals overlap it should be assumed that performance is roughly the same.
- Presentation practices, rankings, and comparisons:
  - Prefer presenting point scores and acknowledging uncertainty rather than relying on country rankings.
  - Rankings reflect relative, not absolute, performance; improvements in ranking can reflect changes elsewhere.
  - For corruption TPIs, staff must not use rankings (ordinal, percentile, or quantile groupings).
  - Cross-country comparisons in Article IV or UFR reports: generally preferable to compare a country’s performance relative to the average/range of a peer group (region, income, or other grouping) rather than to an individual country.
  - Staff may identify specific comparator countries by TPI performance only when it improves analysis and policy recommendations, with explicit justification if rationale is not self-evident.
  - Alternative metrics: distance to frontier (DTF) scores may better reflect deviation from best performer.

### Specific guidance on corruption TPIs (Box 1)
- Governance and corruption assessments should be holistic, relying on quantitative and qualitative information and reflecting authorities’ views.
- Required practices for corruption TPIs:
  - Refrain from using country rankings. Example unacceptable formulation: “Country X ranks 129th in the Worldwide Governance Indicators Control of Corruption Index (WGI-CCI).”
  - Use scores (point estimates) rather than rankings; consider presenting scores relative to peers (ranges or averages).
  - Avoid presentations that appear as rankings (e.g., cross-country “heat maps” with quantile color schemes).
  - Acknowledge uncertainty: report confidence intervals and/or standard errors when available. Example: WGI-CCI publishes a standard error; the 90 percent confidence interval equals score +/- 1.645 times the standard error.

### Risk-based application and Figure 2 guidance
- Apply a risk-based approach to determine required rigor depending on:
  - Whether underlying data are dated.
  - Whether indicators are official data, surveys of hard data, or perception-based.
  - Whether indicator measures macroeconomic/financial concepts directly or concepts where the Fund has limited expertise (governance, corruption, gender, climate).
  - Familiarity with the indicator among staff and authorities, prior Fund use, and stakeholders’ reactions.
  - How the indicator is being used (cross-country regression analysis, input to constructed indicators).
- Figure 2 guidance (continuum summary):
  - Type of analysis:
    - Official data: be aware of changes in sources and methodology over time.
    - Perception-based data: be aware of changes in sources/methodology and uncertainty; consider historical averages if indicator is volatile.
  - Cross-country comparison:
    - Official data: watch for differences in sample size, representativeness, and methodology.
    - Perception-based data: use with caution; be mindful of methodological changes and uncertainty.
  - Type of presentation:
    - Index/score: facilitates identification of drivers of change; for corruption TPIs global rankings are prohibited; present scores relative to range/average of peers.
    - Ranking: recognize rankings reflect relative performance; consider presenting relative to average/range of peers.
    - Distance to frontier: may better reflect change over time relative to frontier.
    - Qualitative discussion: greatest flexibility to capture country context, stakeholder views, and nuances; complements quantitative indicators.

### Assessment framework for TPIs and the Indicators Digest (adapted DQAF)
- The Indicators Digest uses an adaptation of the Fund’s Data Quality Assessment Framework (DQAF, Executive Board endorsed in 2003).
- Four assessed dimensions:
  - Assurances of integrity: impartiality and transparency of production.
  - Methodological soundness: coverage of relevant concepts and documentation.
  - Accuracy and reliability: accuracy, reliability, and timeliness of data sources.
  - Accessibility: access to data, metadata, and relevant contact staff.
- Elements and ratings:
  - Multiple elements cover transparency, concepts and definitions, source data, statistical techniques, revisions policy, data and metadata accessibility, and assistance to users.
  - Each element is rated as observed, partially observed, or not observed; not rated if not applicable or assessment infeasible.
  - Dimension rating generally equals the average rating of its elements.
- Presentation and interpretation:
  - Spider charts indicate assessments across the four dimensions; ratings away from center indicate better data quality.
  - Assessments facilitate relative comparison but do not signal overall superiority; intended as robustness checks.
- Governance and updates:
  - Digest remains a living database with an annual review process led by SPR and STA.
  - Expansion is demand-driven and focused on emerging and macro-critical areas; it will never be exhaustive.
  - STA solely responsible for applying the adapted DQAF assessment to ensure consistency and efficiency.
  - STA will assess new TPIs for inclusion within the annual update process, in coordination with SPR.
  - Updates communicated via outreach events and internal communication; latest version disseminated through SPR and STA internal websites.

### Approaching TPIs not in the Indicators Digest and practical checklist (Box 3)
- Staff can use TPIs inside and outside the Digest; the Digest is not an ex-ante positive/negative list and should not stifle innovation.
- STA assistance:
  - STA may provide a preliminary assessment for new TPIs not included in the Digest or advise staff on how to assess them.
  - Fund staff can request STA to review a TPI not in the Digest; STA, in consultation with SPR, will consider preliminary assessment depending on likely inclusion and use within the Fund.
  - Staff should not produce their own adapted DQAF ratings to preserve consistency and evenhandedness.
  - STA assessments rely on publicly available information and consultations with compilers, focusing on methodology, data sources, collection and verification, confidence bands, and limitations.
- Practical checklist (issues to seek information on):
  - Timeliness of the data.
  - Type of source data (objective vs. subjective per Figure 1).
  - Methodological approach used by the compiler.
  - Structural breaks in methodology impeding time comparisons.
  - For survey-based TPIs: deviations in sample size/representativeness across countries/time.
  - For survey-based TPIs: confidence intervals or standard errors if reported and implications for comparisons.
  - Characteristics of the compiler and its funding source, if available.
- Not necessary to detail all items explicitly in staff reports, but staff use should reflect consideration of these issues.

### Examples of good practice and practices to avoid (Boxes 4 and 5)
- Good-practice features illustrated across multiple staff reports and REO/Chapter excerpts:
  - Acknowledgement that many TPIs are perception-based.
  - Discussion of key indicator characteristics and source data.
  - Use of point scores rather than rankings where appropriate.
  - Consideration of sub-components of indices.
  - Use of peer comparators as groupings/averages rather than naming individual comparator countries.
  - Robustness checks via comparison across multiple TPIs and with official/empirical data.
  - Correct attribution of indicator sources.
- Selected examples (excerpted highlights and exact numeric references preserved):
  - Union of the Comoros (2014 Article IV): acknowledges perception-based TPIs; notes Doing Business ranking of 159th of 189 countries.
  - Cabo Verde (2018 Article IV): notes public debt was 129.5 percent of GDP in 2016, twice the level at end-2009.
  - WEO (April 2018, Chapter 2): uses correct attribution for WGI; uses point scores and peer comparators as grouping.
  - EUR REO (November 2017, Chapter 2): uses WGI and Doing Business, noting they are perception-based and discussing implications; references Annex 2.2 Indicators and Sources.
  - Review of 1997 Guidance Note on Governance: uses WGI-CCI (available annually for 1996–2016 and covering about 200 countries) and checks robustness with Transparency International GCB (available 2003–16 and covers less than 100 countries in each year).
- Practices to avoid (Box 5 examples):
  - Citing individual countries (by name) instead of peer groups for cross-country comparisons.
  - Establishing rankings via point-score bar charts without confidence intervals when available.
  - Omitting source information relevant for interpretation.
  - Relying on rankings without context or authorities’ views (illustrative text: “The business climate in Country X is poor. According to the Doing Business report, the World Bank ranks Country X 120 out of 189 countries. This reflected a deterioration compared to the prior year’s report when Country X ranked 110 out of 189 countries. Country X should prioritize reforms to improve its business climate.”).
  - Failing to acknowledge perception-based TPIs or to justify identification of multiple individual comparator countries in another country’s Article IV report.

*Guidance note: GUIDANCE NOTE FOR THE USE OF THIRD-PARTY INDICATORS — Executive Summary (July 17, 2018).*
*Source: https://www.imf.org/-/media/files/publications/pp/2018/pp071718guidance-note-for-the-use-of-third-party-indicators-in-fund-reports.pdf*

### EXECUTIVE SUMMARY

### EXECUTIVE SUMMARY

### Purpose and scope
- Provides guidance on use of third-party indicators (TPIs) in Fund reports, in line with the policy endorsed by the Board in November 2017.
- Applies to all documents subject to the Fund’s Transparency Policy (i.e., Board documents) and the Executive Board asked staff to follow similar guidelines for other Fund publications, in particular, Regional Economic Outlooks and Staff Discussion Notes.
- Defines a TPI as an indicator compiled by other organizations (i.e., other than the Fund or member country authorities). TPIs can be numerical values (e.g., an index) or categorical values that can be ranked (e.g., a credit rating).
- Note date: July 17, 2018.

### Principles for best practice in TPI use
- The TPI framework consists of three principles for best practice:
  - Transparency
  - Robustness
  - Reflection of stakeholders’ views
- Rationale:
  - Staff’s continued practice of drawing on other institutions’ expertise and estimates is consistent with Executive Board guidance where internal expertise is lacking or limited.
  - A principles-based approach preserves staff flexibility to advance surveillance priorities while promoting practices to limit the risk of flawed analysis and reputational risks.
  - Establishing a positive or negative list of TPIs is neither feasible nor desirable; staff judgment, stakeholder engagement, and review mechanisms are essential.

### Practical considerations when working with TPIs
- No indicator is banned from Fund reports, provided it is used appropriately.
- TPIs should supplement—not replace—an open, candid, robust, and well-documented discussion with the authorities.
- Staff should be mindful of potential pitfalls across types of TPIs to ensure effective use in Fund reports, including issues arising from:
  - Opacity of sources and methodologies.
  - Indicators based on perception or expert judgment.
  - Misattribution of compilers (example: Worldwide Governance Indicators attribution).
- Transparency actions staff should consider (applied with judgment to keep reports streamlined):
  - Consider and explain whether and why TPI use is needed.
  - Clearly identify TPIs and their compilers; be precise to avoid misattribution.
  - Justify indicator choice and explain how the indicator informed assessment.
  - Discuss key characteristics (survey of perceptions vs. hard data vs. expert judgment), timeliness, country/time coverage, and sample representativeness.
  - Acknowledge measurement uncertainties; report confidence intervals and/or standard errors when provided by the compiler.
  - Be aware of factors impacting perceptions of objectivity (mission, funding, governance).
- Robustness-promoting actions:
  - Use TPIs as one input among many (official sources, qualitative input, staff judgment).
  - Supplement with qualitative input and analysis rooted in country context and dialogue with authorities.
  - Refer to multiple indicators from different, independent sources where feasible, while recognizing potential correlation among TPIs that rely on similar inputs.
  - When widely accepted TPIs are used, explicit robustness discussion may not be necessary.
- Stakeholders’ views:
  - Staff should discuss TPI-informed findings with authorities and other key stakeholders and reflect their views in staff reports as appropriate.
  - Process varies by product:
    - Article IV and UFR reports: teams should engage stakeholders on or before mission; authorities’ views should be reflected in reports issued to the Board.
    - Multilateral Policy Issues Documents (World Economic Outlook, Global Financial Stability Report, Fiscal Monitor): may not be feasible to incorporate stakeholders’ views pre-Board; post-Board modifications to reflect membership views are possible as allowed by Transparency Policy.
    - Policy Papers: cross-country analysis may not permit pre-Board stakeholder incorporation; revisions prior to publication can reflect views modified in light of Board discussion (provided Management shares those views), per the Transparency Policy.
    - Regional Economic Outlooks: not discussed by the Board; staff could seek authorities’ views during drafting by consulting relevant Executive Director(s).
    - Staff Discussion Notes and Working Papers: stakeholder views can be obtained when shared with relevant Executive Director(s).

### Expectations for the review process
- A review process undertaken by authoring departments, SPR, and STA will help ensure observance of the principles at the operational level.
- Directors emphasized judicious and evenhanded use of TPIs based on macro-criticality and relevance, including country-specific context in interpretation.

### The Third-Party Indicators Digest
- Serves as a centralized, internal database highlighting strengths and weaknesses of many commonly used TPIs at the Fund.
- Contents and uses:
  - Lists types of source data used, compiler funding source, methodology summary, and research references.
  - Presents an overall assessment intended to assist staff judgment on appropriateness; staff can tailor assessments to country circumstances.
  - Groups commonly used indicators by the concepts they measure and summarizes strengths/weaknesses using an adapted Data Quality Assessment Framework (DQAF), with coverage across countries and time.
- The Indicators Digest is for internal use only; staff should not refer to it in external publications.
- The assessment in the Indicators Digest is not a validation exercise and is not intended to present an ex-ante positive or negative list of indicators acceptable for use.

### Examples and resources
- The guidance note includes:
  - Examples of good practices (excerpts from selected staff reports and Fund documents) to illustrate implementation of principles.
  - Generic examples of practices to avoid.
  - Boxes and figures providing specific guidance on, for example, TPI rankings on corruption, risk-based application of principles, how to approach TPIs not in the Indicators Digest, and good/bad practice examples.

*Guidance note: GUIDANCE NOTE FOR THE USE OF THIRD-PARTY INDICATORS — Executive Summary (July 17, 2018).*

### 15.      When using and presenting TPIs, staff should consider their source data. The source

### GUIDANCE NOTE FOR THE USE OF THIRD-PARTY INDICATORS

### Types of source data and implications
- Source data for TPIs fall along a spectrum of four data types, from more objective to more subjective:
  - Official Data: Reflects official records, laws, reports, or other official observations based on statistics or data generated by government agencies.
  - Survey of Hard Data: Includes surveys involving hard data, official records, or specific outcomes based on quantitative indicators.
  - Expert Judgment: Includes assessments conducted by experts involving ratings or scoring, possibly based on case studies, audit reports, or models.
  - Survey of Perceptions: Includes surveys of experts, business executives, firms, households, or the public-at-large to obtain perceptions, opinions, or experiences, which may be supplemented by interviews.
- Some TPIs are composite indicators that rely on a variety of inputs, including other TPIs.
- The type of source data has implications for the approach to be taken when using and presenting the TPI (see Figure 2 referenced in the source).

### Perception-based TPIs: value and pitfalls
- Perception-based TPIs can yield valuable insights but present specific challenges:
  - Perceptions may reflect a gap between reality and written laws or policies and thus can matter for economic decisions.
  - Individual perceptions can change year-to-year without fully reflecting changes in underlying reality.
  - Cross-country comparisons or time-trend analysis may be unreliable if survey samples are not consistent across countries or change over time.
- Measurement uncertainty:
  - Recognizing measurement uncertainties around point estimates in quantitative discussions or presentations is essential.
  - Some compilers provide standard errors or confidence intervals; staff should incorporate these into analysis and presentation (Figure 2 provides detail).
  - Example: For indicators like the WGI-CCI, the 90 percent confidence interval can be calculated as the score +/- 1.645 times the standard error.

### Presentation practices, rankings, and comparisons
- Prefer presenting point scores and acknowledging uncertainty rather than relying on country rankings:
  - Rankings reflect relative, not absolute, performance and improvements in ranking can reflect changes elsewhere rather than in the country.
  - For corruption TPIs, staff should not use rankings related to TPIs (see Box 1).
- Cross-country comparisons:
  - When used in Article IV or UFR reports, it is generally preferable to compare a country’s performance relative to the average/range of a peer group (based on region, income, or other groupings) rather than to an individual country.
  - Staff should identify other specific countries (not the subject of the report) by TPI performance only when this improves analysis and policy recommendations, with explicit justification in the staff report if rationale is not self-evident.
  - An alternative to point scores or rankings is to cite distance to frontier (DTF) scores, which measure the deviation of a country from the best performer.
- Methodological changes:
  - Changes in a compiler’s methodology may limit ability to compare scores across time before and after the change.
  - For TPIs in the Indicators Digest, methodological changes have been flagged and included in compilers’ methodological notes.

### Specific guidance on corruption TPIs (Box 1)
- Governance and corruption assessments should be holistic, relying on both quantitative and qualitative information and reflecting authorities’ views.
- Use TPIs to complement—not displace—Fund staff analysis and other international organizations’ work.
- For TPIs measuring corruption, staff must:
  - Refrain from using country rankings (ordinal, percentile, or quantile groupings). Example formulations like “Country X ranks 129th in the Worldwide Governance Indicators Control of Corruption Index (WGI-CCI)” are not acceptable.
  - Use scores (point estimates) rather than rankings; consider presenting country scores relative to peers (ranges or averages).
  - Avoid presentations that appear as rankings (e.g., cross-country “heat maps” with quantile color schemes).
  - Acknowledge uncertainty: report confidence intervals and/or standard errors when available. Example: WGI-CCI publishes a standard error; the 90 percent confidence interval equals score +/- 1.645 times the standard error.
- Confidence interval guidance:
  - When confidence intervals overlap it should be assumed that performance is roughly the same.

### Risk-based application and Figure 2 guidance
- Apply a risk-based approach to determine rigor needed depending on:
  - Whether underlying data are dated.
  - Whether indicators are official data, surveys of hard data, or perception-based.
  - Whether the indicator measures macroeconomic/financial concepts directly or concepts where the Fund has limited expertise (e.g., governance, corruption, gender, climate).
  - Familiarity with the indicator among staff and authorities, prior Fund use, and stakeholders’ reactions.
  - How the indicator is being used (e.g., cross-country regression analysis, to develop indicators using TPIs as inputs).
- Figure 2 summary guidance (two ends of continuum shown):
  - Type of analysis:
    - Official data: be aware of changes in sources and methodology over time.
    - Perception-based data: be aware of changes in sources and methodology over time and degree of uncertainty around point estimates; consider using historical averages rather than last point estimates if indicator is volatile.
  - Cross-country comparison:
    - Official data: be aware of cross-country differences in sample size, representativeness, and methodology.
    - Perception-based data: use with caution; be aware of methodological changes and uncertainty.
  - Type of presentation:
    - Index/score: may facilitate identification of drivers of change; for corruption TPIs global rankings are prohibited; present scores relative to range/average of peers and ensure clarity on definition and limits of perception-based indicators.
    - Ranking: recognize rankings reflect relative performance; consider presenting relative to average/range of peers.
    - Distance to frontier: may better reflect change in performance over time relative to frontier.
    - Qualitative discussion: provides greatest flexibility to capture country context, stakeholder views, and nuances; complements quantitative indicators.

### Review process and departmental responsibilities
- Review process will promote proper and consistent application of the framework:
  - Authoring departments have primary responsibility for ensuring application of the framework and should consult STA when questions arise on indicators not included in the Indicators Digest.
  - SPR, in its review of staff reports, will verify adherence to the principles and guidance, consulting STA as needed while exercising judgment to keep reports streamlined.
- Area departments are responsible for ensuring consistency with the TPI framework in other publications (e.g., Selected Issues Papers, Regional Economic Outlooks, Staff Discussion Notes, Working Papers) and for encouraging staff to follow similar guidelines for Fund documents not subject to the Transparency Policy.

### Resources available to staff: Third-Party Indicators Digest
- The Indicators Digest is a central internal database compiling selected TPIs’ characteristics to inform staff judgment.
- The Indicators Digest was originally prepared in November 2017 and updated in May 2018.
- It provides information on 13 TPIs commonly used by Fund staff in political risk, governance and corruption, and business climate.
- The Digest highlights indicators’ statistical characteristics, strengths, and weaknesses.
- The assessment in the Indicators Digest:
  - Is not a validation exercise.
  - Is not intended as an ex-ante positive or negative list of indicators acceptable for use.
  - Does not imply that indicators included are endorsed by the Fund, nor that excluded indicators are less reliable.
- As the Indicators Digest is intended for internal use, staff should not refer to it in external publications.

*Source: https://www.imf.org/-/media/files/publications/pp/2018/pp071718guidance-note-for-the-use-of-third-party-indicators-in-fund-reports.pdf*

### 23.      Although there are no international standards for TPIs, the Indicators Digest uses a

### GUIDANCE NOTE FOR THE USE OF THIRD-PARTY INDICATORS

### Assessment framework for TPIs
- The Indicators Digest uses a structured and transparent method to assess the quality of a TPI based on an adaptation of the Fund’s Data Quality Assessment Framework (DQAF) endorsed by the Executive Board in 2003.
- The adapted DQAF reviews four dimensions related to transparency, compilation, and dissemination:
  - Assurances of integrity: Is the TPI produced on an impartial and transparent basis?
  - Methodological soundness: Does the methodology capture all relevant concepts and definitions? Is it well documented?
  - Accuracy and reliability: Are the data sources used for compiling the TPI accurate, reliable, and timely?
  - Accessibility: Do the users of the TPI have easy access to the data, metadata, and relevant contact staff?

### Elements and ratings
- The adapted DQAF uses multiple elements covering:
  - Transparency, concepts and definitions, source data, statistical techniques, revisions policy, data and metadata accessibility, and assistance to users.
- Based on information made available by the compiler, each element is rated as:
  - observed (best practices),
  - partially observed, or
  - not observed.
- An element is not rated if it is not applicable or the assessment is not feasible.
- The rating of each dimension is generally the average rating of its elements.

### Presentation and interpretation of assessments
- The Indicators Digest facilitates comparison of relative strengths of indicators measuring similar concepts across the four assessed dimensions.
- Each spider chart indicates the assessment of the four dimensions; a rating away from center represents better data quality in that dimension.
- These assessments do not signal that one indicator is superior to others, since indicators differ in country coverage, time series, consistency, and transparency. Assessments are intended to serve as a robustness check.

### Governance, updates, and responsibilities
- The Indicators Digest will remain a living database with an annual review process led by SPR and STA.
- Expansion of the Digest will be demand-driven and focused on emerging and macro-critical areas; it will never be an exhaustive compilation of all possible TPIs.
- STA will be solely responsible for applying the adapted DQAF assessment to TPIs to ensure consistency and efficiency.
  - Centralization aims to provide an evenhanded and streamlined approach and reduce subjectivity.
  - STA will conduct assessments for any new TPIs considered for inclusion within the annual update process, in close coordination with SPR.
- Updates will be communicated to Fund departments through outreach events and internal communication.
  - SPR and STA may organize seminars to present annual updates, prioritizing departments that use new TPIs extensively.
  - Minor technical updates will be notified via email.
  - The latest version of the Indicators Digest will be disseminated through the SPR and STA internal websites.

### How to approach TPIs not in the Indicators Digest
- Staff can use TPIs both inside and outside the Indicators Digest; the Digest is not an ex-ante positive or negative list and should not stifle innovation.
- STA may provide a preliminary assessment for new TPIs not included in the Digest or advise other Fund staff on how to assess them.
  - Fund staff can request STA to review a TPI currently not included in the Digest.
  - In consultation with SPR, STA will consider a preliminary assessment depending on whether the indicator is widely used in the Fund and likely to be included in upcoming updates.
- Staff should not produce their own ratings based on the adapted DQAF, to preserve efficiency, consistency, and evenhandedness.
- STA will assess TPIs based on publicly available information and direct consultations with compilers, focusing on:
  - methodology for constructing the indicator,
  - types of data sources,
  - process of data collection and verification,
  - calculation of confidence bands,
  - any limitations for usage.
- STA may also consult other organizations with strong experience in relevant areas.

### Practical checklist for staff considering TPIs outside the Digest (Box 3)
- Issues staff should seek information on include:
  - Timeliness of the data;
  - Type of source data (i.e., is it an objective or subjective measure of the concept, per Figure 1);
  - Methodological approach used by the compiler;
  - Any structural breaks in the methodology that could impede comparison across time;
  - For survey-based TPIs, any deviations in the size or representativeness of samples that could impede comparison across countries and/or time;
  - For survey-based TPIs, confidence intervals or standard errors if reported by the compiler and their implications for comparisons across countries and/or time;
  - Characteristics of the compiler and its funding source, if available.
- It is not necessary to explicitly detail all this information in the staff report, but staff’s use of the indicator should reflect consideration of these issues.

### Examples of good practices and practices to avoid (Box 4 summary)
- Selected staff reports illustrate good practice elements in applying the TPI framework; these examples vary in extent of disclosure consistent with a risk-based approach.
- Good practice features across examples include:
  - Light touch applied for TPIs that measure economic/financial concepts directly and are frequently used in Fund reports (Poland 2017, Ghana 2017, South Africa 2017);
  - Elaboration on factors relevant for interpretation of TPIs and linking TPI performance to country-specific bottlenecks and reform agendas (Senegal 2016; El Salvador 2018; Colombia 2018);
  - Acknowledgement of perception-based TPIs and discussion of key characteristics and source data (Sri Lanka 2018; El Salvador 2018; Colombia 2018);
  - Presentation of confidence intervals in charts, use of point scores rather than rankings, grouping of peer comparators, and robustness checks by presenting related indicators (Sri Lanka 2018; El Salvador 2018).
- Examples also illustrate pitfalls to avoid, including overreliance on rankings without context, failing to acknowledge methodological changes or sample limitations, and not checking robustness with related indicators.

*Source: GUIDANCE NOTE FOR THE USE OF THIRD-PARTY INDICATORS (excerpt).*

### Box 4. Good Practice Examples (cont.)

### Box 4. Good Practice Examples (cont.)

### Union of the Comoros — Staff Report for the 2014 Article IV Consultation
- Perception-based TPIs acknowledged
- Reflects stakeholders’ views
- Considers subcomponents of index
- Staff pointed out that survey-based indicators point to competitiveness issues related to poor infrastructure and institutional and governance deficiencies that affect the quality of the business environment. Thus, Comoros ranks low on the list of the World Bank’s Doing Business Indicators (159th of 189 countries). Comoros ranks particularly poorly in the areas of starting a business, enforcing contracts, and government effectiveness. Comoros performs similarly poorly on comparative governance and corruption indicators. The authorities felt that progress in these areas had been made, especially by putting in place updated legislation, but they acknowledged that this had not yet resulted in improvements in the rankings and that it is important to ensure the consistent application of the new laws.

### Georgia — Request for Extended Arrangement Under the Extended Fund Facility and Cancellation of Stand-by Arrangement
- Analysis reflects country context and authorities’ reform agenda, and links TPI performance to specific bottlenecks and opportunities
- Charts use Distance to Frontier (DTF) scores and peer comparators
- Robustness checked by comparing results of WEF GCI and World Bank Doing Business indicators, and official and empirical data
- Refer to Annex I in staff report, including:

### Cabo Verde — Staff Report for the 2018 Article IV Consultation
- Perception-based TPIs acknowledged
- Uses point scores rather than rankings
- Peer comparators not identified individually
- Acknowledges key limitations of the TPI
- Narrative highlights:
  - Cabo Verde has made significant progress over the last few decades in economic and social development. Strong economic growth buoyed by a rapid expansion in tourism catapulted the country to middle income status. In the process, the incidence of poverty and extreme poverty, as well as inequality declined.
  - Cabo Verde has also made significant progress over the last two decades in improving governance and fighting corruption. However, during the global financial crisis, economic growth fell sharply as tourism receipts declined. Meanwhile, a significant scaling-up of public investment since 2009, and the support of loss-making SOEs brought public debt to 129.5 percent of GDP in 2016, twice the level at end-2009.

### WHD Regional Economic Outlook: Seizing the Momentum, April 2018, Chapter 2
- Perception-based TPIs acknowledged
- Key characteristics of indicators and their source data discussed
- Uses correct attribution for Worldwide Governance Indicators
- Uses point scores rather than rankings
- Peer comparators identified as grouping rather than individually
- Robustness clarified

### EUR Regional Economic Outlook: Europe Hitting Its Stride, November 2017, Chapter 2
- Perception-based TPIs acknowledged
- Key characteristics of indicators and their source data discussed
- Choice of indicator justified
- Peer comparators identified as grouping (average) rather than individually
- Uses point scores rather than rankings and considers sub-components of index
- Excerpted methodological note:
  - To have quantitative indicators over a long period for more countries and dimensions, and following most previous studies, we also employ data from the World Bank’s Worldwide Governance and Doing Business Indicators, the World Economic Forum, the Varieties of Democracy Institute, and other sources (Annex 2.2). Most of these data are perception based and thus more subjective than other economic indicators. Nevertheless, economic decisions are based on agents’ perceptions of many factors, including governance, effectiveness of the judiciary, and property rights protection. CoE 2015 notes that “...other factors, such as public perception, political culture and safeguards against corruption have a clear impact on the ability of courts and judges to command legitimacy and do their job.” The case studies that follow here rely on many sources to understand the context in which judicial reforms took place.
  - See also: Annex 2.2 Indicators and Sources. In line with the IMF Board paper on the “Use of Third-Party Indicators (TPIs) in Fund Reports” (IMF 2017c), this annex describes the indicators used and their sources. Specific descriptions of indicators used are described in Annex Table 2.2.1.

### Review of 1997 Guidance Note on Governance—A Proposed Framework for Enhanced Fund Engagement
- Perception-based TPIs acknowledged
- Key characteristics of indicators and their source data discussed
- Choice of indicator justified
- Robustness checked by comparing results of WGI-CCI and Transparency International GCB
- Methodological note:
  - The baseline estimates use the Control of Corruption Index scores from the Worldwide Governance Indicators Control of Corruption Index (WGI-CCI), which captures perceptions of corruption. The robustness of the results is checked using the Transparency International Global Corruption Barometer (GCB) indicator—which assesses people’s direct experiences with bribery and details their views on the overall prevalence of corruption in the main institutions in their respective country.
  - The WGI-CCI indicator captures perceptions of the extent to which power is exercised for private gain, including petty and grand forms of corruption, as well as “capture” of the state by elites and private interests. It is compiled and published by staff from the World Bank, Brookings Institutions, and Natural Resource Governance Institute, aggregating from more than 30 different surveys of enterprise, citizen, and expert perceptions. The GCB draws on survey feedback from more than 100,000 respondents in about 100 countries. It addresses people’s direct experiences with bribery and details their views on the overall prevalence of corruption in the main institutions in their countries. It also captures their perception of the government’s anti-corruption performances, amongst others. One shortcoming of the GCB surveys is that they capture the experiences of the population at large, which typically relates to street-level corruption, not grand corruption by high-level officials. Another limitation is that differences in reported bribery might in part be subject to cultural differences in respondents’ willingness to report it. The WGI-CCI is available annually for 1996–2016 and cover about 200 countries, whereas GCB is available for the period2003–16 and covers less than 100 countries in each year.

### Good-Practice Features Illustrated Across Examples
- Acknowledgement that many TPIs are perception-based
- Discussion of key characteristics of indicators and their source data
- Use of point scores rather than rankings where appropriate
- Consideration of sub-components of indices
- Use of peer comparators as groupings or averages rather than naming individual countries
- Robustness checks via comparison across multiple TPIs and with official/empirical data
- Correct attribution of indicator sources

### Box 5. Examples of Practices to Avoid
- Example 1
  - Does not use peer groups for cross-country comparisons, instead citing individual countries (identified by name)
  - Establishes ranking by presenting point scores measuring corruption for individual countries in bar chart
  - Does not reflect confidence intervals, which are available from compiler
  - Omits information from original source that is relevant to the interpretation of the data
- Example 2
  - Does not provide additional context when discussing business climate and simply relies on ranking
  - Does not discuss authorities’ views, despite the authorities having raised several methodological issues and additional context with a bearing on the interpretation of their performance according to the TPI
  - Illustrative text: The business climate in Country X is poor. According to the Doing Business report, the World Bank ranks Country X 120 out of 189 countries. This reflected a deterioration compared to the prior year’s report when Country X ranked 110 out of 189 countries. Country X should prioritize reforms to improve its business climate.
- Example 3
  - Does not acknowledge perception-based TPIs
  - Omits justification for identifying multiple individual countries in another country’s Article IV report, which is not otherwise self-evident

*Source: Box 4 and Box 5, Guidance Note for the Use of Third-Party Indicators in Fund Reports (excerpts).*

---


_Source: https://www.imf.org/-/media/files/publications/pp/2018/pp071718guidance-note-for-the-use-of-third-party-indicators-in-fund-reports.pdf_