On November 10, 2017, the Executive Board of the International Monetary Fund (IMF) discussed a staff paper on “Use of Third-Party Indicators in Fund Reports.”
The use of comparative indicators developed by other organizations, already widespread in core areas of Fund surveillance, is increasing in emerging macro-critical areas handled in response to the call from the membership. Fund staff rely on third-party indicators (TPIs) to measure concepts such as business environment, competitiveness, quality of governance, as well as progress toward social development goals to inform country-specific analysis across time and cross-country comparisons. These indicators play a role in supporting the evidence-based approach to advance work in priority areas such as risks and spillovers, macro-financial surveillance, and structural issues; in emerging macro-critical areas like inequality, gender, and climate change; and in program work. It is anticipated that staff will continue to draw on other institutions’ expertise and estimates, consistent with the Executive Board’s guidance to leverage outside expertise in areas where internal expertise is limited or lacking.
The paper outlines a framework to promote best practice with respect to use of TPIs in Fund reports. It has three elements: (i) a principles-based Guidance Note for staff (forthcoming); (ii) an “Indicators Digest” (a database for internal use only) that compiles selected indicators’ characteristics to inform staff’s judgment, based on the adapted Data Quality Assessment Framework (DQAF); and (iii) the review process to ensure best practice in Fund reports. Third-party indicators should not replace—but rather supplement—an open and candid discussion with the authorities. The framework will apply to all country documents, policy documents, and multi-country documents that are subject to the Fund’s Transparency Policy. To promote best practice, staff are encouraged to follow similar guidelines for other Fund documents that are not subject to the Transparency Policy (e.g., Regional Economic Outlooks and Staff Discussion Notes).
Executive Board Assessment [1]
Executive Directors welcomed the discussion of the use of third‑party indicators (TPIs) in Fund reports. They took note of recent experiences in the Fund and other international organizations, which indicated that TPIs have been useful in facilitating cross‑country comparisons, identifying evidence to help inform analysis and complement assessments in areas where an institution may not gather its own primary data. Directors recognized that staff’s continued practice of drawing on other institutions’ expertise and estimates is consistent with the Executive Board’s guidance in areas where internal expertise is lacking or limited. Nevertheless, they agreed that a more structured approach to using TPIs would help promote best practice and ensure consistency and evenhandedness.
Directors noted that the varied qualities of TPIs currently used by staff present challenges and risks to the Fund’s credibility. They observed that some TPIs are based on hard data, while others are based on qualitative assessment by experts, experience, perception, or composites of various underlying data sources. Directors were concerned about the opacity of some TPIs’ sources and methodologies, and the quality and reliability of TPIs that are based on perception or value judgment and those produced by private entities. The lack of transparency in staff’s choice of some TPIs or their components was also raised as a concern.
Directors welcomed the proposal to address these concerns and promote best practice with respect to use of TPIs in Fund reports. The proposed framework consists of principles‑based guidance for staff; an “Indicators Digest” as an internal, centralized quality assessment database; and an internal review process. Directors noted that this approach provides room for staff judgment and flexibility to make progress on surveillance priorities while avoiding flawed analysis and reputational risks. They also appreciated the proposal to apply this framework to the use of TPIs in all documents that are subject to the Fund’s Transparency Policy, and asked staff to follow similar guidelines for other Fund documents. Many Directors also stressed the need to extend the framework consistently across all Fund publications. Whenever TPIs are used, many Directors called on staff to provide a clear justification, and a few suggested including an appropriate disclaimer.
Directors endorsed the three complementary principles for best practice—transparency in the selection and interpretation of the indicators, robustness checks, and presentation of the views of stakeholders. They stressed the importance of transparency in selecting indicators, including a discussion of their key characteristics and limitations, as well as how results are used to inform the overall assessment. Directors stressed that the use of indicators should be as one of many inputs and a complement to qualitative discussion, which draws on multiple indicators and sources that measure similar concepts and supplements other facts or information to promote robustness. A number of Directors cautioned in this regard that indicators derived from the same underlying data might be correlated and thus biased in the same direction. Directors also emphasized that, while Fund reports should present staff’s independent and candid views, they should include the views of the authorities and other key stakeholders, especially when they have different interpretations of the indicators or further contextual information.
Directors appreciated staff efforts to put together the Indicators Digest, applying an adapted version of the Board‑endorsed Data Quality Assessment Framework. They took note that the Indicators Digest is not an ex‑ante positive or negative list of acceptable indicators for use in staff analysis and Fund work. They also recognized that the Digest, as a living database, would not be an exhaustive compilation of all possible TPIs staff may employ or a validation of those TPIs. Directors agreed that its future expansion should be demand‑driven. Many Directors felt that the Indicators Digest could include even greater details on the nature of data, methodology used for compiling each indicator, and specific components that are actionable for governments.
In applying the framework for use of TPIs, Directors attached particular importance to their judicious and evenhanded use based on macro‑criticality, and considerations of country‑specific contexts in interpreting results. They stressed that TPIs should not replace—but rather supplement—an open, candid, robust, and well‑documented discussion with the authorities. Many Directors also attached priority to data‑based TPIs over perception‑based ones, while a number of other Directors noted the potential value of perception‑based indicators. Many Directors considered it important that the Board be able to access TPI data used by staff. In emerging areas that are deemed macro‑critical and relevant to the Fund’s mandate, many Directors noted that it would be preferable to develop in‑house expertise, while many others saw a need to be mindful of resource implications. Directors underlined the importance of collaboration with other international organizations.
Directors’ views expressed today provide useful inputs into the forthcoming Guidance Note for staff. Directors looked forward to a further opportunity to engage with staff before its finalization.
[1] An explanation of any qualifiers used in summings up can be found here: http://www.imf.org/external/np/sec/misc/qualifiers.htm
