## The Availability, Methodological Soundness, and Scope of Consumer Price Statistics in 2020 — Introduction and Background

## Source details

**Canonical URL:** [The Availability, Methodological Soundness, and Scope of Consumer Price Statistics in 2020 — Introduction and Background](https://www.imf.org/-/media/files/publications/wp/2022/english/wpiea2022053-print-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2022/english/wpiea2022053-print-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2022/english/wpiea2022053-print-pdf.pdf.json)

---

### Survey and Purpose
- Survey launched in March 2021 by the IMF Statistics Department to capture CPI compilation practices in 207 economies.
- Online survey comprised 56 questions.
- Responses prepopulated by IMF economists where available and sent to country CPI compilers for confirmation/updating.
- 52 percent of survey respondents confirmed or updated the prepopulated questionnaire; for remaining countries, responses reflect only information obtained by IMF economists.
- Participating countries grouped for regional comparison using four nomenclatures: Analytical Country Groups, United Nations geographic regions (M49 Regions), IMF Area Departments (AD Regions), and countries covered by various IMF Regional Capacity Development Centers (RCDCs).

### Role and Concept of the CPI
- CPI definition quoted from the Consumer Price Index Manual: Concepts and Methods: “CPIs measure changes over time in the general level of prices of goods and services that households acquire (use or pay for) for the purpose of consumption.”
- Survey aims:
  - monitoring progress on the development of CPI statistics in member economies;
  - assessing some quality-related features of the CPI; and
  - surveillance purposes.

### Production, Publication, and Scope
- CPI release frequency:
  - High-frequency indicator released monthly (or quarterly in some cases).
  - Governments generally adopt a non-revisions policy; revisions occur exceptionally for significant errors.
- SDDS and e-GDDS recommendations and compliance:
  - The SDDS and the e-GDDS recommend monthly dissemination of the CPI.
  - 187 of surveyed countries (90 percent) publish monthly.
  - 16 surveyed countries publish a quarterly CPI.
  - One surveyed country has suspended publication of their CPI and no information is available for three countries.
- CPI weights:
  - International standards recommend updating CPI expenditure weights at least every five years.
  - 106 member countries had a weight reference period within 5 years or less of 2020.
  - Information on the weight reference period is not available for 16 countries.
- Geographic coverage:
  - 67 percent of countries collect expenditure data in both rural and urban areas (weights have full country coverage).
  - Noted regional patterns: many countries in South America and Oceania mostly cover urban areas only; Europe and Asia are regions where weights are more frequently updated.
- Timeliness:
  - e-GDDS encourages dissemination within eight weeks of the reference period; SDDS requires dissemination within four weeks.
  - Only five surveyed countries failed to meet these timeliness requirements.
  - All Advanced Economies and 75 percent of Emerging and Developing Economies release the CPI with no more than a four-week lag.
- Publication of detailed weights:
  - 133 countries publish the detailed weights on their website.

### Data Sources for Weights and Prices
- Two types of data underpin CPI calculation:
  - prices of items in the CPI basket (to measure price changes); and
  - households’ expenditures data (to aggregate elementary indices into higher-level indices).
- Household Budget Survey (HBS):
  - All respondents confirmed using the HBS as a data source to derive and validate CPI weights.
- Other sources for weights:
  - National accounts data are the second most common source; other sources include retail sales data and administrative data.
- Price collection methods and sources:
  - Prices collected locally (data collectors visit physical outlets) or centrally (NSO staff collect prices from national chains, fixed/regulated prices from administrative sources, websites, and scanner data).
  - 80 countries retrieve price data from websites manually.
  - Administrative files are used by 61 member countries for price collection.
- New data sources:
  - Web scraping and scanner data increasingly used; COVID-19 lockdowns accelerated use due to limitations on in-person collection.

### Use of New Data Sources (2020)
- 26 countries reported using scanner data in 2020.
- 31 countries reported using web scraping in 2020.
- Source definitions:
  - “Web scraping is an automated system to retrieve data from websites.”
  - “Scanner data are files transmitted by the retailers that contain data on individual transactions, including date, quantities and values, and detailed variety characteristics.”
- Text note: “These figures are expected to increase in the coming years.”

### Price Collection Period and Frequency
- Price collection should be spread over the reference month to reflect price developments throughout the month.
- Frequency varies by product:
  - School fees: often collected once per year.
  - Some items: collected twice per year.
  - Fresh fruits and vegetables: “should be collected several times during the month.”
- Duration of price collection period varies across countries driven largely by budgetary and staff resources.
- Source observation: “Figure 12 shows that most countries collect prices throughout the month.”
- CPI Manual guidance quoted: ideally collect prices from different outlets throughout the month to reflect average price for the reference month.

### Treatment of Missing Prices
- Temporarily missing (non-seasonal) varieties: recommended to impute prices rather than carry forward previous price or exclude the variety.
- Survey finding: “The vast majority of countries indicated that they impute for missing prices rather than carrying the price forward from the previous period or removing the variety from the index calculation.”
- Permanently missing varieties: replacements should be made; if replacements differ in price-determining characteristics, quality adjustment is required.

### Quality Adjustment Methods and Extent
- Various methods exist to account for changes in quality.
- Table 1 (referenced) displays country counts by region using different quality adjustment methods.
- Eleven countries indicated that they do not perform any quality adjustments.

### Index Calculation Methods
- Two-stage calculation: elementary indices and upper-level indices.
- Elementary indices:
  - CPI Manual recommends geometric mean formulas (Jevons formula).
  - With Jevons, elementary index can be geometric mean of price relatives or relative of geometric mean prices; “With Jevons, both methods provide the same result.”
  - 132 countries reported using the Jevons formula for elementary indices.
- Upper-level indices:
  - Short-term formula (chaining short-term price changes) versus long-term formula (calculating long-term price change directly each month).
  - Short-term formula recommended and more common because it is more flexible, facilitates introduction of new/replacement varieties, and facilitates data validation.
  - Source: “Both methods are used by countries, however the short-term formula is more common.”

### Concepts, Classifications, and Metadata
- Internationally recommended classification: UN’s COICOP.
- COICOP 1999 is used in the large majority of countries.
- COICOP 2018 released; “In the next few years, it is expected that countries will transition to COICOP 2018.”
- Some larger economies (examples in source): “Australia, Brazil, China, India, and the U.S., use their own national classifications instead of COICOP.”
- Metadata and sources/methods documents:
  - Most countries have produced metadata or a sources and methods document on their websites.
  - “The level of detail recorded is usually not adequate to fully assess compliance with international standards and quality standards such as the IMF’s Data Quality Assessment Framework.”
  - Noted deficiency especially in Africa, where most countries do not publish methodology documents on their websites.

### Key Findings and Conclusions
- Overall assessment:
  - CPI statistics are generally of high quality, but significant regional disparities exist.
- Frequency of release:
  - CPIs released monthly in most countries; exceptions include Australia, New Zealand and some other Oceania countries that disseminate quarterly.
- Weight updates:
  - “The frequency and timeliness of CPI weight updates, however, do not follow recommended best practice in many countries,” potentially causing CPIs to be “not adequately representative of current expenditure patterns” and introducing bias.
- Geographic coverage:
  - Majority of countries cover expenditure of both urban and rural areas; a relatively few countries cover the capital city only.
- Timeliness:
  - “Timeliness of publication is generally good with almost every country releasing the CPI less than four weeks after the reference period.”
- Data sources:
  - Weights and prices mostly derived from direct collection from households and outlets.
  - Use of new data sources such as web scraping and scanner data remains limited.
- Compilation methods:
  - Broadly in line with international best-practices for missing price estimation, quality adjustment methods, and index calculation formulas.
- Classification adoption:
  - 157 surveyed countries reported using COICOP 1999.
- Transparency concern:
  - Major concern is lack of publicly available information on CPI sources and methods, particularly acute in Africa; for many countries that publish metadata, “the level of detail is well below full transparency.”

### Looking Forward / Recommendations
- Modernize price collection by increasing use of new data sources (web scraping, scanner data).
- Improve frequency and timeliness of CPI weight updates to better reflect current expenditure patterns and reduce bias.
- Increase publication of detailed metadata/sources and methods to enhance transparency and enable assessment against international standards.
- IMF Statistics Department intentions:
  - Conduct a CPI compilation practices survey on an annual basis to monitor progress and assess global CPI quality.
  - Use survey information to help member countries benchmark their CPI program with regional and global benchmarks.

*IMF Working Paper — The Availability, Methodological Soundness, and Scope of Consumer Price Statistics in 2020 (Introduction and Background).*

### Introduction and Background ............................................................................................

### Introduction and Background

### Survey and Purpose
- In March 2021, the IMF Statistics Department launched a new global survey to capture a broad set of data on the Consumer Price Index (CPI) compilation practices in 207 economies.
- The online survey comprised 56 questions.
- Responses for most questions were prepopulated by IMF economists with the latest available information, where available; the prepopulated questionnaire was then sent to country CPI compilers for confirmation and updating.
- The information was confirmed or updated by 52 percent of survey respondents in the participating countries. For the remaining countries, the responses reflect only the information obtained by IMF economists.
- The survey groups participating countries for regional comparison using four nomenclatures: Analytical Country Groups used in the World Economic Outlook and other IMF publications; United Nations geographic regions (M49 Regions); IMF Area Departments (AD Regions); and countries covered by various IMF Regional Capacity Development Centers (RCDCs).

### Role and Concept of the CPI
- “CPIs measure changes over time in the general level of prices of goods and services that households acquire (use or pay for) for the purpose of consumption.” — Consumer Price Index Manual: Concepts and Methods
- The survey aims to provide comprehensive and timely information on the methodology of the CPI for each member country, facilitating international and regional benchmarking and comparison, useful for:
  - monitoring progress on the development of CPI statistics in member economies;
  - assessing some quality-related features of the CPI; and
  - surveillance purposes.

### Production, Publication, and Scope
- The CPI is a high frequency indicator that is released monthly (or quarterly in some cases).
- Governments generally adopt a non-revisions policy for CPIs; revisions can occur on an exceptional basis when a significant error is detected.
- Box on uses: “A CPI can be used for a variety of purposes, the most common ones being the indexation of wages, rents, contracts, and social security payments; the deflation of household final consumption expenditure in the national accounts; and the use as a general macroeconomic indicator, especially for inflation targeting and for setting interest rates. Elements of a CPI are also often used in the calculation of purchasing power parities and extrapolating purchasing power parities between benchmark years as required in the International Comparison Program (ICP).” — Consumer Price Index Manual: Concepts and Methods
- The IMF’s SDDS and the e-GDDS recommend monthly dissemination of the CPI.
  - 187 of surveyed countries (90 percent) follow these recommendations and publish monthly.
  - 16 surveyed countries publish a quarterly CPI (examples include Australia, New Zealand, and other Oceania countries).
  - One surveyed country has suspended publication of their CPI and no information is available for three countries.
- CPI weights:
  - International standards recommend that CPI expenditure weights should be updated at least every five years.
  - 106 member countries had a weight reference period within 5 years or less of 2020.
  - Information on the weight reference period is not available for 16 countries.
- Geographic coverage:
  - Coverage of expenditure weights in both rural and urban areas indicates national coverage.
  - 67 percent of countries collect expenditure data in both rural and urban areas (weights have full country coverage).
  - Noted regional patterns: many countries in South America and Oceania mostly cover urban areas only; Europe and Asia are regions where weights are more frequently updated.
- Timeliness:
  - The e-GDDS encourages dissemination within eight weeks of the reference period; the SDDS requires dissemination within four weeks of the reference period.
  - Only five surveyed countries failed to meet these timeliness requirements.
  - All Advanced Economies and 75 percent of Emerging and Developing Economies release the CPI with no more than a four-week lag.
- Publication of detailed weights:
  - 133 countries publish the detailed weights on their website.
  - Detailed weights are fundamental for understanding which expenditure categories drive overall CPI changes and for constructing measures such as core inflation.

### Data Sources
- CPI calculation relies on two types of data:
  - prices of the items included in the CPI basket (to measure price changes); and
  - households’ expenditures data (to aggregate elementary indices into higher-level indices).
- Household Budget Survey (HBS):
  - Most countries conduct an HBS to derive expenditure weights.
  - All respondents confirmed using the HBS as a data source to derive and validate the CPI weights.
- Other sources to derive and validate weights:
  - National accounts data are the second most common source.
  - Other sources include retail sales data and administrative data.
- Price collection methods and sources:
  - Prices are collected locally (data collectors visit physical outlets) or centrally (collected by NSO staff, including prices from national chains, fixed/regulated prices from administrative sources, prices from websites, and scanner data).
  - Websites are increasingly used as a source of price data; data from websites can be collected manually or by web scraping.
    - 80 countries are retrieving data from websites manually.
  - Administrative files from national regulatory bodies, ministries, or other sources are used by 61 member countries for price collection.
- New data sources:
  - Web scraping and scanner data are noted as new data sources for CPI prices.
  - Lockdowns and restrictions on movement in response to the COVID-19 pandemic limited in-person price collection in many countries, increasing relevance of alternative sources.

*Italic source attribution: IMF Working Paper — The Availability, Methodological Soundness, and Scope of Consumer Price Statistics in 2020 (Introduction and Background).*

### 2020. These restrictions accelerated the move by several national statistics offices to implement or expand the

### The Availability, Methodological Soundness, and Scope of Consumer Price Statistics in 2020

### Use of New Data Sources for Prices
- 26 countries reported using scanner data in 2020.
- 31 countries reported using web scraping in 2020.
- The text states: "These figures are expected to increase in the coming years."
- Definitions from the source:
  - "Web scraping is an automated system to retrieve data from websites."
  - "Scanner data are files transmitted by the retailers that contain data on individual transactions, including date, quantities and values, and detailed variety characteristics."

### Price Collection Period and Frequency
- Price collection should be spread over the reference month to reflect price developments throughout the month.
- Frequency of price collection varies by product depending on price-change frequency:
  - School fees: often collected once per year because tuition and fees change only at the beginning of each academic year.
  - Some items: may be collected twice per year.
  - Fresh fruits and vegetables: "should be collected several times during the month" due to higher volatility.
- The duration of the price collection period varies across countries and is driven largely by budgetary and staff resources.
- The source notes: "Figure 12 shows that most countries collect prices throughout the month."
- Quoted guidance from the CPI Manual: "Ideally, price collection would be organized so that prices are collected from different outlets throughout the month. This ensures that the prices used to calculate the index more broadly reflect the average price for the reference month."

### Treatment of Missing Prices
- Sampled varieties can be temporarily or permanently missing.
- For temporarily missing varieties (non-seasonal), recommended practice is to impute prices; carrying forward the previous price observation or excluding the missing variety from the index calculation are discouraged.
- The source reports: "The vast majority of countries indicated that they impute for missing prices rather than carrying the price forward from the previous period or removing the variety from the index calculation."
- For permanently missing varieties, replacements should be made; when replacements differ in price-determining characteristics, quality adjustment is required so the CPI reflects only pure price change.

### Quality Adjustment Methods and Extent
- "Different methods can be used to account for changes in quality."
- Table 1 (referenced) displays the number of countries by region that apply different quality adjustment methods.
- Eleven countries indicated that they do not perform any quality adjustments.

### Index Calculation Methods
- CPI calculation occurs in two stages: elementary indices and upper-level indices.
- Elementary indices:
  - The CPI Manual recommends using geometric mean formulas, generally referred to as the Jevons formula.
  - With the Jevons formula, the elementary index can be calculated either as the geometric mean of price relatives or the relative of geometric mean prices; "With Jevons, both methods provide the same result."
  - 132 countries reported using the Jevons formula to calculate elementary indices.
- Upper-level indices:
  - Can be calculated using a short-term formula (chaining short-term price changes) or a long-term formula (calculating long-term price change directly each month).
  - The short-term formula is recommended and more common because:
    - It is more flexible.
    - It facilitates the introduction of new or replacement varieties without imputing base period prices when two successive periods are available.
    - It facilitates data validation by identifying extreme short-term price movements.
  - The source: "Both methods are used by countries, however the short-term formula is more common."

### Concepts, Classifications, and Metadata
- The internationally recommended classification system for the CPI is the United Nations’ Classification of Individual Consumption According to Purpose (COICOP).
- COICOP 1999 is used in the large majority of countries.
- The United Nations released an updated version, COICOP 2018; "In the next few years, it is expected that countries will transition to COICOP 2018."
- Some larger economies (examples in source): "Australia, Brazil, China, India, and the U.S., use their own national classifications instead of COICOP."
- Most countries have produced metadata or a sources and methods document available on their websites, but:
  - "The level of detail recorded is usually not adequate to fully assess compliance with international standards and quality standards such as the IMF’s Data Quality Assessment Framework."
  - The source includes a question: "Is There a Document Explaining Current CPI Compilation Methods Posted on Your Website?" and notes varying responses; in particular, "This is especially true in Africa, where most countries do not publish methodology documents on their websites."

### Key Findings and Conclusions
- CPI statistics are generally of high quality, but there are significant regional disparities.
- Frequency of CPI release:
  - CPIs are released monthly in most countries.
  - Exceptions: Australia, New Zealand and some other countries in Oceania disseminate quarterly.
- Weight updates:
  - "The frequency and timeliness of CPI weight updates, however, do not follow recommended best practice in many countries," which may cause CPIs to be "not adequately representative of current expenditure patterns" and could "introduce bias into the index."
- Geographic coverage:
  - "The CPI in the majority of countries covers the expenditure of both urban and rural areas, with a relatively few countries that cover the expenditure of the capital city only."
- Timeliness:
  - "Timeliness of publication is generally good with almost every country releasing the CPI less than four weeks after the reference period."
- Data sources:
  - Data sources used for weights and prices are mostly limited to direct collection from households and outlets.
  - "The use of new data sources for price collection, such as web scraping and scanner data, is still limited."
- Compilation methods:
  - "Compilation methods are broadly in line with international best-practices and recommendations," notably for missing price estimation, quality adjustment methods, and index calculation formulas.
- Classification adoption:
  - As of 2020, "157 surveyed countries reported using COICOP 1999."
- Transparency concern:
  - "A major point of concern is the lack of information publicly available on the sources and methods used to compile the CPI," particularly acute in Africa.
  - For many countries that publish metadata, "the level of detail is well below full transparency."

### Looking Forward / Recommendations from the Text
- Opportunities to modernize price collection by increasing use of new data sources (web scraping, scanner data).
- Improve frequency and timeliness of CPI weight updates to better reflect current expenditure patterns and reduce bias.
- Increase publication of detailed metadata/sources and methods to enhance transparency and enable assessment against international standards.
- The IMF Statistics Department intends to:
  - Conduct a CPI compilation practices survey on an annual basis to monitor progress and assess the global quality of CPI statistics.
  - Use this information to help member countries benchmark their CPI program with regional and global benchmarks.

*IMF Working Paper No. WP/22/53 — The Availability, Methodological Soundness, and Scope of Consumer Price Statistics in 2020*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2022/english/wpiea2022053-print-pdf.pdf_