## REPUBLIC OF KAZAKHSTAN — Scanner Data for CPI Mission

## Source details

**Canonical URL:** [REPUBLIC OF KAZAKHSTAN — Scanner Data for CPI Mission](https://www.imf.org/-/media/files/publications/cr/2022/english/1kazea2022005.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/cr/2022/english/1kazea2022005.pdf.md)
- [Structured JSON version](/-/media/files/publications/cr/2022/english/1kazea2022005.pdf.json)

---

### Summary of mission outcomes and priority recommendations
- Purpose: Assist the Bureau of National Statistics of the Agency for Strategic Planning and Reforms of the Republic of Kazakhstan (BNS) with continuing its modernization of the Consumer Price Index (CPI). First technical assistance (TA) mission to Kazakhstan on scanner data (SD). Mission completed remotely; based on information available as of November 2021.
- Objective timelines:
  - BNS aims at introducing scanner data in the CPI as a new data source by January 2024.
  - Experimental indices should be tested during 2023 and 2024.
- Expected benefits:
  - Increase in the quality of the CPI.
  - Reduction in CPI production costs and respondent burden over the long run through reduced in-store price collection and automated data transmission and processing.
  - Upfront investment required in human resources skills and IT infrastructure during research and implementation phases.
- Current data access and status:
  - BNS expects access to scanner data through an online platform provided by the State Revenue Committee (SRC). The system is a work in progress and should be implemented by the first quarter of 2022.
  - SRC is completing received retailer data by introducing product codes.
- Immediate coordination needs:
  - Meetings with SRC to secure data transmission, agree on terms, clarify data contents, and confirm alignment of product codes with tight product specifications.
- Recommended phased approach:
  - Introduce scanner data beginning with food products and a small sample of the larger retailers.
  - Recommend SRC focus on coding food products to speed access.
- Methodology focus:
  - Mission focused on two methods: the dynamic basket approach and the multilateral method, weighted time product dummy (WTPD).
- Follow-up:
  - Once methodology is chosen and sufficient data received, a follow-up mission will be needed for practical guidance.

### Priority recommendations and action plan (selected)
- Table 1 summary (priority recommendations):
  - March 2022: Obtain data with the required characteristics. Responsible: BNS and SRC.
  - July 2022: Using three months of data to classify Food products to the COICOP. Responsible: BNS.
- Detailed priority actions and target dates:
  - Introduce scanner data with food products and a small sample of the larger retailers. Target: January 2024.
  - Have the SRC focusing on coding the food products. Target: March 2022.
  - Meet with SRC to clarify the characteristics of the data. Target: March 2022.
  - Create a variable identifying the region and the district to which each observation pertains. Target: March 2022.
  - Download data at least every week to save, analyze and classify to the COICOP. Target: Continuous.
  - Use the first three weeks for the index calculation. Target: Continuous.
  - Create a conversion table to link the product code to the National breakdown of the COICOP. Target: September 2022.
  - Sum the turnover and the quantities of the weekly data. Target: Continuous.
  - Choose the compilation method. Target: September 2022.

*Recommended immediate actions:*
- Introduce scanner data with food products and a small sample of the larger retailers.
- Have the SRC focusing on coding the food products.
- Meet with SRC to clarify the characteristics of the data.
- Create a variable identifying the region and the district to which each observation pertains.
- Download data at least every week to save, analyze and classify to the COICOP.
- Use the first three weeks for the index calculation.

### A. Data specifications
- Expected data access:
  - Access via SRC online platform; implementation expected by the first quarter of 2022.
  - SRC completing data with product codes.
- Recommended scope and sequencing:
  - Stepwise introduction beginning with food products and a small sample of larger retailers.
  - To speed access, SRC should prioritize coding food products.
- Sample data and product coding requirements:
  - Obtain sample data as soon as possible to ensure data structure fits CPI needs.
  - Confirm product codes respect tight product specifications (example: Brand A 0,5 liters vs Brand A 1 liter must have different codes).
- Key variables required in scanner data files (Table 2 — Key Variables of a Scanner Data File):
  - Start date of the week
  - End date of the week
  - Code of the product
  - Brand of the product
  - Quantity in the package
  - Product description
  - Turnover
  - Total quantities sold
  - Sold by units or by weight
  - Region
  - District
  - Retailer
- Retailer coverage and regional mapping:
  - Currently data will be available for three retailers.
  - BNS will receive retailer name and address; create a variable identifying region and district for each observation.
- Data handling frequency:
  - Download data from the online system at least every week to create redundancy and safety.
  - Downloaded data should be analyzed and classified to COICOP on a weekly basis.
  - The first three weeks of data should be used for index calculation.

### B. Data analysis and COICOP classification
- Recommended software and initial checks:
  - Perform all data handling in R for reproducibility and lower error risk.
  - Initial data checks to perform:
    - Checking the file structure
    - Checking the size of the files
    - Checking the number of items per period and district/region
    - Checking for duplicate records
    - Checking for missing values
    - Checking for non-positive values in quantities sold or turnovers
    - Checking the sum of turnover over time
- Conversion table for COICOP mapping:
  - Create a conversion table linking product code to the National breakdown of the COICOP.
  - Conversion table structure: Product code, Product description, COICOP code (National Classification level), COICOP description.
  - Initially time consuming; classification effort declines over time as only new products require manual classification.
- Machine learning considerations:
  - Product labels may be used for classification via text mining and machine learning.
  - Recommendation: use supervised machine learning; a learning dataset is essential.

*Recommended action:*
- Create a conversion table to link the product code to the National breakdown of the COICOP.

### C. Methodology
- Preprocessing:
  - Sum turnover and quantities of the weekly data for three weeks by product code and district.
  - Compute unit price as turnover divided by quantities.
- Two primary methods reviewed:
  - Dynamic basket approach (matched model with monthly re-sample).
  - Multilateral method: Weighted Time Product Dummy (WTPD).
- Dynamic basket approach details:
  - Monthly re-sample based on market shares after COICOP classification at the national classification lowest level.
  - Sampling formula provided (as in source):
    - s_s_m + s_s_m-1 / 2 > 1 / n * λ
    - λ = 1,25
    - n = Number of matching products
    - s_s_m = Market share of a product in month m
    - s_s_m-1 = Market share of a product in month m-1
  - Products transacted in two consecutive months are considered; product included if market share exceeds threshold which varies by number of products in consecutive months for that COICOP.
  - After sample selection, monthly price changes are calculated and a Jevons (geometric average) is used to aggregate prices to national COICOP level.
  - Missing product treatment:
    - Products in previous sample that go missing continue being included and their prices are estimated.
    - For scanner data, missing products are estimated for 14 months.
    - Estimation: previous price multiplied by the rate of change of the same COICOP (same method as in-person price collection).
  - Outliers and dumped prices:
    - Thresholds: price variations higher than 300 percent and lower than 75 percent flagged.
    - Dumped prices: turnover decreases more than 75 percent and prices more than 70 percent.
    - Identified wrong prices are replaced by estimated prices.
- WTPD (multilateral) method details:
  - Uses weighted least squares regression with turnover shares of products acting as weights.
  - Definitions from source:
    - S_it = p_it q_it / ∑ p_it q_it, i ∈ N_t
    - Model: ln p_it = α + Σ_t δ_t D_it + Σ_i γ_i D_i + ε_it
    - [0, T] is the time window
    - N is the number of available products
    - D_it is a dummy variable equal to 1 if product i present in period t, 0 otherwise
    - D_i is a dummy variable equal to 1 if observation relates to product i, 0 otherwise
  - Practical recommendation: use a time window of 25 months; 25 months of data pulled into regression using weighted least squares; indices derived by exponentiating time-dummy coefficients δ_t.
- Data requirements for method choice:
  - Dynamic basket experiments can begin with 6 months of data.
  - WTPD requires a minimum of 13 months of data.
  - Common practice: begin with dynamic approach and move to a multilateral index after about 2 years.
- Splicing for rolling-window methods:
  - A splicing method should be used when using rolling window methods; mission recommends the example in the CPI manual (example uses a 13-month window in the manual).
- Aggregation with existing sources:
  - Once sub-indices from scanner data are obtained, aggregate them with sub-indices produced from in-person price collection following provided aggregation schema.

*Recommended actions:*
- Create a conversion table to link the product code to the National breakdown of the COICOP.
- Sum the turnover and the quantities of the weekly data.
- Choose the compilation method (consider data availability: 6 months for dynamic basket experiments, 13 months minimum for WTPD).

### D. Officials met during the mission
- Kuanyshbek Zhakypbekuly — Director of the Price Statistics Department
- Altynai Kumasheva — Head of the Consumer Price Statistics Directorate, National Statistics Bureau of Kazakhstan
- Assel Zhunussova — Chief expert, National Statistics Bureau of Kazakhstan
- Kanat Ukibasov — Chief expert, National Statistics Bureau of Kazakhstan
- Meirambek Zhumadildin — Chief expert, National Statistics Bureau of Kazakhstan

*Source: Technical Assistance Report — Scanner Data for CPI Mission (November 8–12, 2021), IMF Country Report No. 22/141.*

---


_Source: https://www.imf.org/-/media/files/publications/cr/2022/english/1kazea2022005.pdf_