## Section I. Detailed Technical Assessment and Recommendations

## Source details

**Canonical URL:** [Section I. Detailed Technical Assessment and Recommendations](https://www.imf.org/-/media/files/publications/tar/2024/english/tarea2024058-print-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/tar/2024/english/tarea2024058-print-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/tar/2024/english/tarea2024058-print-pdf.pdf.json)

---

### Summary of mission outcomes and priority recommendations
- A technical assistance (TA) mission was conducted in Vienna, Austria to assist the State Statistics Service of Ukraine (SSSU) with introducing administrative scanner data for the compilation of the consumer price index (CPI).
- The SSSU plans to use administrative scanner data from the State Tax Service (STS), covering transactions recorded by cash register machines that retailers have been legally required to use.
- Incorporating scanner data will improve the quality of the CPI and reduce field price collection by improving regional, product, and temporal coverages.
- Weighted price indices can be compiled because price and revenue information are available in scanner data.
- Pre-processing by the STS is required before transmission to the SSSU. Transactions referring to the same item sold in all outlets of a region during a given month should be combined. In the absence of a formal item code, the text label must be used to identify transactions of the same item.
- An average price (unit value) is obtained by dividing total revenue by total number of units sold. Some data cleaning is required before calculating these average prices.
- The SSSU has developed string-searching algorithms to identify patterns in item labels and automatically assign items to COICOP categories.
- The SSSU should agree with the STS on data characteristics to be made available: pre-processing calculations, variables included, frequency and timeliness, and provision of back data for the first transmission. Update the memorandum of understanding (MOU) accordingly.
- The SSSU should link scanner-data items to the lowest level of the CPI hierarchy (the 335 representative products, 7-digit COICOP), review and update classifications suggested by string-searching, establish processes and tools to support classification, and implement outlier methods to detect and remove items with unusual prices.
- Experimental price indices should be calculated using the fixed basket method for a limited selection of products, at the representative product by region level, to facilitate consistent comparisons with current price series and future integration.
- Action plan priority recommendations:
  - August 1st 2024: Arrange regular access to the administrative scanner data — Responsible: SSSU.
  - December 31st 2024: Calculate experimental price indices from the administrative scanner data — Responsible: SSSU.

### Action plan (selected items)
- Data access and data preparation
  - High — Agree with the STS on the structure of the data to be transmitted to the SSSU and sign the additional agreement — Target Completion Date: May 15st 2024
  - High — Prepare the data flow and transmission protocol according to the agreement — Target Completion Date: June 1st 2024
  - High — Start accessing the data on a monthly basis — Target Completion Date: July 1st 2024
  - High — Confirm the quality of the received data files — Target Completion Date: August 1st 2024
- Classification
  - High — Specify pattern matching rules for classification — Target Completion Date: Completed
  - Medium — Discuss with the STS about availability of additional information on the classification of the items — Target Completion Date: September 1st 2024
  - Medium — Build a reference table for archiving the link between items and COICOP categories — Target Completion Date: September 1st 2024
  - High — Review and update pre-classified data for the products that are in-scope of the pilot study — Target Completion Date: September 1st 2024
- Data cleaning
  - Medium — Discuss with the STS on applying the selected outlier methods during the initial transformations of the raw data — Target Completion Date: May 1st 2024
  - Medium — Decide on the thresholds for the outlier and dumping filters and implement these rules on the data — Target Completion Date: October 1st 2024
- Index compilation
  - High — Draw samples for the initial basket — Target Completion Date: October 1st 2024
  - High — Decide on the principles for the treatment of replacements — Target Completion Date: October 1st 2024
  - Medium — Start preparing the IT tools for index compilation — Target Completion Date: November 1st 2024
  - High — Calculate experimental price indices for a selection of products — Target Completion Date: December 31st 2024
  - High — Compare the results to the current price series and adjust the methodology if needed — Target Completion Date: December 31st 2024
  - Medium — Update the methodological document for the pilot study — Target Completion Date: February 1st 2025

### Data collection and data preparation — findings and recommendations
- Data characteristics and scale
  - The STS data are big data; one day of transactions can amount to over 77 million lines, each line representing a transaction for a specific item.
  - Currently data are collected from over 900,000 cash register machines across the country.
- Recommended STS pre-processing and transmitted variables
  - Some initial transformations and pre-aggregation should be done by the STS before transmission to reduce data size and protect confidentiality.
  - Recommended variables to be supplied for a given item in a given region in a given month:
    - Date (month)
    - Label describing the item (text string)
    - Region code
    - Total revenue (for a given item/region/month combination)
    - Total number of units sold (for a given item/region/month combination)
    - An average price (obtained as total revenue divided by total number of units sold)
    - COICOP category
    - Number of transaction lines in the raw data underlying the calculations of the average price (for quality control)
  - If revenue or number of units sold cannot be supplied due to confidentiality, alternative indicators of item “importance” should be considered (for example, revenue transformed into a revenue share by category, or transmitting the rank of an item in its category).
- Dimensions for calculating average prices (unit values)
  - Time dimension:
    - Manual price collection is conducted between the 1st and the 25th day of each month.
    - STS data are available with a delay of approximately 5 days.
    - A data supply at the end of the month would allow processing for publishing the CPI on the 9th day of the following month.
  - Outlet/regional dimension:
    - Regional stratification of the CPI consists of 25 regions.
    - Average prices should be calculated according to these same regions; data should refer to transactions of an entire region, not only current CPI price collection cities.
    - Possible unit value bias may arise when combining transactions from different outlet types, but outlet-level supply is likely not possible due to confidentiality.
  - Product dimension:
    - Barcodes are not included in the STS data because retailers are not required to supply them.
    - In the absence of a formal item identifier, the label (text string) must be used; text string cleaning is necessary to group identical items with slightly different labels.
- Unit value calculation
  - Average prices should be calculated as unit values: total revenue divided by total number of units sold for the transactions within the specification of the average price.
  - Unit values differ from arithmetic averages of transaction-line prices because more than one unit may be sold in a single transaction line.
- Initial quality checks by SSSU
  - Verify the size of the file, check included variables, and compile basic summary statistics for main variables upon reception.
- Recommended actions (Data collection and preparation)
  - The SSSU to agree with the STS on the data to be transmitted and formalize the data supply by updating the MOU, addressing data variables, frequency, timeliness, and STS pre-processing.
  - The SSSU to prepare for reception of the data and agree with the STS on a data transmission protocol.
  - The SSSU to start receiving data on a regular basis and confirm the data meet agreed requirements.

### Classification — findings and recommendations
- Link to CPI classification
  - Items in the scanner data must be connected to the CPI classification. The Ukrainian CPI uses COICOP broken down into 335 representative products at the lowest (7-digit COICOP) level.
  - The mission recommended linking scanner-data items to the 7-digit COICOP to calculate price series with the same level of detail as the current CPI.
- Existing SSSU tools and required processes
  - The SSSU has developed a string-searching algorithm that identifies patterns in item labels and can automatically link an item to a 5-digit or even a 7-digit COICOP category, depending on the product.
  - Pattern matching is imperfect; text mining rules must be maintained as item-label patterns change over time.
  - Review and possibly update the classification suggestions made by the string-searching algorithm; in the short term this review can be done manually, focusing on items in-scope of the pilot and on most sold items.
- Reference table and operational process
  - Establish processes and tools to support classification: store item–COICOP relations in a separate reference table that is merged with monthly scanner data files to obtain COICOP categories.
  - Benefits of a reference table:
    - Each item is classified once when it enters the reference table.
    - Ensures classification consistency across regions and across time.
    - Initial phase requires processing many items at once; subsequent monthly updates will augment the reference table with new, unclassified items, and be less resource demanding.
- Additional methods (paragraph 26)
  - Subject to the availability of another classification to which the scanner data are mapped, a correspondence table could be built between the categories of this auxiliary classification and COICOP to enable automatic assignment of a COICOP category by bridging via the auxiliary classification.
  - Machine Learning (ML) methods can be used for automatic coding of the data:
    - ML methods create classifiers that learn from already classified items.
    - Once a classifier is estimated, new incoming items can be automatically classified.
    - A fully classified training data set must be available to estimate a classifier and evaluate its performance.
  - Recommended Actions:
    - The SSSU to specify pattern matching rules in order to pre-classify the scanner data as much as possible.
    - The SSSU to create a reference table for maintaining the COICOP-item relation.
    - The SSSU to additionally review the classification proposed by the pattern matching method for the items that are in-scope of the pilot study.
    - The SSSU to explore other methods that could further improve the classification of the data.

### Data cleaning — methods and thresholds
- Purpose and levels
  - Outlier detection methods can identify and remove unusual observations that could introduce additional volatility or bias in final price indices.
  - Data cleaning should be conducted at two levels:
    - At the level of an individual transaction line (raw data).
    - At the level at which average prices are calculated (pre-aggregated data).
- Raw data cleaning
  - Remove transactions with close-to-zero prices only where these are likely errors; reduced prices linked to genuine discounts should remain because reduced prices should be covered in the CPI.
  - The Interquartile range (IQR) method can be applied on raw data to identify transactions with unusual prices:
    - The IQR method sets upper and lower bounds derived from the price distribution of similar transactions (for example transactions with the same item sold on the same day).
    - A transaction is considered to be an outlier if its price lies above or below a certain threshold defined with the help of the third and first quartiles of the price distribution.
    - The SSSU has already made some tests with the IQR method; results confirmed applicability in this context.
  - The IQR method could also be applied with respect to the number of units sold in a transaction; lines with a large number of units may reflect small wholesale transactions, bulk registrations by the retailer, or bulk purchases and may be prudent to exclude.
- Pre-aggregated data cleaning
  - Outlier detection should also be applied at the level at which average prices are calculated; an item in a given month is considered an outlier if the change in the average price compared to the previous month is above or below certain thresholds based on price relatives (not price levels).
  - Typical values for an outlier filter in scanner data applications are l=0.25 and u=4.
- Dumping filter
  - A dumping filter should be implemented to avoid potential downward bias from end-of-lifecycle very low prices.
  - A dumping filter flags an observation if both price and quantity changes are “low”.
  - Typical values for a dumping filter in scanner data applications are d_p=0.7 and d_q=0.3.
- Quality control variable
  - The number of transaction lines underlying the average price should be included for quality control.
- Recommended Actions
  - The SSSU to discuss with the STS applying the selected outlier methods during the initial transformations of the raw data.
  - The SSSU to decide on the thresholds for the outlier and dumping filters and to flag the data accordingly before the data enter the index compilations.

### Index compilation and data integration — methods, pilot design, and implementation
- Key features to address
  - The set of available items in a region changes month-to-month.
  - Methods should account for both prices and the economic importance of items.
- Fixed basket method (preferred for pilot)
  - Relies on a representative set of items selected in the base period; the initial “basket” should be selected with the help of formal sampling methods (for example, revenue shares can be used to determine significance).
  - Items are selected for inclusion via cutoff sampling; revenue shares for selecting the basket could be based on data from several months to reduce seasonality impact.
  - In the fixed basket method the same set of items is priced in each month; replacements must be selected when items become unavailable:
    - Steps in month t:
      - Access the basket of the previous month t-1.
      - For each item of the t-1 basket, check if the item is still available (or representative) in month t.
        - If YES: add the item to basket of month t.
        - If NO: select a replacement item to be added to the basket of month t.
      - Calculate a Jevons index (geometric mean of price relatives) for the month t basket.
    - Replacement treatments can include overlap pricing, quantity adjustment, direct comparison, or mean imputation.
  - The fixed basket method favors comparability over representativity and should be combined with a regular (e.g. annual) resampling schedule.
  - The mission concluded that the fixed basket method is the best option for the pilot study because it is easier to implement and less data demanding than multilateral methods; it facilitates evaluation and interpretation of indices.
- Weighted/multilateral methods
  - Scanner data enable weighted index compilation methods using all items, with weights according to importance.
  - The Törnqvist price index is preferred theoretically; it is a weighted version of the Jevons index using average revenue shares of the items in the two comparison periods.
  - Monthly chaining of Törnqvist indices leads to chain drift and is not recommended; the chain drift problem can be solved by using the Törnqvist index as building blocks of a multilateral method.
- Practical pilot implementation
  - The mission provided initial training and Excel files to illustrate methods; preliminary calculations were made on STS test data.
  - For the pilot study:
    - The sampling method for the initial fixed basket and the method for treatment of replacements must be decided.
    - The pilot study will help identify the appropriate design; experimental price indices should be compared to currently calculated series and the compilation method revisited if needed.
    - Price indices should be calculated only for a limited selection of products during the pilot phase because the CPI team continues regular CPI calculation alongside experimental work.
    - The SSSU plans to focus on some food products (COICOP 01.1) and pharmaceutical products (COICOP 06.1.1), which together make up a significant share (more than 40 percent) in the CPI basket.
  - Commensurability
    - The Jevons and Törnqvist price indices satisfy the “commensurability test” so prices need not be transformed to a common unit of measurement to calculate these indices.
    - However, average prices demand a common unit of measurement; unit information is not available in a standardized manner in scanner data and must be extracted from labels, which sometimes lack unit information.
  - Integration strategy
    - Scanner data indices should be calculated at the level of the representative product by region—the same level used for elementary price indices from manual collection—to permit consistent comparison and facilitate integration.
    - In future, a stratum could be based either fully on field price collection or fully on scanner data; the SSSU can continue to use annually updated CPI expenditure weights to combine stratum indices.
- Recommended Actions
  - The SSSU to decide on a specific sampling and substitution methodology for the fixed basket method.
  - The SSSU to calculate experimental prices indices for a selection of products using the fixed basket method.
  - The SSSU to compare the experimental price indices to the current price series and adjust the scanner data methodology if needed.
  - The SSSU to update the methodological document describing the pilot study.
- Implementation sequencing and timelines
  - Work on scanner data will span several years. In 2024 priority is to secure regular access and calculate experimental price indices, compare them to current series, and adjust methodology.
  - Depending on pilot outcomes, CPI production systems could be adjusted during 2025 for integration in the CPI by January 2026.
  - Integration is typically incremental (e.g., first integrate food products and pharmaceuticals, then extend scope).
- Production process implications
  - Processing scanner data will create new central-level tasks while reducing regional-level field price collection.
  - Main processes for using scanner data in CPI:
    - (i) data collection and data preparation;
    - (ii) classification;
    - (iii) data validation; and
    - (iv) index compilation and data integration.

### Officials met during the mission
- Ms. Nadiia Profatska — Deputy Director-Head of Division for Consumer Price Statistics, Department for Price Statistics, SSSU
- Ms. Anastasiia Simson — Senior Specialist of the Division for Consumer Price Statistics, Department for Price Statistics, SSSU
- Mr. Pavlo Polikarchuk — IT and Data expert
- Ms. Mariia Daniuk — IT and Data expert

*Source: Section I. Detailed Technical Assessment and Recommendations — IMF Technical Assistance Report – Ukraine Consumer Price Index Mission*

### Section I. Detailed Technical Assessment and Recommendations ...................................................... 4

### Section I. Detailed Technical Assessment and Recommendations

### Summary of mission outcomes and priority recommendations
- A technical assistance (TA) mission was conducted in Vienna, Austria to assist the State Statistics Service of Ukraine (SSSU) with introducing administrative scanner data for the compilation of the consumer price index (CPI).
- The SSSU plans to use administrative scanner data from the State Tax Service (STS), covering transactions recorded by cash register machines that retailers have been legally required to use.
- Incorporating scanner data will improve the quality of the CPI and reduce field price collection by improving regional, product, and temporal coverages.
- Weighted price indices can be compiled because price and revenue information are available in scanner data.
- Pre-processing by the STS is required before transmission to the SSSU. Transactions referring to the same item sold in all outlets of a region during a given month should be combined. In the absence of a formal item code, the text label must be used to identify transactions of the same item.
- An average price (unit value) is obtained by dividing total revenue by total number of units sold. Some data cleaning is required before calculating these average prices.
- The SSSU has developed string-searching algorithms to identify patterns in item labels and automatically assign items to COICOP categories.
- The SSSU should agree with the STS on data characteristics to be made available: pre-processing calculations, variables included, frequency and timeliness, and provision of back data for the first transmission. Update the memorandum of understanding (MOU) accordingly.
- The SSSU should link scanner-data items to the lowest level of the CPI hierarchy (the 335 representative products, 7-digit COICOP), review and update classifications suggested by string-searching, establish processes and tools to support classification, and implement outlier methods to detect and remove items with unusual prices.
- Experimental price indices should be calculated using the fixed basket method for a limited selection of products, at the representative product by region level, to facilitate consistent comparisons with current price series and future integration.
- Action plan priority recommendations (Table 1):
  - August 1st 2024: Arrange regular access to the administrative scanner data — Responsible: SSSU.
  - December 31st 2024: Calculate experimental price indices from the administrative scanner data — Responsible: SSSU.

### Action plan (selected items from Table 2)
- Topic: Data access and data preparation
  - High — Agree with the STS on the structure of the data to be transmitted to the SSSU and sign the additional agreement — Target Completion Date: May 15st 2024
  - High — Prepare the data flow and transmission protocol according to the agreement — Target Completion Date: June 1st 2024
  - High — Start accessing the data on a monthly basis — Target Completion Date: July 1st 2024
  - High — Confirm the quality of the received data files — Target Completion Date: August 1st 2024
- Topic: Classification
  - High — Specify pattern matching rules for classification — Target Completion Date: Completed
  - Medium — Discuss with the STS about availability of additional information on the classification of the items — Target Completion Date: September 1st 2024
  - Medium — Build a reference table for archiving the link between items and COICOP categories — Target Completion Date: September 1st 2024
  - High — Review and update pre-classified data for the products that are in-scope of the pilot study — Target Completion Date: September 1st 2024
- Topic: Data cleaning
  - Medium — Discuss with the STS on applying the selected outlier methods during the initial transformations of the raw data — Target Completion Date: May 1st 2024
  - Medium — Decide on the thresholds for the outlier and dumping filters and implement these rules on the data — Target Completion Date: October 1st 2024
- Topic: Index compilation
  - High — Draw samples for the initial basket — Target Completion Date: October 1st 2024
  - High — Decide on the principles for the treatment of replacements — Target Completion Date: October 1st 2024
  - Medium — Start preparing the IT tools for index compilation — Target Completion Date: November 1st 2024
  - High — Calculate experimental price indices for a selection of products — Target Completion Date: December 31st 2024
  - High — Compare the results to the current price series and adjust the methodology if needed — Target Completion Date: December 31st 2024
  - Medium — Update the methodological document for the pilot study — Target Completion Date: February 1st 2025

### Data collection and data preparation (detailed findings and recommendations)
- Data characteristics and scale
  - The STS data are big data; one day of transactions can amount to over 77 million lines, each line representing a transaction for a specific item.
  - Currently data are collected from over 900,000 cash register machines across the country.
- Recommended STS pre-processing and transmitted variables
  - Some initial transformations and pre-aggregation should be done by the STS before transmission to reduce data size and protect confidentiality.
  - Recommended variables to be supplied for a given item in a given region in a given month:
    - Date (month)
    - Label describing the item (text string)
    - Region code
    - Total revenue (for a given item/region/month combination)
    - Total number of units sold (for a given item/region/month combination)
    - An average price (obtained as total revenue divided by total number of units sold)
    - COICOP category
    - Number of transaction lines in the raw data underlying the calculations of the average price (for quality control)
  - If revenue or number of units sold cannot be supplied due to confidentiality, alternative indicators of item “importance” should be considered (for example, revenue transformed into a revenue share by category, or transmitting the rank of an item in its category).
- Dimensions for calculating average prices (unit values)
  - Time dimension:
    - Manual price collection is conducted between the 1st and the 25th day of each month.
    - STS data are available with a delay of approximately 5 days.
    - A data supply at the end of the month would allow processing for publishing the CPI on the 9th day of the following month.
  - Outlet/regional dimension:
    - Regional stratification of the CPI consists of 25 regions.
    - Average prices should be calculated according to these same regions; data should refer to transactions of an entire region, not only current CPI price collection cities.
    - Possible unit value bias may arise when combining transactions from different outlet types, but outlet-level supply is likely not possible due to confidentiality.
  - Product dimension:
    - Barcodes are not included in the STS data because retailers are not required to supply them.
    - In the absence of a formal item identifier, the label (text string) must be used; text string cleaning is necessary to group identical items with slightly different labels.
- Unit value calculation
  - Average prices should be calculated as unit values: total revenue divided by total number of units sold for the transactions within the specification of the average price.
  - Unit values differ from arithmetic averages of transaction-line prices because more than one unit may be sold in a single transaction line.
- Initial quality checks by SSSU
  - Verify the size of the file, check included variables, and compile basic summary statistics for main variables upon reception.
- Recommended actions (Data collection and preparation)
  - The SSSU to agree with the STS on the data to be transmitted and formalize the data supply by updating the MOU, addressing data variables, frequency, timeliness, and STS pre-processing.
  - The SSSU to prepare for reception of the data and agree with the STS on a data transmission protocol.
  - The SSSU to start receiving data on a regular basis and confirm the data meet agreed requirements.

### Classification (detailed findings and recommendations)
- Link to CPI classification
  - Items in the scanner data must be connected to the CPI classification. The Ukrainian CPI uses COICOP broken down into 335 representative products at the lowest (7-digit COICOP) level.
  - The mission recommended linking scanner-data items to the 7-digit COICOP to calculate price series with the same level of detail as the current CPI.
- Existing SSSU tools and required processes
  - The SSSU has developed a string-searching algorithm that identifies patterns in item labels and can automatically link an item to a 5-digit or even a 7-digit COICOP category, depending on the product.
  - Pattern matching is imperfect; text mining rules must be maintained as item-label patterns change over time.
  - Review and possibly update the classification suggestions made by the string-searching algorithm; in the short term this review can be done manually, focusing on items in-scope of the pilot and on most sold items.
- Reference table and operational process
  - Establish processes and tools to support classification: store item–COICOP relations in a separate reference table that is merged with monthly scanner data files to obtain COICOP categories.
  - Benefits of a reference table:
    - Each item is classified once when it enters the reference table.
    - Ensures classification consistency across regions and across time.
    - Initial phase requires processing many items at once; subsequent monthly updates will augment the reference table with new, unclassified items, and be less resource demanding.

### Data cleaning (summary of recommendations)
- Outlier detection and filtering
  - Some data cleaning is required before calculating average prices, including outlier detection and removal of items with unusual prices.
  - Recommended that the SSSU discuss with the STS applying selected outlier methods during initial transformations of the raw data.
  - Decide on thresholds for outlier and dumping filters and implement these rules on the data.
- Quality control variable
  - The number of transaction lines underlying the average price should be included for quality control.

### Index compilation and data integration (recommended approach)
- Pilot methodology and scope
  - Calculate experimental price indices using the fixed basket method for a limited selection of products (for example some food products or pharmaceuticals).
  - Calculate indices at the representative product by region level (same level as elementary price indices from manual collection) to facilitate consistent comparisons with current series and future integration.
  - The fixed basket method is closer to current practice, less data demanding, and easier to interpret than more advanced index methods.
- Implementation sequencing
  - Work on scanner data will span several years. In 2024 priority is to secure regular access and calculate experimental price indices, compare them to current series, and adjust methodology.
  - Depending on pilot outcomes, CPI production systems could be adjusted during 2025 for integration in the CPI by January 2026.
  - Integration is typically incremental (e.g., first integrate food products and pharmaceuticals, then extend scope).
- Production process implications
  - Processing scanner data will create new central-level tasks while reducing regional-level field price collection.
  - Main processes for using scanner data in CPI:
    - (i) data collection and data preparation;
    - (ii) classification;
    - (iii) data validation; and
    - (iv) index compilation and data integration.

*Source: Section I. Detailed Technical Assessment and Recommendations — IMF Technical Assistance Report – Ukraine Consumer Price Index Mission*

### 26. The SSSU should continue to explore other methods to further optimize the classification

### 26. The SSSU should continue to explore other methods to further optimize the classification

### Classification of scanner data (paragraph 26)
- Subject to the availability of another classification to which the scanner data are mapped, a correspondence table could be built between the categories of this auxiliary classification and COICOP to enable automatic assignment of a COICOP category by bridging via the auxiliary classification.
- Machine Learning (ML) methods can be used for automatic coding of the data:
  - ML methods create classifiers that learn from already classified items.
  - Once a classifier is estimated, new incoming items can be automatically classified.
  - A fully classified training data set must be available to estimate a classifier and evaluate its performance.
- Recommended Actions:
  - The SSSU to specify pattern matching rules in order to pre-classify the scanner data as much as possible.
  - The SSSU to create a reference table for maintaining the COICOP-item relation.
  - The SSSU to additionally review the classification proposed by the pattern matching method for the items that are in-scope of the pilot study.
  - The SSSU to explore other methods that could further improve the classification of the data.

*IMF | Technical Assistance Report – Ukraine Consumer Price Index Mission | 9*

### D. Data cleaning (paragraphs 27–32)
- Purpose:
  - Outlier detection methods can identify and remove unusual observations that could introduce additional volatility or bias in final price indices.
  - Data cleaning should be conducted at two levels:
    - At the level of an individual transaction line (raw data).
    - At the level at which average prices are calculated (pre-aggregated data).
- Raw data cleaning:
  - Remove transactions with close-to-zero prices only where these are likely errors; reduced prices linked to genuine discounts should remain because reduced prices should be covered in the CPI.
  - The Interquartile range (IQR) method can be applied on raw data to identify transactions with unusual prices:
    - The IQR method sets upper and lower bounds derived from the price distribution of similar transactions (for example transactions with the same item sold on the same day).
    - A transaction is considered to be an outlier if its price lies above or below a certain threshold defined with the help of the third and first quartiles of the price distribution.
    - The SSSU has already made some tests with the IQR method; results confirmed applicability in this context.
  - The IQR method could also be applied with respect to the number of units sold in a transaction; lines with a large number of units may reflect small wholesale transactions, bulk registrations by the retailer, or bulk purchases and may be prudent to exclude.
- Pre-aggregated data cleaning:
  - Outlier detection should also be applied at the level at which average prices are calculated; an item in a given month is considered an outlier if the change in the average price compared to the previous month is above or below certain thresholds based on price relatives (not price levels).
  - Typical values for an outlier filter in scanner data applications are l=0.25 and u=4.
- Dumping filter:
  - A dumping filter should be implemented to avoid potential downward bias from end-of-lifecycle very low prices.
  - A dumping filter flags an observation if both price and quantity changes are “low”.
  - Typical values for a dumping filter in scanner data applications are d_p=0.7 and d_q=0.3.
- Recommended Actions:
  - The SSSU to discuss with the STS applying the selected outlier methods during the initial transformations of the raw data.
  - The SSSU to decide on the thresholds for the outlier and dumping filters and to flag the data accordingly before the data enter the index compilations.

*IMF | Technical Assistance Report – Ukraine Consumer Price Index Mission | 10*

### E. Index compilation and data integration (paragraphs 33–46)
- Key features scanner-data methods must address:
  - The set of available items in a region changes month-to-month.
  - Methods should account for both prices and the economic importance of items.
- Fixed basket method (paragraphs 34–37):
  - Relies on a representative set of items selected in the base period; the initial “basket” should be selected with the help of formal sampling methods (for example, revenue shares can be used to determine significance).
  - Items are selected for inclusion via cutoff sampling; revenue shares for selecting the basket could be based on data from several months to reduce seasonality impact.
  - In the fixed basket method the same set of items is priced in each month; replacements must be selected when items become unavailable:
    - Steps in month t:
      - Access the basket of the previous month t-1.
      - For each item of the t-1 basket, check if the item is still available (or representative) in month t.
        - If YES: add the item to basket of month t.
        - If NO: select a replacement item to be added to the basket of month t.
      - Calculate a Jevons index (geometric mean of price relatives) for the month t basket.
    - Replacement treatments can include overlap pricing, quantity adjustment, direct comparison, or mean imputation.
  - The fixed basket method favors comparability over representativity and should be combined with a regular (e.g. annual) resampling schedule.
  - The mission concluded that the fixed basket method is the best option for the pilot study because it is easier to implement and less data demanding than multilateral methods; it facilitates evaluation and interpretation of indices.
- Weighted/multilateral methods (paragraphs 38–39):
  - Scanner data enable weighted index compilation methods using all items, with weights according to importance.
  - The Törnqvist price index is preferred theoretically; it is a weighted version of the Jevons index using average revenue shares of the items in the two comparison periods.
  - Monthly chaining of Törnqvist indices leads to chain drift and is not recommended; the chain drift problem can be solved by using the Törnqvist index as building blocks of a multilateral method.
- Practical implementation and pilot scope (paragraphs 40–46):
  - The mission provided initial training and Excel files to illustrate methods; preliminary calculations were made on STS test data.
  - For the pilot study:
    - The sampling method for the initial fixed basket and the method for treatment of replacements must be decided.
    - The pilot study will help identify the appropriate design; experimental price indices should be compared to currently calculated series and the compilation method revisited if needed.
    - Price indices should be calculated only for a limited selection of products during the pilot phase because the CPI team continues regular CPI calculation alongside experimental work.
    - The SSSU plans to focus on some food products (COICOP 01.1) and pharmaceutical products (COICOP 06.1.1), which together make up a significant share (more than 40 percent) in the CPI basket.
  - Commensurability:
    - The Jevons and Törnqvist price indices satisfy the “commensurability test” so prices need not be transformed to a common unit of measurement to calculate these indices.
    - However, average prices demand a common unit of measurement; unit information is not available in a standardized manner in scanner data and must be extracted from labels, which sometimes lack unit information.
  - Integration strategy:
    - Scanner data indices should be calculated at the level of the representative product by region—the same level used for elementary price indices from manual collection—to permit consistent comparison and facilitate integration.
    - In future, a stratum could be based either fully on field price collection or fully on scanner data; the SSSU can continue to use annually updated CPI expenditure weights to combine stratum indices.
- Recommended Actions:
  - The SSSU to decide on a specific sampling and substitution methodology for the fixed basket method.
  - The SSSU to calculate experimental prices indices for a selection of products using the fixed basket method.
  - The SSSU to compare the experimental price indices to the current price series and adjust the scanner data methodology if needed.
  - The SSSU to update the methodological document describing the pilot study.

*IMF | Technical Assistance Report – Ukraine Consumer Price Index Mission | 11–13*

### F. Officials met during the mission (paragraph F)
- Ms. Nadiia Profatska — Deputy Director-Head of Division for Consumer Price Statistics, Department for Price Statistics, SSSU
- Ms. Anastasiia Simson — Senior Specialist of the Division for Consumer Price Statistics, Department for Price Statistics, SSSU
- Mr. Pavlo Polikarchuk — IT and Data expert
- Ms. Mariia Daniuk — IT and Data expert

*IMF | Technical Assistance Report – Ukraine Consumer Price Index Mission | 13*

---


_Source: https://www.imf.org/-/media/files/publications/tar/2024/english/tarea2024058-print-pdf.pdf_