## 1geoea2021008 - Section 1

## Source details

**Canonical URL:** [1geoea2021008 - Section 1](https://www.imf.org/-/media/files/publications/cr/2021/english/1geoea2021008.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/cr/2021/english/1geoea2021008.pdf.md)
- [Structured JSON version](/-/media/files/publications/cr/2021/english/1geoea2021008.pdf.json)

---

### Summary of mission outcomes and priority recommendations
- Mission purpose: assist the National Statistics Office of Georgia (Geostat) with continuing its modernization of the Consumer Price Index (CPI) focusing on scanner data (SD). Mission delivered remotely; sessions recorded. (Mission dates: July 12–16, 2021.)
- Geostat target: publish the CPI with SD for food products and non-alcoholic as well as alcoholic beverages with data from two main supermarket chains by January 2023. Experimental indices to be tested during 2022.
- Current focus: loading, inspection, and processing of raw data files; mapping of products to the national breakdown of COICOP 1999 (national seven-digit level / elementary aggregate (EA) level).
- Data transmission status:
  - SD currently cover the first ten days of each month and are tentatively submitted to Geostat by the 15th of the month.
  - Data transmitted in Excel format via e-mail or WeTransfer.com; some months missing and frequent delays.
  - No written agreement currently exists with data providers; mission provided examples of written agreements (e.g., MoU) for SD transmission.
- Market coverage and representativity:
  - Two supermarket chains currently provide SD and have outlets in all six CPI cities.
  - The two chains cover 17 percent of retail trade turnover. A third chain would bring that number to 32 percent.
  - Despite limited coverage, SD can be introduced in CPI compilation using turnover as weight to aggregate supermarket indices and field-collected prices.
- Classification and correspondence table issues:
  - Geostat created and maintains a manual correspondence table between EANs and national COICOP EAs; it should rely on EANs and product descriptions in addition to supermarket categories and sub-categories.
  - After linking supermarket data there was an insufficient number of different products for many city–EA–month combinations.
  - Classification file contains duplicate records where the same product is attributed to different EAs.
- Skills and processing:
  - Geostat staff have some prior knowledge in R; mission provided live R training and delivered R code to load and inspect raw files.
  - Recommendation: perform all SD handling in R for reproducibility and error reduction; avoid Excel for large-scale SD processing.

Priority recommendations (selected from action plan):
- October 2021: Establish a firm written commitment (MoU) with the supermarket chains on the data deliveries at the management level. Responsible: Geostat.
- Continuous: Continue negotiations with other main supermarket chains on potential data deliveries. Responsible: Geostat.
- December 2021: Improve the quality of the manual classification and assess it vis-à-vis the raw data. Responsible: Geostat.

*Source: International Monetary Fund, Report on Scanner Data for CPI Mission, July 12–16, 2021 (prepared by Vanda Guerreiro and Jens Mehrhoff).*

### Detailed technical assessment and recommendations

#### A. Introduction — rationale and resource implications
- SD content and advantage:
  - SD files contain, for each individual item, value of sales and number of units sold in a city during a given period; allow use of "explicit," monthly updated, item-level weights.
  - Advantages of SD versus in-store collection:
    - Information on actual expenditure for all items sold.
    - Price data available over longer periods per item.
    - Better coverage of discounts; ability to exclude items not actually sold; item sample updated monthly.
- Expected benefits: increase in CPI quality; reduction in CPI production costs and respondent burden in the long run.
- Upfront investment required: human resources skills and IT infrastructure for research and implementation phase.
- Recommended action:
  - Plan for an increase of resources needed in the research and implementation phase to develop a data analysis and management system (including classification to COICOP).

#### B. SD transmission and coverage — current state and actionable items
- Timeline and scope:
  - First SD obtained in April 2020 from two large supermarket chains; negotiations began end-2019.
  - Data mainly for food and non-alcoholic and alcoholic beverages.
- File delivery and quality issues:
  - Data often received with delay; some months missing.
  - Currently transmitted in Excel; recommendation is automated weekly or daily delivery covering the full first two weeks of each month.
- Clarification questions for providers (examples):
  - Whether duplicates of EANs on a given day and in a given city are due to non-aggregation across outlets.
  - Why unit prices differ between outlets of the same supermarket chain in the same city (if aggregation confirmed).
  - Which types of discounts are included and how they are recorded (usual EAN or separate code).
  - Whether re-launches (same product with new packaging and different EAN) can be identified through stock-keeping units.
  - Whether product descriptions can be parsed into brand, packaging size, etc.
- Data stability request:
  - Require data providers to inform Geostat in advance of any supermarket product categorization changes.
  - Request that the agreed file structure remain stable with regard to variables included, their naming, and the detail of aggregation (EANs, cities, days, outlets).
- Market concentration:
  - Georgian retail market has a relatively low degree of concentration: two chains cover 17 percent of retail trade turnover; inclusion of a third chain would increase coverage to 32 percent.
- Recommended actions:
  - Establish an MoU with supermarket chains on content and format of data deliveries, including automatization.
  - Receive data on a weekly or daily basis covering the full first two weeks of each month to balance weekdays.
  - Assess future developments in market structure to understand data coverage and representativity.
  - Continue discussions with other retailers on potential data deliveries.

#### C. Data analysis and management — processing, quality checks, and gaps
- R training and code:
  - Mission provided live R coding for loading raw files and inspecting data; jointly produced R code has been tested and is available for further development.
  - R steps prepared to perform:
    - Checking the file structure
    - Checking the size of the files
    - Checking the number of items per period and city
    - Checking for duplicate records
    - Checking for missing values
    - Checking for non-positive values in quantities sold or turnovers
    - Checking the sum of turnover
- Current practice and recommendation:
  - Initial steps currently performed manually in Excel; mission recommends all data handling in R for reproducibility and to manage scale.
- Processing flow implemented during mission:
  - Filter raw SD by regions; analyze errors, missing data, and out-of-scope product categories.
  - Drop suspect data; aggregate disaggregate retailer information to chain-wide monthly totals.
  - Combine individual months into a single file for further index processing.
  - Link retailer data with EAN–EA conversion table and restrict data set to in-scope products and the six CPI cities.
- Identified analytical gap:
  - After linking with EAN–EA correspondence table, an insufficient number of different products exists for many city–EA–month combinations to calculate indexes.
  - Classification file contains duplicate records assigning same product to different EAs.
- Recommended actions:
  - Process SD in R from raw files to final outputs.
  - Participate in trainings on programming in R.
  - Analyze limited product coverage of city–EA combinations after linking with the EAN–EA correspondence table (classification and raw data).

#### D. Classification to COICOP — current practice and required formalization
- Current classification practice:
  - Georgian CPI is based on COICOP 1999 with a national breakdown to the six-digit level; some six-digit aggregates consist of different items effectively creating a seven-digit classification.
  - Geostat currently uses a seven-digit classification in practice.
- Required formalization:
  - The current seven-digit classification should be formalized so that no six-digit level EA is assigned different items.
  - If there is no further breakdown of the six-digit level, the seventh digit should be zero; otherwise the seventh digit should be numbered consecutively.
- Correspondence table guidance:
  - Correspondence table should rely on EANs and product descriptions in addition to supermarket product categories and sub-categories.
  - Consistency of manual classification should be assessed, for example by counting the number of EAs used for a given product category and checking for duplicate assignments.
- Recommended actions and milestones (priority items from action plan):
  - August 2021: Classify the SD to the national seven-digit level of COICOP. (Priority: H)
  - January 2022: Formalize the current classification as being at the seven-digit level. (Priority: M)
  - December 2021: Assess the consistency of the manual classification, particularly for new EANs. (Priority: H)
  - December 2021: Check the validity and consistency of the manual classification and assess its quality. (Priority: H)
  - January 2022: Analyze the limited product coverage of city–EA combinations after linking with the EAN–EA correspondence table, in terms of both the classification and the raw data. (Priority: H)
  - Continuous: Classify all products, even those out-of-scope, and keep separate conversion tables for each retailer. (Priority: L)

### Action-plan highlights and timelines (selected priority items)
- Outcome target: CPI with SD for food products and beverages published by January 2023.
- Immediate and near-term milestones:
  - August 2021: Classify SD to national seven-digit COICOP level. (Priority: H)
  - August 2021: Participate in trainings on programming in R. (Priority: L)
  - September 2021: Process SD in R from raw files to final outputs. (Priority: H)
  - September 2021: Receive data on a weekly or daily basis covering the full first two weeks of each month. (Priority: L)
  - October 2021: Establish an MoU with supermarket chains on content, format, and automatization of data deliveries. (Priority: H)
  - November 2021: Plan for an increase of resources needed in research and implementation phase. (Priority: H)
  - December 2021: Improve and assess manual classification quality and consistency; classify all products and maintain retailer-specific conversion tables. (Priority: H / L)
  - January 2022: Analyze limited product coverage for city–EA–month combinations; formalize seven-digit classification. (Priority: H / M)

*Source: International Monetary Fund, Report on Scanner Data for CPI Mission, July 12–16, 2021 (prepared by Vanda Guerreiro and Jens Mehrhoff).*

### 1geoea2021008 - Section 2

### Classification standards and practice
- SD should be classified to the national seven-digit level of COICOP to help with later integration of SD in the CPI and blending with traditionally collected prices.
- All EANs are classified to the national six-digit level of COICOP.
- A correspondence table between the six-digit national classification and the EAN codes was manually created and is maintained by Geostat.
- Every month new EANs that appear for the first time are again manually classified and added to the correspondence table.
- The classification process is supported by information on product categories used and supplied by the supermarket chains; however, the supermarket product categorization is not aligned to COICOP and is thus, while helpful, not reliable.
- The classification should be based on EAN codes and product descriptions.

### Data quality issues and consistency checks
- Some supermarket product categories are distributed across a wide range of different EAs.
- There are some misclassifications by the supermarket chains and some product categories are rather generic.
- The consistency of the manual classification should be assessed, for example by means of the number of EAs used for a given product category.
- In case of re-launches the same product description may have different EANs; different EANs referring to that product description should be mapped onto one EA.
- Products that are out-of-scope are not classified and dropped, e.g., non-food items.
- SD contains products that are currently not covered by the CPI because they have a very low expenditure share in the household budget survey.
- The correspondence table between EAN and EA developed by Geostat needs improvements: there were duplicated records where the same individual item as identified by the unique combination of EAN, product description, category, and sub-subcategory is attributed to multiple EAs.
- The correspondence table should rely more on EANs and product descriptions than on the supermarket product categories and sub-categories.
- Linking the classification with the retailer data leads to a good portion of the data being deleted as being out-of-scope, but the number of individual items in the linked data set falls short of the number of records in the correspondence table; the quality of the classification should be checked thoroughly.

### Mapping procedures and operational recommendations
- When one supermarket product category is classified to one EA—assuming the categories are sufficiently stable and reliable—the results from manual classification can be transferred to a set of automatic rules.
- Should more detailed information become available or be extracted from the product descriptions, such as brand, this information could be used likewise.
- More advanced techniques using machine learning algorithms are not recommended at this juncture.
- Geostat should keep a separate converter list for each retailer.
  - These lists should be consistent for the same combinations of EAN and description.
  - Information from one retailer could be used to classify the other retailer if the products are the same.
- It is proposed to classify all products, even those that are out of scope, to a dump category (e.g., 9999999). This would allow checking the functioning of the retailer-converter link; non-food items will still be dropped.

### Recommended Actions
- Formalize the current classification as being at the seven-digit level.
- Classify the SD to that national seven-digit level of COICOP.
- Check the validity and consistency of the manual classification and assess its quality.
- Assess the consistency of the manual classification, particularly for new EANs.
- Classify all products, even those out-of-scope, and keep separate conversion tables for each retailer.

### Officials met during the mission
- Gogita Todradze — Executive Director of Geostat
- Giorgi Tetrauli — Head of Price Statistics Department, Geostat
- Quji Bichia — Senior Specialist in the Consumer Price Statistics Division, Price Statistics Department, Geostat

*Source: 1geoea2021008 - Section 2*

### Section 1

### 1geoea2021008 - Section 1

### Summary of mission outcomes and priority recommendations
- Mission purpose: assist the National Statistics Office of Georgia (Geostat) with continuing its modernization of the Consumer Price Index (CPI) focusing on scanner data (SD). Mission delivered remotely; sessions recorded. (Mission dates: July 12–16, 2021.)
- Geostat target: publish the CPI with SD for food products and non-alcoholic as well as alcoholic beverages with data from two main supermarket chains by January 2023. Experimental indices to be tested during 2022.
- Current focus of mission: loading, inspection, and processing of raw data files; mapping of products to the national breakdown of COICOP 1999 (national seven-digit level / elementary aggregate (EA) level).
- Data transmission status:
  - SD currently cover the first ten days of each month and are tentatively submitted to Geostat by the 15th of the month.
  - Data transmitted in Excel format via e-mail or WeTransfer.com; some months missing and frequent delays.
  - No written agreement currently exists with data providers; mission provided examples of written agreements (e.g., MoU) for SD transmission.
- Market coverage:
  - Two supermarket chains currently provide SD and have outlets in all six CPI cities.
  - The two chains cover 17 percent of retail trade turnover. A third chain would bring that number to 32 percent.
  - Despite limited coverage, SD can be introduced in CPI compilation using turnover as weight to aggregate supermarket indices and field-collected prices.
- Classification and correspondence table:
  - Geostat created and maintains a manual correspondence table between EANs and national COICOP EAs; it should rely on EANs and product descriptions in addition to supermarket categories and sub-categories.
  - The correspondence table needs improvements: after linking supermarket data there was an insufficient number of different products for many city–EA–month combinations; classification file contains duplicate records where the same product is attributed to different EAs.
- Skills and processing:
  - Geostat staff have some prior knowledge in R; mission provided live R training and delivered R code to load and inspect raw files.
  - Recommendation: perform all SD handling in R for reproducibility and error reduction; avoid Excel for large-scale SD processing.
- Priority recommendations (from action plan):
  - October 2021: Establish a firm written commitment (MoU) with the supermarket chains on the data deliveries at the management level. Responsible: Geostat.
  - Continuous: Continue negotiations with other main supermarket chains on potential data deliveries. Responsible: Geostat.
  - December 2021: Improve the quality of the manual classification and assess it vis-à-vis the raw data. Responsible: Geostat.

*Source: International Monetary Fund, Report on Scanner Data for CPI Mission, July 12–16, 2021 (prepared by Vanda Guerreiro and Jens Mehrhoff).*

### Detailed technical assessment and recommendations

#### A. Introduction — rationale and resource implications
- SD files contain, for each individual item, value of sales and number of units sold in a city during a given period; allow use of "explicit," monthly updated, item-level weights.
- Advantages of SD versus in-store collection:
  - Information on actual expenditure for all items sold.
  - Price data available over longer periods per item.
  - Better coverage of discounts; ability to exclude items not actually sold; item sample updated monthly.
- Expected benefits: increase in CPI quality; reduction in CPI production costs and respondent burden in the long run.
- Upfront investment required: human resources skills and IT infrastructure for research and implementation phase.
- Recommended action:
  - Plan for an increase of resources needed in the research and implementation phase to develop a data analysis and management system (including classification to COICOP).

#### B. SD transmission and coverage — current state and actionable items
- Timeline and status:
  - First SD obtained in April 2020 from two large supermarket chains; negotiations began end-2019.
  - Data mainly for food and non-alcoholic and alcoholic beverages.
- File delivery and quality issues:
  - Data often received with delay; some months missing.
  - Currently transmitted in Excel; recommendation is automated weekly or daily delivery covering the full first two weeks of each month.
- Questions to clarify with data providers (examples provided by mission):
  - Whether duplicates of EANs on a given day and in a given city are due to non-aggregation across outlets.
  - Why unit prices differ between outlets of the same supermarket chain in the same city (if aggregation confirmed).
  - Which types of discounts are included and how they are recorded (usual EAN or separate code).
  - Whether re-launches (same product with new packaging and different EAN) can be identified through stock-keeping units.
  - Whether product descriptions can be parsed into brand, packaging size, etc.
- Data stability request:
  - Require data providers to inform Geostat in advance of any supermarket product categorization changes.
  - Request that the agreed file structure remain stable with regard to variables included, their naming, and the detail of aggregation (EANs, cities, days, outlets).
- Market concentration:
  - Georgian retail market has a relatively low degree of concentration: two chains cover 17 percent of retail trade turnover; inclusion of a third chain would increase coverage to 32 percent.
- Recommended actions:
  - Establish an MoU with supermarket chains on content and format of data deliveries, including automatization.
  - Receive data on a weekly or daily basis covering the full first two weeks of each month to balance weekdays.
  - Assess future developments in market structure to understand data coverage and representativity.
  - Continue discussions with other retailers on potential data deliveries.

#### C. Data analysis and management — processing, quality checks, and gaps
- R training and code:
  - Mission provided live R coding for loading raw files and inspecting data; jointly produced R code has been tested and is available for further development.
  - R steps prepared to perform:
    - Checking the file structure
    - Checking the size of the files
    - Checking the number of items per period and city
    - Checking for duplicate records
    - Checking for missing values
    - Checking for non-positive values in quantities sold or turnovers
    - Checking the sum of turnover
- Current Geostat practice: initial steps performed manually in Excel; mission recommends all data handling in R for reproducibility and to manage scale.
- Data processing flow implemented during mission:
  - Filter raw SD by regions; analyze errors, missing data, and out-of-scope product categories.
  - Drop suspect data; aggregate disaggregate retailer information to chain-wide monthly totals.
  - Combine individual months into a single file for further index processing.
  - Link retailer data with EAN–EA conversion table and restrict data set to in-scope products and the six CPI cities.
- Identified analytical gap:
  - After linking with EAN–EA correspondence table, an insufficient number of different products exists for many city–EA–month combinations to calculate indexes.
  - Classification file contains duplicate records assigning same product to different EAs.
- Recommended actions:
  - Process SD in R from raw files to final outputs.
  - Participate in trainings on programming in R.
  - Analyze limited product coverage of city–EA combinations after linking with the EAN–EA correspondence table (classification and raw data).

#### D. Classification to COICOP — current practice and required formalization
- Current classification:
  - Georgian CPI is based on COICOP 1999 with a national breakdown to the six-digit level; some six-digit aggregates consist of different items effectively creating a seven-digit classification.
  - Geostat currently uses a seven-digit classification in practice.
- Required formalization:
  - The current seven-digit classification should be formalized so that no six-digit level EA is assigned different items.
  - If there is no further breakdown of the six-digit level, the seventh digit should be zero; otherwise the seventh digit should be numbered consecutively.
- Correspondence table guidance:
  - Correspondence table should rely on EANs and product descriptions in addition to supermarket product categories and sub-categories.
  - Consistency of manual classification should be assessed, for example by counting the number of EAs used for a given product category and checking for duplicate assignments.
- Recommended actions and milestones (priority items from action plan):
  - August 2021: Classify the SD to the national seven-digit level of COICOP. (Priority: H)
  - January 2022: Formalize the current classification as being at the seven-digit level. (Priority: M)
  - December 2021: Assess the consistency of the manual classification, particularly for new EANs. (Priority: H)
  - December 2021: Check the validity and consistency of the manual classification and assess its quality. (Priority: H)
  - January 2022: Analyze the limited product coverage of city–EA combinations after linking with the EAN–EA correspondence table, in terms of both the classification and the raw data. (Priority: H)
  - Continuous: Classify all products, even those out-of-scope, and keep separate conversion tables for each retailer. (Priority: L)

### Action-plan highlights and timelines (selected priority items)
- Outcome target: CPI with SD for food products and beverages published by January 2023.
- Immediate and near-term milestones:
  - August 2021: Classify SD to national seven-digit COICOP level. (Priority: H)
  - August 2021: Participate in trainings on programming in R. (Priority: L)
  - September 2021: Process SD in R from raw files to final outputs. (Priority: H)
  - September 2021: Receive data on a weekly or daily basis covering the full first two weeks of each month. (Priority: L)
  - October 2021: Establish an MoU with supermarket chains on content, format, and automatization of data deliveries. (Priority: H)
  - November 2021: Plan for an increase of resources needed in research and implementation phase. (Priority: H)
  - December 2021: Improve and assess manual classification quality and consistency; classify all products and maintain retailer-specific conversion tables. (Priority: H / L)
  - January 2022: Analyze limited product coverage for city–EA–month combinations; formalize seven-digit classification. (Priority: H / M)

*Source: International Monetary Fund, Report on Scanner Data for CPI Mission, July 12–16, 2021 (prepared by Vanda Guerreiro and Jens Mehrhoff).*

### Section 2

### 1geoea2021008 - Section 2

### Classification standards and practice
- SD should be classified to the national seven-digit level of COICOP to help with later integration of SD in the CPI and blending with traditionally collected prices.
- All EANs are classified to the national six-digit level of COICOP.
- A correspondence table between the six-digit national classification and the EAN codes was manually created and is maintained by Geostat.
- Every month new EANs that appear for the first time are again manually classified and added to the correspondence table.
- The classification process is supported by information on product categories used and supplied by the supermarket chains; however, the supermarket product categorization is not aligned to COICOP and is thus, while helpful, not reliable.
- The classification should be based on EAN codes and product descriptions.

### Data quality issues and consistency checks
- Some supermarket product categories are distributed across a wide range of different EAs.
- There are some misclassifications by the supermarket chains and some product categories are rather generic.
- The consistency of the manual classification should be assessed, for example by means of the number of EAs used for a given product category.
- In case of re-launches the same product description may have different EANs; different EANs referring to that product description should be mapped onto one EA.
- Products that are out-of-scope are not classified and dropped, e.g., non-food items.
- SD contains products that are currently not covered by the CPI because they have a very low expenditure share in the household budget survey.
- The correspondence table between EAN and EA developed by Geostat needs improvements: there were duplicated records where the same individual item as identified by the unique combination of EAN, product description, category, and sub-subcategory is attributed to multiple EAs.
- The correspondence table should rely more on EANs and product descriptions than on the supermarket product categories and sub-categories.
- Linking the classification with the retailer data leads to a good portion of the data being deleted as being out-of-scope, but the number of individual items in the linked data set falls short of the number of records in the correspondence table; the quality of the classification should be checked thoroughly.

### Mapping procedures and operational recommendations
- When one supermarket product category is classified to one EA—assuming the categories are sufficiently stable and reliable—the results from manual classification can be transferred to a set of automatic rules.
- Should more detailed information become available or be extracted from the product descriptions, such as brand, this information could be used likewise.
- More advanced techniques using machine learning algorithms are not recommended at this juncture.
- Geostat should keep a separate converter list for each retailer.
  - These lists should be consistent for the same combinations of EAN and description.
  - Information from one retailer could be used to classify the other retailer if the products are the same.
- It is proposed to classify all products, even those that are out of scope, to a dump category (e.g., 9999999). This would allow checking the functioning of the retailer-converter link; non-food items will still be dropped.

### Recommended Actions
- Formalize the current classification as being at the seven-digit level.
- Classify the SD to that national seven-digit level of COICOP.
- Check the validity and consistency of the manual classification and assess its quality.
- Assess the consistency of the manual classification, particularly for new EANs.
- Classify all products, even those out-of-scope, and keep separate conversion tables for each retailer.

### Officials met during the mission
- Gogita Todradze — Executive Director of Geostat
- Giorgi Tetrauli — Head of Price Statistics Department, Geostat
- Quji Bichia — Senior Specialist in the Consumer Price Statistics Division, Price Statistics Department, Geostat

*Source: 1geoea2021008 - Section 2*

---


_Source: https://www.imf.org/-/media/files/publications/cr/2021/english/1geoea2021008.pdf_