## ppt-diewert-on-the-ottawa-group-after-30-years

## Source details

**Canonical URL:** [ppt-diewert-on-the-ottawa-group-after-30-years](https://www.imf.org/-/media/files/data/cpi/companion-publication/ppt-diewert-on-the-ottawa-group-after-30-years.pdf)

## Other formats

- [Markdown version](/-/media/files/data/cpi/companion-publication/ppt-diewert-on-the-ottawa-group-after-30-years.pdf.md)
- [Structured JSON version](/-/media/files/data/cpi/companion-publication/ppt-diewert-on-the-ottawa-group-after-30-years.pdf.json)

---

### Introduction — Purpose, history, and structure
- The Ottawa Group on Price Indices held its first meeting in Ottawa on October 31–November 2, 1994; the meeting described is the 18th meeting of the Group and is also taking place in Ottawa.
- Stated Group purpose (Ottawa Group): “The focus of the Group is on applied research, particularly in the area of consumer price indices. The Group examines advantages and disadvantages of various concepts, methods and procedures in the context of realistic operational environments, supported by concrete examples whenever possible. Participants are specialists and practitioners who work for, or are advisers to, statistical agencies in different countries or international organisations. The Group meets about every two years.”
- Documentation and stewardship:
  - The Ottawa Group website has posted almost all of the papers (540) and presentation slides presented at the 18 meetings, constituting an extensive source on CPI theory and practice.
  - Website maintenance moved from Statistics Canada and the Australian Bureau of Statistics to Carsten Boldsen at the UNECE in Geneva.
- Presentation structure outlined by Diewert:
  - Section 2: Review of alternative approaches to index number theory available to Statistical Offices in 1994.
  - Section 3: Review of the 1989 ILO Manual on the CPI by Ralph Turvey.
  - Section 4: Detailed discussion of the first Ottawa Group meeting.
  - Section 5: Discussion of the 2004 Consumer Price Index Manual and noted problems.
  - Section 6: Discussion of the CPI Manual that came out in 2020.
  - Section 7: Prospects for future developments.
  - Appendix A: Consumer demand approach to chain drift and quality adjustment.

### Index number theory circa 1994 — main approaches and elementary indices
- Four bilateral index number theory approaches available in 1994 when prices and quantities on the same products were available for two periods:
  - Basket approaches;
  - Stochastic approaches;
  - Test or Axiomatic approaches;
  - Economic approaches.
- Comparative formula results (1994):
  - Fisher index P_F(p_0,p_t,q_0,q_t) emerges as a “best” choice under basket, test and economic approaches.
  - Törnqvist-Theil index P_T(p_0,p_t,q_0,q_t) emerges as a “best” choice under stochastic and economic approaches.
  - Diewert (1978) showed Fisher and Törnqvist-Theil approximate each other to the second order around p_0 = p_t and q_0 = q_t.
- When only price information is available (elementary indexes):
  - Usable approaches: stochastic and test approaches (basket and economic require quantities).
  - Three elementary formulae used by NSOs at first aggregation stage when only price data available: Carli, Dutot, Jevons.
  - Test approach evaluation:
    - Jevons indicated as best by tests; Carli fails the Time Reversal Test and exhibits an upward bias; Dutot fails Fisher’s (1922) Commensurability Test (not invariant to units).
    - Dalén (1992) endorsed Jevons for satisfying more tests.
    - Jevons satisfies the Circularity Test: P(p1,p2)P(p2,p3) = P(p1,p3) and the Multiperiod Identity Test: P(p1,p2)P(p2,p3)P(p3,p1) = 1.
  - Empirical motivation: Bohdan Szulc (Schultz) found chained Carli could be very upward biased when prices “bounce” (Szulc (1983; 552-554)); Statistics Canada moved from Carli to Dutot and Jevons.

### Limitation of the 1994 theory
- Major problem: 1994 theory directly or implicitly assumes underlying prices and quantities p_tn and q_tn are positive so price ratios p_tn/p_1t always exist; this underlies matched-price comparisons and fails when products are missing in some periods.

---

### The 1989 Turvey ILO Consumer Price Indices Manual — Appendix 1 (Resolution) and main themes
- Resolution from the Fourteenth International Conference of Labour Statisticians (ILO, October 28 to November 6, 1987) titled “Resolution concerning consumer price indices.”
- Purpose and structure (quoted): CPI measures changes over time in the general level of prices of goods and services a reference population acquires, uses or pays for; CPI estimated as weighted averages of elementary aggregate indices using samples of prices from specified outlets or sources.
- Data source expectation: product prices expected to come from retail outlets rather than from actual household expenditures.
- Aggregation guidance: preference for geometric means at the elementary level though precise functional form not prescribed.
- Weights guidance (Turvey (1989; 126)):
  - Weights should be annual weights representative of household purchases and updated as resources permit.
  - Weights derivation: household expenditure survey preferred, representative of household size, income, region, socio-economic group.
  - Upper-level weights: “The weights should be examined periodically... In any case, they should be revised at least once every ten years.” Turvey (1989; 126).
- Practical problems flagged by the Resolution and Turvey:
  - Likely lack of elementary-level weights.
  - Monthly price collection via retail surveys may be inconsistent with weights from household surveys.
  - Strongly seasonal products conflict with use of annual weights.
  - Disappearing products replacement by similar product price is problematic.
  - Scanner data was nascent at the time, limiting options.

### Szulc (Appendix 7, 1987) — arithmetic/geometric/harmonic constructs and implications
- Szulc represents a Lowe index using annual expenditure weights for a past reference year and monthly elementary price indexes for N categories.
- Price level definitions (period t, t = 0,1) and corresponding indexes:
  - Arithmetic price level: P_A_t ≡ Σ_{n=1}^N (p_tn / N).
  - Geometric price level: P_G_t ≡ (Π_{n=1}^N p_tn)^{1/N}.
  - Harmonic price level: P_H_t ≡ [Σ_{n=1}^N (1/N)(p_tn)^{-1}]^{-1}.
  - Period-to-period indexes: P_A(p0,p1) ≡ P_A_1 / P_A_0; P_G(p0,p1) ≡ P_G_1 / P_G_0; P_H(p0,p1) ≡ P_H_1 / P_H_0.
  - Mean-of-ratios alternatives:
    - P_A^*(p0,p1) ≡ Σ_{n=1}^N (1/N)(p_1n / p_0n) (Carli P_C).
    - P_G^*(p0,p1) ≡ (Π_{n=1}^N (p_1n / p_0n))^{1/N} (Jevons P_J).
    - P_H^*(p0,p1) ≡ [Σ_{n=1}^N (1/N)(p_1n / p_0n)^{-1}]^{-1}.
  - Identities and inequalities:
    - P_G(p0,p1) = P_G^*(p0,p1) = Jevons P_J(p0,p1).
    - P_A(p0,p1) = Dutot P_D(p0,p1).
    - P_A^*(p0,p1) = Carli P_C(p0,p1).
    - Inequality: P_A^*(p0,p1) ≥ P_G^*(p0,p1) ≥ P_H^*(p0,p1) (Hardy, Littlewood and Polya (1934; 26)).
- Szulc observations:
  - Dutot may overweight higher-priced heterogeneous products; Carli likely gives higher measures and chained Carli can be spectacularly upward biased; Dutot and Jevons are transitive and satisfy circularity.
  - Canada historically used chained Dutot for tradition/understandability but reasons were weak.

### Turvey main text — scope, conceptual choices, and measurement challenges
- Audience/purpose (Turvey (1989; 1)): manual aimed at practising statisticians, users of CPIs, students, and governments needing to know resources required for reliable indexes.
- Emphasis on practice over academic theory: “no compiler... can hope to obtain new weights more than once a year... whereas much of the literature deals with other types of index.” Turvey (1989; 1).
- Transition driven by scanner data emergence in the 1980s; statisticians gradually realized scanner data’s value.
- Population perspective choice:
  - Question: Sales in a region versus purchases by its residents? This affects index purpose and sampling (Turvey (1989; 10)).
- Imputation policy: whether to include imputed own-account production or income in kind depends on primary CPI use; suggests separate indexes for domestic central banking purposes (less imputations) and for consumption measurement (with imputations) (Turvey (1989; 12)).
- Measurement challenges and three consumption concepts (Turvey (1989; 15–16)):
  - Acquisition: value of goods/services delivered during a period.
  - Use: value of goods/services actually consumed during a period.
  - Payment: total payments made during a period.
- Owner Occupied Housing (OOH): seven alternative treatments listed by Turvey (1989; 17–19):
  - (A) Net acquisitions;
  - (B1) User cost (1): mortgage interest + depreciation;
  - (B2) User cost (2): opportunity cost of capital + depreciation − capital gains;
  - (B3) User cost (3): estimated rental value;
  - (C1) Payment (1): cash outlays (down payments, mortgage interest and repayments);
  - (C2) Payment (2): cash outlays on mortgage interest and repayments;
  - (C3) Payment (3): cash outlays on mortgage interest, excluding repayments.
- Policy implications for OOH:
  - Choice depends on CPI purpose (central banks versus national accounts versus government indexing).
  - Acquisitions approach used by HICP; HICP excludes OOH due to difficulties.
  - Turvey recommended treating consumer durables similarly to OOH where appropriate (Turvey (1989; 25)).
- Index timing, hedonic regression, and geometric means:
  - For deflation and most economic analysis CPI should relate to the period of the money flow though prices often collected at a point.
  - Hedonic regressions important for property price indexes (Turvey (1989; 80–82)).
  - Geometric mean endorsement: “many statisticians regard the use of geometric means as the best solution... Statisticians should have the courage of their convictions.” Turvey (1989; 90–92).

---

### The First Ottawa Group Meeting (Ottawa, October 31–November 2, 1994)
- Meeting purpose (Jacob Ryten, Statistics Canada): bring together specialists to exchange ideas on measuring price change and propose concrete solutions.
- Two focus issues agreed:
  - Micro-level aggregation and its macro effects.
  - Detection and estimation of CPI bias.
- Empirical and methodological debates:
  - Schultz (Szulc) demonstrated chained Carli at elementary level can produce large upward bias when prices are volatile; recommended Jevons at elementary level from Ontario micro data (Dec 1988–Jan 1994).
  - U.S. BLS research (Reinsdorf, Moulton et al.) found official indexes rose faster than average price trends due to elementary formula bias (Carli) rather than outlet substitution; led to adoption of Jevons at elementary level.
  - Sellwood (1994) on HICP: HICP’s main use should serve central banks; imputed OOH and household production should not appear in HICP; debated Carli vs. Dutot: Dutot satisfies circularity and time reversal; Carli fails both.
- Scanner data, unit values, and aggregation:
  - Alain Saglio (1994) using two years of Nielsen scanner data on chocolate bars in France:
    - Average price index decreased 1.6% per year; Laspeyres decreased 0.2% per year — difference attributed to substitution across outlets and brands.
  - Scanner data enabled computation of elementary indexes with price and quantity, using unit values or superlative indexes.
  - Diewert (1995a) advocated unit values representing prices in elementary indexes; unit values decompose into outlet, brand, package effects.
  - Ivancic and Fox (Australian scanner data): weighted time-product-dummy hedonic regressions supported aggregating prices across stores within same supermarket chain in many cases.
- Data access dilemma for NSOs (Diewert):
  - Option (i): buy scanner data from private firms → loss of NSO control.
  - Option (ii): set up an NSO information-processing subsidiary → risk of unfair competition.
  - Diewert (1995a; 24) concluded public discussion needed; NSOs will join electronic data sources in some form.
- Summary empirical additive annual bias estimates for a typical official CPI (Diewert (1995a; 35)):
  - commodity substitution bias: .2% per year
  - outlet substitution bias: .25% per year
  - linking bias: perhaps .1% per year
  - new goods bias: at least .25% per year
  - summed upward bias: at least .8% per year
  - Note: biased elementary formulas would add further upward bias.
- Follow-up agenda proposed by Jacob Ryten (ten topics), leading to recognition of need for a revised CPI Manual and the 2004 Manual (editor Peter Hill).

---

### The First Ottawa Group CPI Manual and subsequent developments (2004 and 2020 Manuals)
- Seasonal products:
  - 2004 Manual Chapters 22 and 23 provided detailed treatment; Chapter 22 recommended Rolling Year Mudgett-Stone indexes and year over year monthly indexes (month-specific weights) as checks.
  - 2004 Manual conclusion (pessimistic): “It is evident that more research needs to be carried out on the problems associated with the index number treatment of seasonal commodities. There is, as yet, no consensus on what is best practice in this area.” ILO/IMF/OECD/UNECE/Eurostat/The World Bank (2004; 417).
- Durable goods and OOH (Chapter 23, 2004 Manual):
  - User cost should be decomposed into land and structure components.
  - Acquisitions approach gives much smaller weight to OOH than user cost or rental equivalence.
  - Both Turvey and 2004 Manuals concluded multiple approaches to OOH are needed for different CPI purposes; NSOs have not routinely published alternative OOH indexes.
- Chaining, dissimilarity indexes, and chain drift:
  - 2004 Manual recommended chaining when adjacent periods are more similar (ILO/IMF/OECD/UNECE/Eurostat/The World Bank (2004; 281)); referenced Diewert (2002) for dissimilarity measures.
  - Ivancic and Fox (2011b) found dissimilarity indexes insufficient to resolve chaining decisions: empirical summary (Ivancic and Fox (2011b; 5)):
    - chaining appropriate in 47 out of 76 cases;
    - no clear evidence in 20 cases;
    - chaining not satisfactory in 9 out of 76 cases.
  - Chaining Carli, Laspeyres or Paasche often leads to massive chain drift when prices “bounce”; chaining superlative indexes can also produce drift.
- De Haan scanner-data evidence on chain drift (Jan de Haan, 2008):
  - Data: more than 100 supermarkets, weeks 1–191 (2005 week 1 to 2008 week 35) for hundreds of detergents.
  - Average number of matched products per consecutive two-week period: 53 (range 43–63).
  - Example: product “XXX Tablets” normal price ≈ 6.5 Euros; on sale for 12 of 191 weeks at ~half price; weekly sales rose from virtually 0 to over 3000 for 7 sale weeks.
  - Documented chain drift by week 191:
    - Chained Fisher index ended at 4.95% of week 1 value.
    - Chained Törnqvist index ended at 7.43% of week 1 value.
    - Chained Jevons index (bilateral matched products at each link) ended at 76.65% of week 1 index level.
  - Chain drift typically downward due to purchaser stockpiling on sale weeks, though upward drift can occur.
- Responses: multilateral indexes and Rolling Window GEKS
  - De Haan proposed increasing period length or fixed base indexes; fixed base unreliable with rapid product turnover (de Haan problem).
  - Ivancic, Fox and Diewert (2009) proposed multilateral indexes over a window (circularity satisfied) — GEKS and Weighted Time Product Dummy (WTPD).
  - Limitation: products present in only one period of the window have no effect on multilateral indexes (TPD, WTPD, GK).
  - Rolling Window GEKS (Ivancic, Fox and Diewert (2009, 2011)):
    - Example window length: 13 months to include seasonal products.
    - Compute GEKS for months 1–13; when month 14 arrives compute GEKS for months 2–14 and use ratio month14/month13 to update permanent index; repeat.
    - RWGEKS showed little difference versus GEKS over short spans, but Fox, Levell and O’Connell (2024) indicate RWGEKS can still exhibit chain drift over longer periods.
  - Linking strategies and research:
    - Ivancic, Diewert and Fox (2011): link last two periods in new window to last index value from previous window.
    - Krsinich (2016): window splice linking to second period in previous window.
    - De Haan (2015; 27) suggested “half splice” linking in middle of old window.
    - Ivancic, Diewert and Fox (2011; 33) proposed geometric average of all links for last period.
    - Diewert and Fox (2021) recommended average or mean linking as statistically safest.
    - Diewert and Shimizu (2024) proposed ever-expanding window (pursued in Appendix A).
- 2020 Manual (Consumer Price Index Manual: Concepts and Methods; editor Brian Graf; lead institution IMF):
  - Purpose: recommended methods for CPI compilation; companion Theory Manual (not finalized) explains underlying economic/statistical theory.
  - Additions: new chapters including Chapter 10 on Scanner Data (Jan de Haan), covering multilateral indexes.
  - Structure: 14 chapters and 7 Appendices; new Appendices include HICP, COICOP transitions, Spatial Comparisons, Basic Index Number Formulas, CPI Research Agenda.
- Research agenda items in 2020 Manual (Appendix 7) for future Ottawa Group focus include:
  - scanner data, web scraped prices, price updating of expenditure weights, administrative data, plutocratic vs democratic weighting, credit card data for household-specific indexes, calculating elementary indexes using expenditure weights, quality adjustment, seasonal products, target price indices, choice of higher-level formula, long-term vs short-term links, retrospective superlative calculations, CPIs for different groups/geographies, hard-to-measure services, insurance and financial services, OOH, digitalization, well being and sustainability.
- 2020 Manual observation on welfare measurement: coverage of free goods/services (education, health, parks) relates to cost of goods versus cost of living indices and conditional versus unconditional cost of living indices; suggests inviting experts across official statistics to discuss wellbeing measurement. IMF/ILO/UNECE/Eurostat/OECD/The World Bank (2020; 458).

---

### Looking Ahead — Diewert’s predictions, hedonic progress, household time allocation, and demand-based multiperiod indexes
- Predictions on CPI production and seasonality:
  - Diewert expected NSOs to produce families of indexes, at least:
    - (i) a Nonrevisable CPI and
    - (ii) a Revisable CPI.
  - Expected separate CPI indexes for seasonal strata:
    - one index focusing on year-over-year price change in the same month (seasonal baskets),
    - another measuring month-to-month price change.
  - Observation: routine production of multiple CPI families and separate seasonal-strata indexes “has not come about” or “has also not materialized.”
- Hedonic regression models:
  - Diewert (2001; 111) anticipated many unresolved hedonic regression problems would be solved; progress appears to have occurred (Triplett (2004); Diewert and Shimizu (2024) section 2).
- Household time allocation and Becker extensions:
  - Once household time allocation information becomes available, integrating Becker’s (1965) allocation of time and extensions to household nonmarket production and medical economics could improve CPI and welfare measurement.
  - Some progress: Schreyer and Diewert (2014); Diewert, Fox and Schreyer (2017).
  - Practical constraint: household time surveys progress is slow; measuring household time allocation conceptually difficult.
- Appendix — Consumer demand approach to multiperiod indexes (key ideas):
  - A simplified version of Diewert and Feenstra (2017) (2022) approach to estimating reservation prices is "not more complicated than calculating GK price indexes or running a Time Product Dummy regression."
  - Approach estimates the consumer’s utility function directly rather than the dual unit cost function; direct estimation workable when products are missing in some periods because quantities of missing products are observed as 0 while reservation prices are unobserved.
  - Two-period illustrative example (new-product bias):
    - Period 1: only product 1 available; budget e1 > 0; price p11 > 0; quantity q11 = e1 / p11.
    - Period 2: new product 2 appears; assume e2 = e1 and p21 = p11; utility-maximizing choice (q21,q22) generally differs from (q11,0) unless preferences are linear.
    - If preferences are linear: no new-product bias (q1* = q11).
    - Typical nonlinearity implies linear models produce a quantity index that is too low and a corresponding price index with upward new-product bias in the period the new product first appears.
    - Fundamental implication: all linear utility models (including GK indexes, bilateral superlative matched-price indexes and hedonic time-dummy models) are susceptible to new-product bias in the initial period.
    - Correcting bias requires observing later periods (period 3) when both products are available and relative prices change to identify curvature of the indifference curve.
  - Practical note: implementing the demand-based approach to estimate reservation prices is feasible and may address chain drift and quality adjustment issues.

*Content attributed to the Ottawa Group; Diewert; Turvey (1989); Szulc (1987); Schultz; Armknecht, Moulton and Stewart; Reinsdorf; Moulton; Sellwood; Dalén; Saglio; Ivancic and Fox; de Haan; Ivancic, Fox and Diewert; Fox, Levell and O’Connell; Krsinich; Diewert and Shimizu; Diewert and Feenstra; and the 2004 and 2020 CPI Manuals and their institutional editors and contributors.*

### Introduction

### ppt-diewert-on-the-ottawa-group-after-30-years - Introduction

### Purpose and history of the Ottawa Group
- The Ottawa Group on Price Indices (also known as the United Nations International Working Group on Price Indices) held its first meeting in Ottawa on October 31- November 2, 1994.
- The present meeting described is the 18th meeting of the Group and is also taking place in Ottawa.
- The Group’s purpose (as stated by the Ottawa Group website) is: “The focus of the Group is on applied research, particularly in the area of consumer price indices. The Group examines advantages and disadvantages of various concepts, methods and procedures in the context of realistic operational environments, supported by concrete examples whenever possible. Participants are specialists and practitioners who work for, or are advisers to, statistical agencies in different countries or international organisations. The Group meets about every two years.”

### Documentation and stewardship
- The Ottawa Group website has posted almost all of the papers (540) and presentation slides that were presented at the 18 meetings.
- These materials constitute an extremely valuable source on the theory and practice of Consumer Price Index construction.
- Over the years, Statistics Canada and the Australian Bureau of Statistics maintained the website.
- Maintenance of the website has been turned over to Carsten Boldsen at the UNECE in Geneva.

### Origins and founding participants
- The Ottawa Group was initiated after discussions among Paul Armknecht (Bureau of Labor Statistics), Bert Balk (Statistics Netherlands) and Bohdan Schultz (Statistics Canada) following a meeting of the Joint UNECE/ILO group on Consumer Price Indices in Geneva.
- Bohdan Schultz, then Head of Consumer Price Index Research at Statistics Canada, proposed that Statistics Canada convene a meeting in Ottawa to discuss CPI problems; Jacob Ryten approved convening the first meeting in Ottawa in 1994.
- The author reports having attended all Ottawa Group meetings except the meeting in Washington.

### Structure of the presentation (overview of sections)
- Section 2: Review of alternative approaches to index number theory available to Statistical Offices in 1994.
- Section 3: Review of the 1989 ILO Manual on the CPI, largely written by Ralph Turvey.
- Section 4: Detailed discussion of the first meeting of the Ottawa Group.
- Section 5: Discussion of the 2004 Consumer Price Index Manual and noted problems with advice in that document.
- Section 6: Discussion of the CPI Manual that came out in 2020.
- Section 7: Prospects for future developments.
- Appendix A: Explanation of a consumer demand approach to constructing practical price indexes that can deal with the chain drift problem and quality adjustment.

### Index number theory circa 1994 — main approaches
- Four main approaches to bilateral index number theory available in 1994 when price and quantity data on the same products were available for two periods:
  - Basket approaches;
  - Stochastic approaches;
  - Test or Axiomatic approaches;
  - Economic approaches.
- Summary of comparative results:
  - The Fisher index P_F(p_0,p_t,q_0,q_t) emerges as a “best” choice of formula using the basket, test and economic approaches.
  - The Törnqvist-Theil index, P_T(p_0,p_t,q_0,q_t) emerges as a “best” choice using the stochastic and economic approaches.
  - Diewert (1978) showed that Fisher and Törnqvist-Theil approximate each other to the second order around a point where p_0 = p_t and q_0 = q_t.

### Index number theory when only price information is available (elementary indexes)
- When only price information is available, usable approaches are the stochastic approach and the test approach; the basket and economic approaches require quantity information.
- The three elementary index number formulae used by NSOs at the first stage of aggregation when only price information was available: Carli, Dutot and Jevons.
- Test approach evaluation of elementary indexes:
  - The stochastic approach alone cannot identify the “best” formula, but the test approach indicates Jevons is best.
  - The Carli index does not satisfy the Time Reversal Test and exhibits an upward bias.
  - The Dutot index does not satisfy Fisher’s (1922) Commensurability Test (not invariant to changes in units of measurement).
  - Dalén (1992) endorsed Jevons because it satisfied more tests than Carli and Dutot.
  - Jevons satisfies the Circularity Test (10): P(p1,p2)P(p2,p3) = P(p1,p3) and the weaker Multiperiod Identity Test (11): P(p1,p2)P(p2,p3)P(p3,p1) = 1.
- Empirical motivation:
  - Bohdan Szulc (Schultz) found that upward bias from chained Carli indexes could be very high in situations where prices “bounce” (Szulc (1983; 552-554)).
  - As a result, Statistics Canada was among the first Statistical Agencies to switch from Carli to Dutot and Jevons for elementary indexes.

### Limitation of the 1994 theory
- A major problem with the 1994 theory is that it directly or implicitly assumes underlying prices and quantities, p_tn and q_tn, are positive.
- This assumption implies price ratios p_tn/p_1t always exist as finite numbers, enabling theories applied to comparisons between any two periods to be based on matched prices.

### The 1989 Turvey ILO Consumer Price Indices Manual
- In 1989, the International Labour Organization (ILO) published Consumer Price Indices: An ILO Manual written primarily by Ralph Turvey (1989), Director of the Labour Information and Statistics Department at the ILO in Geneva.
- The Manual includes 8 Appendices; Appendices 2 to 8 are articles reprinted from the ILO Bulletin of Labour Statistics.
- The presentation reviews Appendices 1 and 7 before reviewing the main Manual.

*Presented material from the Introduction of "ppt-diewert-on-the-ottawa-group-after-30-years" (PDF chapter/section).*

### 3. The 1989 Turvey Consumer Price Indices Manual: Appendix 1

### 3. The 1989 Turvey Consumer Price Indices Manual: Appendix 1

### Resolution (Appendix 1) — purpose, structure, and data sources
- The Resolution is titled “Resolution concerning consumer price indices” and was made by the Fourteenth International Conference of Labour Statisticians, which met at the ILO Headquarters in Geneva, October 28 to November 6, 1987.
- Purpose and structure (quoted): “The purpose of a consumer price index is to measure changes over time in the general level of prices of goods and services that a reference population acquire, use or pay for consumption. A consumer price index is estimated as a series of summary measures of the period-to-period proportional change in the prices ofa fixed set of consumer goods and servicesof constant quantity and characteristics, acquired, used or paid for by the reference population. Each summary measure is constructed as aweighted average of a large number of elementary aggregate indices. Each of the elementary aggregate indices is estimatedusing a sample of pricesfor a defined set of goods and services obtained in, or by residents of, a specific region from a given set of outlets or other sources of consumption goods and services.”
- Data source expectation: product prices are expected to come from retail outlets rather than from actual household expenditures.
- Aggregation guidance: the precise functional form at the elementary level is not spelled out, but there is a preference for the use of geometric means.

### Weights: sources, frequency, and recommended practice
- Weights derivation (quoted): “In deriving the weights of the elementary aggregates, a household expenditure survey is usually the main source of data. As far as resources permit, such surveys should be representative of household size, income level, regional location, socio-economic group and any other factors which may have a bearing on household expenditure patterns. The period of the survey should be a normal one (or temporary abnormalities should be adjusted in determining the weighting pattern) and should preferably cover a whole year if seasonal variations in expenditure patterns are important.” Turvey (1989; 126).
- Key points on weights:
  - Weights should be annual weights.
  - Annual weights should be representative for monthly (or quarterly) purchases of consumer goods and services made by households.
  - Weights at the elementary level may be unavailable; if available they should be used.
- Frequency for updating upper-level weights (quoted): “The weights should be examined periodically, and particularly if economic circumstances have changed significantly, to ascertain whether they still reflect current expenditure or consumption patterns. The weights should be revised or adjusted if the review shows that this is not the case. In any case, they should be revised at least once every ten years.” Turvey (1989; 126).

### Practical problems and limitations identified by the Resolution and Turvey
- Potential problems emerging from ILO methodology:
  - A probable lack of weighting at the elementary level.
  - Monthly prices will mainly be collected by retail outlet surveys but expenditures and expenditure weights will mainly be collected by different household surveys and the resulting surveys may be inconsistent.
  - The existence of strongly seasonal products (products that are only available at certain seasons of the year) is not consistent with the use of annual weights.
- Disappearing products: the Resolution proposed replacing a disappearing product price with the price of a similar product; Turvey notes this is somewhat problematic.
- Historical context: scanner data was just beginning to emerge at the time of the Turvey Manual, limiting statisticians’ options.

### Szulc (Appendix 7) — elementary (micro) indices and arithmetic/geometric/harmonic constructs
- Appendix 7 title: “Price indices below the basic aggregation level.” Szulc’s Appendix was published in 1987.
- Szulc’s methodology:
  - Represents a Lowe index using annual expenditure weights for a past reference year “a” and monthly elementary price indexes for N categories of goods and services.
  - Explains how past year annual expenditure shares can be price updated to make the CPI into a Lowe index.
- Price level definitions (period t, t = 0,1):
  - Arithmetic price level: P_A_t ≡ Σ_{n=1}^N (p_tn / N).
  - Geometric price level: P_G_t ≡ (Π_{n=1}^N p_tn)^{1/N}.
  - Harmonic price level: P_H_t ≡ [Σ_{n=1}^N (1/N)(p_tn)^{-1}]^{-1}.
- Corresponding period-to-period indexes: P_A(p0,p1) ≡ P_A_1 / P_A_0; P_G(p0,p1) ≡ P_G_1 / P_G_0; P_H(p0,p1) ≡ P_H_1 / P_H_0.
- Alternative direct mean-of-ratios definitions:
  - P_A^*(p0,p1) ≡ Σ_{n=1}^N (1/N)(p_1n / p_0n).
  - P_G^*(p0,p1) ≡ (Π_{n=1}^N (p_1n / p_0n))^{1/N}.
  - P_H^*(p0,p1) ≡ [Σ_{n=1}^N (1/N)(p_1n / p_0n)^{-1}]^{-1}.
- Identities and inequalities:
  - P_G(p0,p1) = P_G^*(p0,p1) = Jevons index P_J(p0,p1).
  - P_A(p0,p1) = Dutot index P_D(p0,p1).
  - P_A^*(p0,p1) = Carli index P_C(p0,p1).
  - Inequality: P_A^*(p0,p1) ≥ P_G^*(p0,p1) ≥ P_H^*(p0,p1) (cited from Hardy, Littlewood and Polya (1934; 26)).
- Szulc’s observations on elementary indexes:
  - Dutot index may overweight higher-priced products and is problematic with heterogeneous products.
  - Carli index likely gives a higher measure of price change; chained Carli can be spectacularly upward biased if prices “bounced”.
  - Dutot and Jevons indexes are transitive (satisfy the circularity test).
  - Canada mostly used a chained Dutot index for tradition and understandability, though reasons are not strong and other factors might be considered in future.

### Turvey main text — scope, objectives, and implications for practice
- Turvey’s stated audience and purpose (quoted): “This manual is aimed at practising statisticians who have to construct or revise a consumer price index (CPI). It reflects the international resolution on the subject, reprinted in Appendix 1, but goes beyond it in discussing matters of detail which could not be covered in such a brief document. It is also designed to help the users of consumer price indices, including students of economics, who need to learn about the problems and limitations of these indices. Finally, it is addressed to governments which need to know about the resources required by their statisticians to produce a reliable index.” Turvey (1989; 1).
- Turvey’s stance on academic index number theory: the manual focuses on practice; much academic literature was considered irrelevant in 1989 because official statisticians could rarely obtain new weights more than once a year and data used to compute new weights refer to the past. Quote: “The manual deals with the practice of consumer price index numbers and does not attempt to survey the academic literature on the subject... no compiler of a consumer price index, whether it be monthly or quarterly, can hope to obtain new weights more than once a year at the most, and the data used to compute new weights always refer to the past rather than to the present, whereas much of the literature deals with other types of index.” Turvey (1989; 1).
- Transition due to data availability:
  - In the 1980s retail outlets began recording all product sales electronically (scanner data), changing the data environment.
  - It took time for statisticians and private companies to realize the value of scanner data for CPI construction.

### Conceptual choices: population perspective, imputations, and the index purpose
- National (permanent resident) versus domestic perspective (quoted): “Sales in a region or purchases by its residents? ... In either case, there will be a difference between observing the prices paid within the region and the prices paid by residents of the region. This raises questions concerning both the purposes of the index, the subject of this chapter, and about the sampling aspects of index construction, discussed in a later chapter.” Turvey (1989; 10).
- Imputed values and index purpose (quoted): “No one doubts that for a number of purposes the addition of the imputed value of any own-account production and of any income in kind should be added to a measure of money income or consumption to obtain a measure of total income or consumption. ... But when the sales to be deflated or the incomes to be deflated, evaluated or determined include no value imputations, then the price index should not include them either. Hence, whether or not to include imputed items should depend onwhat is the most important purpose for which the consumer price index is to be used.” Turvey (1989; 12).
- Policy implication: the paragraph supports having separate indexes—one largely free of imputations for central banking/domestic inflation monitoring and another including imputations to better measure actual household consumption.

### Measurement challenges: insurance, matching payments, and consumption concepts
- Difficult-to-measure categories discussed: insurance, financial services, property insurance (gross premiums versus net premiums), and matching payments for use.
- Three consumption concepts (quoted):
  - Acquisition: “the total value of all goods and services delivered during a given period, irrespective of whether they were wholly paid for or not during the period, should be taken into account.”
  - Use: “the total value of all goods and services actually consumed during a given period should be taken into account.”
  - Payment: “the total payments made for goods and services during a given period, without regard to whether they were delivered or not, should be taken into account.” Turvey (1989; 15-16).

### Owner Occupied Housing (OOH) — seven approaches and data implications
- Turvey defines three broad approaches and presents seven alternative treatments (quoted with labels and questions):
  - (A) Net acquisitions: change through time in the total purchase value of a sample of new owner-occupied dwellings similar to new acquisitions in the reference period?
  - (B1) User cost (1): change in mortgage interest and conventional depreciation at replacement cost for a sample of owner-occupied dwellings?
  - (B2) User cost (2): change in opportunity cost of invested capital value, plus depreciation, less accruing capital gains?
  - (B3) User cost (3): change in estimated rental value of a sample of owner-occupied dwellings?
  - (C1) Payment (1): change in cash outlays on down payments, mortgage interest and repayments?
  - (C2) Payment (2): change in cash outlays on mortgage interest and repayments?
  - (C3) Payment (3): change in cash outlays on mortgage interest, excluding repayment elements?
  - (Quoted) Turvey (1989; 17-19).
- Choice of OOH treatment depends on index purpose:
  - Acquisitions approach may be preferred by central banks wanting limited imputations.
  - National accounts may prefer a user cost approach.
  - Governments may prefer a payments approach for indexing pensions and transfers.
- Practical and conceptual issues:
  - Acquisitions approach requires focusing on sales with new structures; sales of used houses between households cancel out and should not affect CPI.
  - Imputation is needed to decompose purchase price into land and structure components because depreciation applies to structure but not to land.
- HICP practice: the European Economic Union’s Harmonized Index of Consumer Prices (HICP) uses the acquisitions approach and has struggled with OOH; solution thus far has been to exclude OOH from the HICP.
- Turvey’s recommendation: treat consumer durables similarly to OOH where appropriate: “If these arguments are not accepted, the case for treating consumer durables and owner-occupied dwellings, as far as practicable, in the same way is a strong one.” Turvey (1989; 25).

### Index timing, hedonic regression, and geometric means
- Index timing choice (period versus point): Turvey argues indices used for deflating income, expenditure or sales should relate to the period of the money flow; for economic analysis most statistics relate to a period and the CPI should do the same. Practical considerations often force price statisticians to collect prices at a point in time. Turvey (1989; 25).
- Hedonic regressions: Turvey describes key features on pages 80-82 — regress price on price-determining characteristics; particularly important for property price indexes.
- Geometric mean endorsement (quoted): “In view of this superiority, it is not surprising that many statisticians regard the use of geometric means as the best solution. Why, then, are they so seldom used? One reason is that they make the calculations difficult, but this argument loses its validity once computers are used for calculating the index. A second reason is that it may be feared that their use is too difficult to explain to users of the index. But most indices have features which are difficult to explain, and in any case, the degree of complexity that is acceptable is growing, through time, in most countries. Statisticians should have the courage of their convictions.” Turvey (1989; 90-92).

*Content attributed to the Fourteenth International Conference of Labour Statisticians (ILO, October 28 to November 6, 1987), Turvey (1989), and Szulc (Appendix 7, 1987).*

### 4. The First Ottawa Group Meeting

### 4. The First Ottawa Group Meeting

### Meeting purpose and scope
- The first Meeting of the Ottawa Group took place in Ottawa over October 31 to November 2, 1994. Jacob Ryten of Statistics Canada, the Chair of the Meeting, explained the working group’s purpose: to bring together an independent forum of specialists from different countries to exchange ideas on crucial problems of measuring price change and to propose concrete solutions.
- The first meeting agreed to concentrate discussion on two main issues:
  - the micro level aggregation and its macro effects, and
  - how to detect and estimate the bias of the consumer price index.
- Ryten listed related research areas for participants to address, including:
  - Relevance of the CPI as an indicator of the Cost of Living Index,
  - Harmonisation of European price indices,
  - Importance of agreeing on definition and concept of cost of living vs. fixed basket,
  - Defining a wider measurement of inflation (ex. whole economy price index),
  - Need for another economic approach to monitor inflation.
- The European Union’s HICP was under construction at this time and required a harmonized strategy.

### Key empirical findings and methodological debates
- Bohdan Schultz’s work (referencing earlier 1983/1987 papers) showed that at the elementary index level the chained Carli formula could lead to tremendous upward bias if prices were volatile; at the macro level, fixed-basket indices are subject to possible upward substitution bias. Schultz’s analysis of Statistics Canada monthly micro data for over 50 commodities (collected from December 1988 to January 1994 in the province of Ontario) found an upward bias in the Carli index and recommended changing to the Jevons formula at the elementary level.
- U.S. BLS research summarized by Paul Armknecht, Brent Moulton and Kenneth Stewart (1994) and reviewed in later work found:
  - Reinsdorf (1993) compared trends in US average prices for relatively homogeneous product groups with official BLS price indexes and found official indexes increased substantially faster; example: the official index for food showed average annual increases during the 1980’s of 4.2% per year while the weighted mean of average prices grew at only 2.1% per year.
  - Subsequent work (Moulton (1993), Reinsdorf and Moulton (1997), Reinsdorf (1998)) showed the observed bias was due to elementary formula bias (use of the Carli formula) rather than outlet substitution bias; this research ultimately led to the use of the Jevons formula at the elementary level.
- Sellwood (1994) on HICP design:
  - Argued the main use of the HICP should be to measure inflation across member countries in a way useful to Central Banks; imputed prices for Owner Occupied Housing and Household Production should not appear in an HICP.
  - Debated Carli vs. Dutot at the elementary level: Dutot gives too much weight to higher priced items (argument favoring Carli) but Dutot satisfies circularity and time reversal tests while Carli fails both tests.
  - Expressed skepticism about the classical economic approach to CPI construction: “The classical model of a consumer maximising utility by applying a rational decision process to balance costs and benefits seems remote from common experience.” Sellwood (1994; 3).
- Jörgen Dalén (1992, 1994) and Carruthers, Sellwood and Ward (1980) developed numerical approximations explaining systematic differences among elementary index formulae and initiated a test approach to elementary indexes. The Jevons index satisfied the eight bilateral-index tests Dalén examined.

### Scanner data, unit values, and aggregation issues
- Alain Saglio (1994) used two years of scanner data (Nielsen) on chocolate bar sales in France and found:
  - An index of average prices decreased 1.6% per year while the corresponding Laspeyres index decreased only 0.2% per year.
  - He attributed the difference to substitution effects: households switched to lower cost outlets and brands and the Laspeyres index did not capture these substitution effects.
- The introduction of scanner data to the Ottawa Group made it possible to compute elementary indexes using price and quantity information, enabling use of unit values or superlative indexes at the elementary level where transaction data exist.
- Diewert (1995a) advocated use of unit values to represent prices used in an elementary index number formula, noting the companion quantity is total quantity purchased during the period by the group of consumers in scope.
- Diewert highlighted that unit values for a homogeneous commodity can be decomposed into outlet effect, brand effect, and packaging effect; fine classification by outlets, brands and packages should be done if requisite data are available, but empirical aggregation choices may reduce required fineness.
- Ivancic and Fox (using Australian scanner data) ran weighted time-product-dummy hedonic regressions and found empirical support for aggregating prices across stores belonging to the same supermarket chain and across three of four chains studied, suggesting hedonic clustering methods for scanner-data aggregation.

### Data access, agency choices, and practical implications
- Diewert raised the dilemma for Statistical Agencies in accessing scanner data:
  - Option (i): buy data from private information-processing firms (leads to loss of control by the Statistical Agency).
  - Option (ii): set up an information-processing subsidiary to compete with the private firm (may lead to charges of unfair competition).
- Diewert concluded more public discussion is required but that Statistical Agencies will eventually be forced to join the electronic highway in one form or another. Diewert (1995a; 24).

### Summary estimates of CPI biases
- Diewert (1995a; 35) summarized empirical evidence with additive annual bias estimates for a typical official CPI:
  - commodity substitution bias: .2% per year
  - outlet substitution bias: .25% per year
  - linking bias: perhaps .1% per year
  - new goods bias: at least .25% per year
  - summed upward bias of at least .8% per year
- Diewert noted that if a Statistical Agency also used a biased elementary price index formula, this would add additional upward bias. These estimates are similar to the Boskin Commission estimates because they drew on the same available studies.

### Follow-up agenda and outcomes
- In closing remarks Jacob Ryten proposed ten topics for future Ottawa Group meetings (topics remain relevant):
  - How to establish price indices for difficult areas such as insurance and gambling fees payable to a state agent.
  - Product quality adjustment and the use of hedonic methods.
  - The necessary steps for the harmonization of CPI.
  - Clarifying concepts (e.g. What is a cost of living index?).
  - Treatment of new products and outlets in price indices.
  - Treatment of durable goods in the CPI.
  - Index formulae at macro level, including the linking problem.
  - Organization and techniques related to price surveys.
  - Linkage between temporal and spatial price comparisons.
  - Problem of seasonality and price indices.
- The need for a revised CPI Manual was recognized in 1998 by international organisations and ultimately resulted in the Consumer Price Index Manual: Theory and Practice (finished in 2004; editor Peter Hill). The 2004 Manual’s theoretical chapters reflected the state of index number theory discussed in the Ottawa Group in the mid-1990s.

*Source: Ottawa Group (1994); Diewert (1994, 1995a); Sellwood (1994); Schultz/Szulc (1994); Armknecht, Moulton and Stewart (1994); Dalén (1992, 1994); Saglio (1994); Ivancic and Fox (2011a) as presented in the meeting materials and proceedings.*

### 5. The First Ottawa Group CPI Manual

### The First Ottawa Group CPI Manual

### Treatment of seasonal products
- The Turvey Manual discussed seasonal products and user costs for housing, but Chapters 22 and 23 of the 2004 Manual discussed these topics in much more detail.
- Chapter 22 recommended Rolling Year Mudgett-Stone indexes and year over year monthly indexes (with month specific weights) as checks on seasonal product components of the official month to month CPI.
- The 2004 Manual concluded pessimistically on strongly seasonal products: “It is evident that more research needs to be carried out on the problems associated with the index number treatment of seasonal commodities. There is, as yet, no consensus on what is best practice in this area.” ILO/IMF/OECD/UNECE/Eurostat/The World Bank (2004; 417).

### Treatment of durable goods and Owner Occupied Housing (OOH)
- Chapter 23 provided a more comprehensive treatment of durable goods and OOH than the Turvey Manual.
- The 2004 Manual noted that the user cost for a dwelling unit should be decomposed into separate user cost terms for the land and structure components of the property.
- The Manual observed that the acquisitions approach tends to give a much smaller weight to OOH than the user cost and rental equivalence approaches.
- Both the Turvey and 2004 Manuals concluded that more than one approach to the treatment of OOH is required to address different CPI purposes.
- Despite this, National Statistical Offices have not provided alternative indexes for OOH on a regular basis.

### Chaining, dissimilarity indexes, and chain drift
- The 2004 Manual discussed choosing between fixed base/direct indexes versus chained indexes and recommended chaining when prices and quantities of adjacent periods are more similar than more distant periods (ILO/IMF/OECD/UNECE/Eurostat/The World Bank (2004; 281)).
- The Manual referenced Diewert (2002) for explicit measures of absolute and relative price dissimilarity and suggested that in practice chaining would normally be “best”.
- Ivancic and Fox (2011b) questioned the sufficiency of dissimilarity indexes for choosing when to chain: they found dissimilarity indexes “do not appear to be sufficient to resolve the issue of when to chain.”
- Empirical summary from Ivancic and Fox (2011b; 5):
  - For their data set, chaining was appropriate in 47 out of 76 cases.
  - In 20 cases there was no clear evidence on the issue of chaining.
  - In 9 out of 76 cases chaining was not satisfactory.
- Chaining a Carli, Laspeyres or Paasche index often results in massive chain drift if prices “bounce”; even chaining superlative indexes can produce drift.

### Scanner data, de Haan’s evidence, and magnitude of chain drift
- Jan de Haan (2008) used scanner data from more than 100 supermarkets covering week 1 in 2005 to week 35 in 2008 (191 weeks) for hundreds of detergents.
- Key empirical facts from de Haan’s study:
  - Average number of matched products in each consecutive two week period: 53.
  - Range of matched products: between 43 and 63.
  - Example product (XXX Tablets): normal price about 6.5 Euros; went on sale for 12 of the 191 weeks with sale price approximately one half of the regular price; weekly sales rose from virtually 0 to over 3000 for 7 of the 12 sale weeks.
- De Haan documented massive chain drift:
  - Chained Fisher index in week 191 ended at 4.95% of the week 1 value.
  - Chained Törnqvist index in week 191 ended at 7.43% of the week 1 value.
  - Chained Jevons index using bilateral matched products at each link ended at 76.65% of the week 1 index level.
- Chain drift is typically downward due to purchaser stockpiling when goods are on sale, though upward chain drift can also occur (Feenstra and Shapiro (2003); Persons (1928; 100-105) numerical example).

### Responses: multilateral indexes and the Rolling Window GEKS
- De Haan proposed several mitigations including increasing the length of the period or using fixed base indexes, but fixed base indexes become unreliable with rapid product turnover (de Haan problem).
- Ivancic, Fox and Diewert (2009) proposed multilateral indexes over a window of consecutive periods to eliminate chain drift within the window (circularity test satisfied).
- Two suggested multilateral indexes: the GEKS index and the Weighted Time Product Dummy (WTPD) index.
- Multilateral methods have a limitation: a product present in only one period of the window has no effect on resulting price levels, so new products entering in period T have no effect when multilateral indexes rely on matched product prices (this holds for TPD, WTPD and GK price levels).
- Ivancic, Fox and Diewert (2009, 2011) proposed a Rolling Window GEKS methodology:
  - Choose a window length of 13 months (to include strongly seasonal products).
  - Compute GEKS multilateral price levels for months 1–13.
  - When month 14 data arrive, compute GEKS for months 2–14 and use the ratio of month 14 to month 13 price levels to update the month 13 level of the permanent index; repeat for subsequent months.
  - Their comparison of RWGEKS to GEKS constructed over the entire 15 month period showed little difference for that short span.
- Subsequent evidence (Fox, Levell and O’Connell (2024)) indicates that the Rolling Window Method can still exhibit some chain drift over longer time periods.
- Linking strategies for rolling windows:
  - Ivancic, Diewert and Fox (2011) suggested linking the movement of the rolling window indexes for the last two periods in the new window to the last index value generated by the previous window.
  - Krsinich (2016) proposed linking to the previous window index value for the second period in the previous window (a “window splice”).
  - De Haan (2015; 27) suggested linking in the middle of the old window (“half splice” as termed by ABS (2016; 12)).
  - Ivancic, Diewert and Fox (2011; 33) proposed the geometric average of all links for the last period in the new window to observations in the old window as a linking factor.
  - Diewert and Fox (2021) examined alternative linking strategies and recommended average or mean linking as statistically safest.
  - Diewert and Shimizu (2024) suggested using an ever expanding window; this strategy is pursued in Appendix A.

### The 2020 CPI Manual and new material
- Work to update the 2004 Manual began in 2015 and culminated in Consumer Price Index Manual: Concepts and Methods 2020 (editor Brian Graf; lead institution IMF).
- The 2020 Manual purpose: “The Manual is intended for the benefit of agencies that compile CPIs, as well as users of CPI data. It explains in some detail the methods that are recommended for use to calculate a CPI. A separate companion publication, Consumer Price Index Theory, explains the underlying economic and statistical theory on which the methods are based.” IMF/ILO/UNECE/Eurostat/OECD/The World Bank (2020; xi).
- The companion Theory Manual has not been finalized due to rapid CPI theoretical research; draft chapters of the CPI Theory Manual are available on the IMF CPI website.
- The 2020 Manual:
  - Added new chapters including Chapter 10 on Scanner Data (written by Jan de Haan), covering multilateral indexes.
  - Contains 14 chapters and 7 Appendices.
  - New Appendices include topics such as The Harmonised Index of Consumer Prices (European Union), COICOP 1999 and COICOP 2018, Spatial Comparisons, Basic Index Number Formulas, and The Consumer Price Index Research Agenda.

### Research agenda and forward-looking items
- Appendix 7 of the 2020 CPI Manual lists research topics for future Ottawa Group focus:
  - Scanner data
  - Web scraped prices
  - Price updating of expenditure weights
  - The use of administrative data to form indexes
  - Plutocratic versus democratic weighting of households
  - The use of credit card data to form household specific price indexes
  - Calculating elementary indexes using expenditure weights
  - Quality adjustment
  - The treatment of seasonal products
  - The use of target price indices for the CPI
  - On the choice of formula for calculating higher level price Indices
  - The use of long-term and short-term Links
  - Retrospective Calculations of Superlative Price Indexes
  - Consumer price indexes for different groups and geographic areas
  - Measuring “hard to measure” services
  - Insurance and financial services
  - Owner Occupied Housing
  - Digitalization
  - Well being and sustainability
- The 2020 Manual observed on measuring welfare with free goods and services: “The issue of a CPI for measuring economic wellbeing is not restricted to the effects of digitalization, but also includes further discussion on the coverage of the CPI and the treatment of different types of goods and services provided for free, potentially including public goods and services such as education, health, safety, or parks. The issue relates to the discussion of cost of goods indices versus cost of living indices, and conditional versus unconditional cost of living indices, where additional experiences and guidance would be useful. It may be useful to invite experts from other areas of official statistics to discuss measuring welfare and economic well-being.” IMF/ILO/UNECE/Eurostat /OECD/The World Bank (2020; 458).

### Anticipated data and methodological developments
- Diewert (2001) speculated on future data submissions and methods:
  - Firms submitting price and quantity data via the internet to NSOs has occurred.
  - Diewert was less sanguine about NSOs obtaining household price and quantity data directly, but market research firms now provide household purchase data (credit/debit card data) to researchers and NSOs (at a price); these data suffer from lack of specific product codes and highly aggregated product categories.
  - Web scraping of prices would lead to more accurate CPIs and better information on used durable goods to facilitate user cost approaches to durable purchases.

*Diewert — discussion of The First Ottawa Group CPI Manual and subsequent Ottawa Group CPI Manuals (sections 5–7 of the supplied content).*

### 7. Looking Ahead

### 7. Looking Ahead

### Diewert’s predictions on CPI production and seasonality
- Diewert expected statistical agencies (NSOs) would produce families of indexes, notably at least 2 CPIs: (i) a Nonrevisable CPI and (ii) A Revisable CPI.
- He expected different CPIs to be produced to suit different purposes.
- For strongly seasonal products, he predicted NSOs would produce at least two indexes for seasonal product strata:
  - one index focusing on year over year price change in the same month (using seasonal baskets), and
  - another index attempting to measure month to month price change.
- The prediction that NSOs would routinely produce multiple CPI families and separate seasonal-strata indexes "has not come about" or "has also not materialized" as indicated in the text.

### Hedonic regression models and methodological progress
- Diewert (2001; 111) anticipated that many unresolved problems with hedonic regression models would be solved.
- The text reports that this seems to be the case, citing Triplett (2004) and section 2 of Diewert and Shimizu (2024).

### Household time allocation and extensions of Becker’s framework
- Diewert speculated that once information on households’ allocation of time became available, many theoretical innovations in modeling household behavior would be incorporated into CPI design.
- Three innovations singled out:
  - (i) Implementation of Becker’s (1965) theory of the allocation of time.
    - Some progress has been made; see Schreyer and Diewert (2014) and Diewert, Fox and Schreyer (2017).
    - Incorporating the time constraint into standard consumer theory would enable better valuation of the contribution of free goods and services to household standard of living.
  - (ii) Implementation of an extended version of Becker’s model to cover household nonmarket production.
  - (iii) Implementation of an extended version of Becker’s model to medical economics.
- Key point: obtaining information on household allocation of time is the key to improving measurement of the cost of living and household welfare.
- Progress in implementing household time surveys has been slow and will probably continue to be slow.
- The basic conceptual difficulty: it is not conceptually simple to measure the household allocation of time across various activities.

### Appendix — The Consumer Demand Approach to Multiperiod Indexes: key ideas
- Implementing a simplified version of the Diewert and Feenstra (2017) (2022) approach to estimation of reservation prices is "not more complicated than calculating GK price indexes or running a Time Product Dummy regression."
- The approach estimates the consumer’s utility function directly rather than the dual unit cost function.
  - The expenditure (cost) approach to estimating consumer preferences is not workable when there are missing products in some periods.
  - Direct estimation of utility functions (as in forming GK indexes) is workable.
- When a product is not available in a period:
  - the quantity consumed is 0 and this can be observed,
  - the corresponding reservation price cannot be observed,
  - this observability structure allows direct estimation of utility functions.

### Simple two-period example illustrating new-product bias
- Period 1:
  - only product 1 is available.
  - consumer has a budget equal to e1 > 0 to spend on products in this category.
  - price of product 1 is p11 > 0.
  - consumer purchases q11 = e1 / p11 units of product 1 in period 1.
- Period 2:
  - a new product 2 appears.
  - consumer has e2 to spend on the two products and faces p21 q1 + p22 q2 = e2.
  - for simplicity, assume e2 = e1 and p21 = p11.
  - consumer could choose q1 = q11 and q2 = 0 in period 2, but any point on the budget line that starts at q11 is feasible.
  - the utility-maximizing choice in period 2 is the point (q21, q22) where the highest attainable indifference curve is tangent to the budget line.
- Implication for linear vs. nonlinear preferences:
  - If preferences are linear, the budget line would be the highest attainable indifference curve and q1* would coincide with q11 = q21, producing no bias.
  - Typically, preferences are not linear; linear utility function models lead to a quantity index that is too low and the corresponding price index has an upward new product bias in the period when the new product first appears.
  - This fundamental problem affects all linear utility function models, including GK indexes, bilateral superlative indexes that rely on matched product prices and their multilateral extensions, and hedonic regression models that use time dummy variables.
- Estimating the consumer’s curved indifference curve requires waiting for period 3 data:
  - one must observe periods where both products 1 and 2 are available and the relative price of products 1 and 2 has changed in period 3 to identify the curvature and correct bias.

---


_Source: https://www.imf.org/-/media/files/data/cpi/companion-publication/ppt-diewert-on-the-ottawa-group-after-30-years.pdf_