## wpiea2019040 ## Source details **Canonical URL:** [wpiea2019040](https://www.imf.org/-/media/files/publications/wp/2019/wpiea2019040.pdf) ## Other formats - [Markdown version](/-/media/files/publications/wp/2019/wpiea2019040.pdf.md) - [Structured JSON version](/-/media/files/publications/wp/2019/wpiea2019040.pdf.json) --- ### Key findings and implications - Correcting for classification errors increases the U.S. unemployment rate: on average the true unemployment rate is 0.8 percentage points higher than the official unemployment rate. - Flow decomposition (contributions to unemployment fluctuations): - The job finding probability ("outs of unemployment") remains the more important margin in explaining unemployment fluctuations. - The job separation probability ("ins to unemployment") matters more than previously estimated: its contribution rises to around 30 percent relative to the previous estimate of 20 percent (Shimer, 2012). - The participation (nonparticipation) margin contributes much less once measurement error is corrected: around 10 percent in this paper vs previous estimates of 20 to 30 percent (Elsby, Hobijn and Sahin, 2015). - Overall interpretation: “outs” still dominate, but “ins” are more important than earlier studies suggested; policymakers should pay closer attention to the job separation margin and less to the nonparticipation margin when addressing unemployment fluctuations. - The proposed methodology can be applied to any labor force survey in which labor force status is observed for three periods. ### Comparison of measurement-error correction methods - Reinterview-based methods (Abowd and Zellner, 1985; Poterba and Summers, 1986): - Provide reporting-error matrices with substantial off-diagonal misclassification probabilities (examples shown for 1977:1-1982:4 in the source). - Caveats: reinterview surveys are not readily accessible to outside researchers, are from more than 30 years ago, and reinterviews themselves may be biased (small subsample, telephone mode, time lag, respondent access to original answers). - Latent-variable / HMM approaches (Feng and Hu, 2013; Hu, 2008; Hu, 2017; current paper): - Infer true labor force status from multiple periods of reported status without external data. - Feng and Hu (2013) find the official unemployment rate underestimated by approximately 2.1 percentage points between 1996 and 2011. - The current paper follows the latent-variable approach but imposes a first-order Markov assumption on true labor force transitions. - Practical advantage of first-order HMM: restricts inferred flow probabilities to the plausible range (between 0 and 100) and is usable for flow analyses; some alternative adjustments can imply flow probabilities outside the [0,100] range. ### Model, assumptions, and identification - Model specification: - Hidden Markov Model (HMM) / latent variable approach estimating reporting-error matrix B (3×3) and true transition matrix A (3×3) for labor force statuses E (1), U (2), O (3). - Main assumptions: - Assumption 1 (Conditional Independence): Pr(Yt | Ỹt, Yτ≠t, Ỹτ≠t) = Pr(Yt | Ỹt). - Assumption 2 (First Order Markov): Pr(Ỹt+1 | Ỹt, Ỹt−1, ..., Ỹ1) = Pr(Ỹt+1 | Ỹt); transition probabilities aij = Pr(Ỹt+1 = j | Ỹt = i). - Assumption 3 (Linear independence): Distributions Pr(Yt | Yt−1 = k) for k = 1,2,3 are linearly independent; implies ΦYt,Yt−1 is invertible. - Assumption 4 (Distinctiveness / invertibility): AB is invertible and diagonal matrices Dk|Ỹt are distinct; ensures distinct eigenvalues and uniqueness of eigenvectors. - Assumption 5 (Ordering): Pr(Yt = i | Ỹt = i) > Pr(Yt = i | Ỹt = j) for all i ≠ j. - Identification formulas (as in source): - Reporting-error matrix B recovered via eigenvalue–eigenvector decomposition of Φ1,Yt,Yt−1 Φ−1Yt,Yt−1. - Distribution Π of true labor force status recovered by inverting ΦYt|Ỹt. - True transition matrix A recovered via Aᵀ = (Bᵀ)⁻¹ ΦYt,Yt−1 B⁻¹ diag(Π)⁻¹. - Estimation: - Maximum likelihood estimation (MLE) of the HMM jointly estimates B and A (guarantees a local maximum and uses the entire sample). - Eigen-decomposition provides global identification but uses only a subsample; MLE results are generally similar to eigen-decomposition results. - Robustness checks verify first-order Markov is adequate for three-month labor market transitions. ### Data, matching, and weighting - Data: Current Population Survey (CPS) January 1996 to September 2018. - CPS interviews approximately 60,000 households and 100,000 to 150,000 individuals monthly. - Rotation scheme: households interviewed 4 consecutive months, rotated out 8 months, then interviewed for final 4 months → tracking up to 8 months over 16 months. - Matching procedure: - Match CPS monthly files for three consecutive months using Madrian and Lefgren (2000) algorithm on household identifier, household replacement number, personal identifier, sex, age (allowed to differ by 1), and race. - Eligible rotation groups for three months of history ≈ rotation groups 1, 2, 5, and 6 → about one-half of initial survey sample can be matched; attrition/coding errors further reduce matches. - Re-weighting: - To correct nonrandom attrition, sample weights for matched individuals multiplied by inverse of predicted matching probability conditional on gender, age, race, education, and initial employment status using a logit model. - Results reported as robust to the re-weighting procedure. ### Reporting-error and adjusted-rate quantitative findings - Reporting error patterns: - Misreporting is most severe among the unemployed: unemployed workers are more likely to misreport as out of the labor force than as employed across methods (HMM, FH, AZ, PS). - Feng and Hu (FH) find close to 40 percent of unemployed misreport their status; FH estimates are much larger than HMM, AZ, PS. - HMM estimates lie between FH and AZ/PS and are generally similar to Abowd and Zellner (1985) and Poterba and Summers (1986). - Adjusted unemployment rates (average differences from reported official rate): - FH-adjusted: 2.6 percentage points higher. - HMM-adjusted: 0.8 percentage points higher. - PS-adjusted: 0.24 percentage points higher. - AZ-adjusted: 0.54 percentage points higher. - Relative ordering: FH > HMM > AZ/PS > reported unadjusted. ### Selected misclassification probabilities and transition probabilities (values preserved) - Misclassification probabilities Pr(j | ˜i) = Pr(Y_t=j | ˜Y_t=i) (in percentage points): - Pr(E | ˜E): AZ 98.8, PS 97.7, FH 97.9, HMM 98.7 (0.06) - Pr(U | ˜E): AZ 0.2, PS 0.5, FH 0.6, HMM 0.4 (0.04) - Pr(O | ˜E): AZ 1.0, PS 1.7, FH 1.5, HMM 0.9 (0.04) - Pr(E | ˜U): AZ 1.9, PS 3.8, FH 17.3, HMM 7.5 (0.65) - Pr(U | ˜U): AZ 88.6, PS 84.8, FH 62.5, HMM 76.4 (0.98) - Pr(O | ˜U): AZ 9.5, PS 11.5, FH 20.2, HMM 16.1 (0.84) - Pr(E | ˜O): AZ 0.5, PS 1.2, FH 2.9, HMM 1.4 (0.09) - Pr(U | ˜O): AZ 0.3, PS 0.6, FH 0.2, HMM 1.0 (0.10) - Pr(O | ˜O): AZ 99.2, PS 98.2, FH 96.9, HMM 97.6 (0.13) - Note: Bootstrap standard errors based on 500 simulations reported in parentheses for HMM entries shown. - Average transition probabilities Pr(j | i) = Pr(Y_t=j | Y_{t-1}=i) (in percentage points): - Pr(E | E): R* 95.8, AZ 97.2, PS 98.6, FH 100.5, HMM 98.3 - Pr(U | E): R* 1.3, AZ 1.2, PS 0.7, FH -0.4, HMM 0.7 - Pr(O | E): R* 2.9, AZ 1.6, PS 0.6, FH -0.2, HMM 1.0 - Pr(E | U): R* 24.7, AZ 21.9, PS 16.0, FH 0.8, HMM 15.3 - Pr(U | U): R* 50.7, AZ 60.3, PS 69.0, FH 91.4, HMM 77.7 - Pr(O | U): R* 24.5, AZ 17.8, PS 15.0, FH 7.7, HMM 7.0 - Pr(E | O): R* 5.3, AZ 2.5, PS 0.3, FH -1.4, HMM 1.4 - Pr(U | O): R* 3.1, AZ 2.3, PS 1.8, FH 1.0, HMM 0.9 - Pr(O | O): R* 91.6, AZ 95.1, PS 97.8, FH 100.4, HMM 97.7 - Average durations (in months): - E: R* 23.5, AZ 35.9, PS 73.9, FH -193.2, HMM 58.2 - U: R* 2.0, AZ 2.5, PS 3.2, FH 11.7, HMM 4.5 - O: R* 12.0, AZ 20.6, PS 46.1, FH -242.1, HMM 43.8 - Note: Negative or extreme FH durations illustrate pathologies when corrected transition matrices imply absorbing states. ### Costs and benefits of imposing the First Order Markov assumption - Benefits: - Consistent with macro labor literature and enables direct comparison with Shimer (2012) and Elsby et al. (2015). - Imposes bounds on transition probabilities (between 0 and 100 percent), avoiding implausible adjusted probabilities. - Avoids additional attrition and matching-error from linking CPS over longer rotations (matching over first four consecutive months ≈ 90 percent of eligible rotation groups; matching at 12th survey drops to approximately 65 percent). - Uses three months of labor force history, reducing matching attrition. - Costs and limitations: - Micro evidence shows duration dependence and heterogeneity; first-order Markov imposes homogeneity and ignores duration dependence. - May misrepresent individual-level transition dynamics, but for short periods the approximation can be innocuous. - Robustness checks performed: - Simulate second-order Markov DGP and estimate first-order HMM; show distribution of true labor force history and measurement error matrix can generally be recovered. - Allow transition probabilities to vary over time while keeping reporting-error matrices fixed. ### Adjusted transition probabilities and contributions to unemployment fluctuations - HMM-adjusted probabilities: - HMM imposes first-order Markov and jointly estimates time-varying true transition probabilities; ensures probabilities stay within [0,100]. - HMM implies truly unemployed workers have a high probability of incorrectly reporting as nonparticipant: Pr(Y_t = O | ˜Y_t = U) = 16.1. - HMM average job finding probability (Pr(˜Y_{t+1}=E | ˜Y_t=U)): around 15 percent. - Unadjusted job finding probability: 25 percent; AZ: 22 percent; PS: 16 percent. - HMM average transition from employment to unemployment: 0.7 percent. - Measurement-error adjustments generally revise unadjusted flows in the same direction as AZ, PS, and FH; degree of adjustment: FH > HMM > AZ/PS. - Contributions to unemployment rate and employment-population (EP) ratio (Table 5 values preserved): - λ_{EU} contributions: - Unemployment Rate 1996-2007: R 0.20, AZ 0.24, HMM 0.30 - Unemployment Rate 1996-2018: R 0.20, AZ 0.25, HMM 0.27 - Employment-Population Ratio 1996-2007: R 0.14, AZ 0.11, HMM 0.16 - Employment-Population Ratio 1996-2018: R 0.22, AZ 0.18, HMM 0.12 - λ_{UE} contributions: - Unemployment Rate 1996-2007: R 0.53, AZ 0.57, HMM 0.53 - Unemployment Rate 1996-2018: R 0.46, AZ 0.50, HMM 0.52 - Employment-Population Ratio 1996-2007: R 0.49, AZ 0.39, HMM 0.21 - Employment-Population Ratio 1996-2018: R 0.59, AZ 0.50, HMM 0.23 - Combined margins and other entries (values preserved in Table 5): - λ_{UE} + λ_{EU} (Unemployment Rate 1996-2007): R 0.73, AZ 0.81, HMM 0.83 - λ_{OU} + λ_{UO} (Unemployment Rate 1996-2007): R 0.24, AZ 0.13, HMM 0.09 - λ_{EO} + λ_{OE} (Unemployment Rate 1996-2007): R 0.41, AZ 0.42, HMM 0.53 - Key interpretive outcomes: - “Outs” (job finding) still explain more fluctuation than “Ins” (job separation), but HMM adjustments increase the role of “Ins” relative to unadjusted series. - Example (1996-2007): reported λ_{UE} = 0.53 and λ_{EU} = 0.2 (unadjusted); HMM reports λ_{UE} = 0.53 and λ_{EU} = 0.30. Fraction of “In”: - without adjustment: 100× 0.2/(0.53+0.2) ≈27.4percent - with HMM: 100× 0.3/(0.53+0.3) ≈36.1percent - Participation margin explains approximately 10 percent of unemployment fluctuations under HMM (vs 20 to 30 percent in prior literature). ### Policy recommendations and takeaways - Policymakers should: - Pay closer attention to the job separation margin when designing policies to mitigate unemployment fluctuations (e.g., policies discouraging firings, encouraging adjustment via hours rather than headcount). - Place less emphasis on the nonparticipation (participation) margin than previous literature suggested. - Measurement correction matters: unemployment levels and flow contributions change materially once classification errors are accounted for; employ correction methods that ensure adjusted probabilities remain within feasible bounds. ### Appendix: validity of First Order Markov assumption (simulation evidence) - Simulation design: - DGP: second-order Markov process Pr(Ỹt | Ỹt−1, Ỹt−2). - Generated N = 60000 observations; used measurement-error matrix B_DGP and second-order transition matrix Ψ (example Ψ rows/columns reported in percentages). - Example Ψ entries (rows indexed by (Yt−1,Yt); columns Yt+1 = E,U,O) include: - EE: 98.3 0.7 1.0 - EU: 13.7 79.3 7.0 - EO: 1.8 0.8 97.4 - UE: 97.8 1.1 1.1 - UU: 12.4 81.5 6.1 - UO: 2.5 0.0 97.5 - OE: 98.1 0.2 1.7 - OU: 14.6 78.9 6.5 - OO: 1.4 0.9 97.6 - Estimation results (first-order HMM on simulated data): - Estimated measurement-error matrix B closely matches B_DGP; selected comparisons (percentage points): - Pr(E | Ỹ = E): B_DGP 98.7 vs B Estimate 98.7 [98.4,99.0] - Pr(E | Ỹ = U): B_DGP 7.0 vs B Estimate 6.2 [3.2,9.1] - Pr(U | Ỹ = U): B_DGP 77.8 vs B Estimate 78.1 [73.6,82.6] - Pr(O | Ỹ = U): B_DGP 15.2 vs B Estimate 15.8 [11.4,20.2] - Pr(O | Ỹ = O): B_DGP 97.5 vs B Estimate 97.7 [97,98.3] - Implied first-order transition probabilities (percentage points) recovered closely: - Pr(Ỹ = E | Ỹ = E): DGP 98.3 vs HMM 98.3 [97.9,98.7] - Pr(Ỹ = E | Ỹ = U): DGP 13.9 vs HMM 12.8 [8.8,16.8] - Pr(Ỹ = U | Ỹ = U): DGP 79.5 vs HMM 80.5 [74.8,86.3] - Bootstrap Confidence Intervals based on 500 simulations reported in brackets. - Accuracy metrics: - Weighted mean absolute error (WMAE) between true distribution Pr(Ỹt+1 = k, Ỹt = j, Ỹt−1 = i)DGP and estimated Pr(·)HMM is 0.0014 percentage points. - Average probability of a possible employment path cited as 3.7 percent (reference scale). - Sensitivity checks (B_DGP_max and B_DGP_min: ±2 standard deviations on off-diagonals): - WMAE results: - WMAE for B_DGP: 0.0014 percentage points. - WMAE for max case: 0.001 percentage points. - WMAE for min case: 0.06 percentage points. - Conclusion: first-order HMM recovers the true parameters well even when the DGP is second-order Markov; some misclassification probabilities involving nonparticipation (O) are less precisely recovered but overall robustness holds. *Source: wpiea2019040 (PDF).* ### 2.0 ppts higher between 1996 and 2018. This paper further quantifies the contributions to business- ### wpiea2019040 - 2.0 ppts higher between 1996 and 2018. This paper further quantifies the contributions to business- ### Key findings and implications - Correcting for classification errors increases the U.S. unemployment rate: on average the true unemployment rate is 0.8 percentage points higher than the official unemployment rate. - Flow decomposition (contributions to unemployment fluctuations): - The job finding probability ("outs of unemployment") remains the more important margin in explaining unemployment fluctuations. - The job separation probability ("ins to unemployment") matters more than previously estimated: its contribution rises to around 30 percent relative to the previous estimate of 20 percent (Shimer, 2012). - The participation (nonparticipation) margin contributes much less once measurement error is corrected: around 10 percent in this paper vs previous estimates of 20 to 30 percent (Elsby, Hobijn and Sahin, 2015). - Overall interpretation: “outs” still dominate, but “ins” are more important than earlier studies suggested; policymakers should pay closer attention to the job separation margin and less to the nonparticipation margin when addressing unemployment fluctuations. - The proposed methodology can be applied to any labor force survey in which labor force status is observed for three periods. ### Comparison of measurement-error correction methods - Reinterview-based methods (Abowd and Zellner, 1985; Poterba and Summers, 1986): - Provide reporting-error matrices (examples shown for 1977:1-1982:4) with substantial off-diagonal misclassification probabilities (tables reported in the source). - Caveats: reinterview surveys are not readily accessible to outside researchers, are from more than 30 years ago, and the reinterviews themselves may be biased (small subsample, telephone mode, time lag, respondent access to original answers). - Latent-variable / HMM approaches (Feng and Hu, 2013; Hu, 2008; Hu, 2017; current paper): - Infer true labor force status from multiple periods of reported status without external data. - Feng and Hu (2013) find the official unemployment rate underestimated by approximately 2.1 percentage points between 1996 and 2011. - The current paper follows the latent-variable approach but imposes a first-order Markov assumption on true labor force transitions. - Some alternative adjustment methods (Poterba and Summers, 1986; Feng and Hu, 2013) can imply flow probabilities outside the [0,100] range in some cases; the latent-variable HMM with a first-order Markov assumption restricts probabilities to a reasonable range (between 0 and 100) and is thus usable for flow analyses (e.g., Shimer, 2012; Elsby, Hobijn and Sahin, 2015). ### Model, assumptions, and identification - Model: Hidden Markov Model (HMM) / latent variable approach estimating reporting-error matrix B (3×3) and true transition matrix A (3×3) for labor force statuses E (1), U (2), O (3). - Assumption 1 (Conditional Independence): Pr(Yt | Ỹt, Yτ≠t, Ỹτ≠t) = Pr(Yt | Ỹt). The reporting error probabilities depend only on the true labor force status in the current period. - Assumption 2 (First Order Markov): The true labor force status Ỹt follows a first-order Markov process: Pr(Ỹt+1 | Ỹt, Ỹt−1, ..., Ỹ1) = Pr(Ỹt+1 | Ỹt). Transition probabilities aij = Pr(Ỹt+1 = j | Ỹt = i). - Assumption 3 (Linear independence): Distributions Pr(Yt | Yt−1 = k) for k = 1,2,3 are linearly independent; implies ΦYt,Yt−1 is invertible. - Assumption 4 (Distinctiveness / invertibility): AB is invertible and diagonal matrices Dk|Ỹt are distinct; ensures distinct eigenvalues of Φ1,Yt,Yt−1 Φ−1Yt,Yt−1 and uniqueness of eigenvectors. - Assumption 5 (Ordering): Pr(Yt = i | Ỹt = i) > Pr(Yt = i | Ỹt = j) for all i ≠ j; pins down label ordering (employed, unemployed, nonparticipant). - Identification and formulas: - Observed joint distributions relate to latent parameters via matrix equations (see equations (1)–(9) in the source). - Reporting-error matrix B is recovered via eigenvalue–eigenvector decomposition of Φ1,Yt,Yt−1 Φ−1Yt,Yt−1 (equation (7)). - Distribution of true labor force status Π recovered by inverting ΦYt|Ỹt (equation (8)). - True transition matrix A recovered via Aᵀ = (Bᵀ)⁻¹ ΦYt,Yt−1 B⁻¹ diag(Π)⁻¹ (equation (9)). - Estimation: - The paper uses maximum likelihood estimation (MLE) of the HMM (jointly estimates reporting-error matrix and true transition probabilities), which guarantees a local maximum and uses the entire sample. - Eigenvalue–eigenvector decomposition provides global identification but uses only a subsample (e.g., Yt+1 = employed) and estimates can vary across subsamples; MLE results are generally similar to eigen-decomposition results. - Robustness: the paper verifies the robustness of the first-order Markov assumption for three-month labor market transitions; even if the true DGP is not first-order Markov, a first-order specification can generally capture transitions for three months. - Practical issues with alternative corrections: - Some correction methods can imply flow probabilities exceeding 100 percent or below 0 percent; the HMM with first-order Markov constraint keeps probabilities in [0,100]. ### Data and scope - The analysis uses U.S. Current Population Survey (CPS) data with three-period observed labor force statuses; the methodology applies to any labor force survey with three-period observations. - Comparisons cited: - Abowd and Zellner (1985) and Poterba and Summers (1986) reinterview matrices (tables shown for 1977:1-1982:4). - Feng and Hu (2013) classification-error estimates (January 1996–December 2009) indicating an approximate 2.1 percentage points understatement of the reported unemployment rate between 1996 and 2011. ### Policy recommendations and takeaways - Policymakers should: - Pay closer attention to the job separation margin when designing policies to mitigate fluctuations in the unemployment rate. - Place less emphasis on the nonparticipation (participation) margin than previous literature suggested. - Measurement correction matters: unemployment levels and flow contributions change materially once classification errors are accounted for; multiple correction methods should be considered, with attention to ensuring adjusted probabilities remain within feasible bounds. ### Additional source information (from the source text) - JEL Classification Numbers: E24, E32, J32, J6 - Keywords: Gross Worker Flows, Current Population Survey, Classification Errors, Unemployment - Author’s E-Mail Address: ishibata@imf.org - Prior circulation title: “Reassessing Classification Errors in the Analysis of Labor Market Dynamics.” *Source: wpiea2019040 (PDF).* ### 3.1 Costs and Benefits of First Order Markov Assumption ### 3.1 Costs and Benefits of First Order Markov Assumption ### Rationale and benefits of the first order Markov assumption - Aggregate labor market indicators (unemployment rate, employment rate) are stocks at a point in time; their evolution can be analyzed through gross flows between employment (E), unemployment (U), and nonparticipant (O) over two consecutive months, which is fully consistent with the first order Markov assumption for labor flows. - Calculating Pr(Y_{t+1}=j | Y_t=i) for i,j ∈ {E,U,O} using two consecutive months is equivalent to estimating first order Markov transition probabilities (e.g., job finding probability = proportion of currently unemployed who find a job next month). - Benefits highlighted: - Consistency with the bulk of macro labor literature (explicitly or implicitly used by Blanchard and Diamond 1990, Mortensen and Pissarides 1994, Shimer 2005, Shimer 2008, Shimer 2012, Elsby, Hobijn, and Sahin 2015). - Enables direct comparison with previous studies (e.g., Shimer 2012, Elsby et al. 2015). - Imposes bounds on transition probabilities (between 0 and 100 percent), avoiding implausible adjusted probabilities that can arise from reporting-error estimates (e.g., some Feng and Hu (2013) adjustments imply probabilities below 0 or above 100 percent). - Avoids additional attrition and matching-error sources inherent in linking CPS over a 9-month span (matching over first four consecutive months has matching rates around 90 percent of eligible rotation groups; matching at 12th survey drops to approximately 65 percent), since this paper uses three months of labor force history. ### Costs and limitations of the first order Markov assumption - Micro labor literature shows labor market transitions are duration dependent and heterogeneous (e.g., long-term unemployed have lower reemployment chances than recent unemployed), so individual-level data are inconsistent with first order Markov. - The first order Markov assumption imposes homogeneity and ignores duration dependence; it can thus misrepresent transition dynamics at the micro level. - Despite this, for short periods the first-order Markov approximation can be innocuous; the paper conducts robustness checks to assess validity. ### Robustness check summary (as described) - Author tests validity of the first order Markov assumption (Assumption 2) in Appendix by: - Assuming the data generating process is a second order Markov process. - Estimating the model using simulated data. - Showing that the distribution of true labor force history and measurement error matrix can generally be recovered. - Allowing transition probabilities for labor force states to vary over time while keeping reporting error matrices fixed. ### Practical motivation vs alternative approaches - The paper chooses first order Markov transitions over Feng and Hu (2013) linking approach to avoid attrition-related errors when matching across longer CPS rotations. - Truncating adjusted probabilities that fall outside [0,1] (from reporting-error corrections) can produce absorbing states and undefined average durations; the first order Markov constraint helps avoid these pathologies. --- ### Empirical approach and data (summary) - Data: Current Population Survey (CPS) January 1996 to September 2018. - CPS interviews approximately 60,000 households and 100,000 to 150,000 individuals monthly. - Households interviewed 4 consecutive months, rotated out 8 months, then interviewed for final 4 months → allows tracking up to 8 months over 16 months. - Matching procedure: - Match CPS monthly files for three consecutive months using Madrian and Lefgren (2000) algorithm (matching period t and t+k for k ∈ {1,2}) on household identifier, household replacement number, personal identifier, sex, age (allowed to differ by 1), and race. - Eligible rotation groups for three months of history ≈ rotation groups 1, 2, 5, and 6 → about one-half of initial survey sample can be matched; attrition/coding errors further reduce matches. - Re-weighting of matched sample: - To correct nonrandom attrition, sample weights for matched individuals are multiplied by inverse of predicted matching probability conditional on gender, age, race, education, and initial employment status using a logit model. - Results reported as robust to the re-weighting procedure. --- ### Key quantitative findings (reporting errors, adjusted rates, transition probabilities, durations) - Reporting error patterns: - Misreporting is most severe among the unemployed: unemployed workers are more likely to misreport as out of the labor force than as employed in all four methods compared (HMM, FH, AZ, PS). - Feng and Hu (2013) (FH) find close to 40 percent of unemployed misreport their status, which is much larger than estimates from HMM, AZ, and PS. - HMM estimates lie between FH and AZ/PS and are generally similar to historical classification error estimates in Abowd and Zellner (1985) and Poterba and Summers (1986). - Adjusted unemployment rates: - FH-adjusted unemployment rate is on average 2.6 percentage points higher than the reported official unemployment rate. - HMM-adjusted unemployment rate is about 0.8 percentage points higher than the reported (unadjusted) unemployment rate. - PS and AZ adjusted series are 0.24 and 0.54 percentage points higher than the reported unemployment rate on average, respectively. - In relative terms: FH > HMM > AZ/PS > reported unadjusted. - Selected entries from Table 4 (misclassification probabilities, transition probabilities, average durations) - Misclassification probabilities (Pr(j | \tilde{i}) = Pr(Y_t=j | \tilde{Y}_t=i), in percentage points): - Pr(E | \tilde{E}): AZ 98.8, PS 97.7, FH 97.9, HMM 98.7 (0.06) - Pr(U | \tilde{E}): AZ 0.2, PS 0.5, FH 0.6, HMM 0.4 (0.04) - Pr(O | \tilde{E}): AZ 1.0, PS 1.7, FH 1.5, HMM 0.9 (0.04) - Pr(E | \tilde{U}): AZ 1.9, PS 3.8, FH 17.3, HMM 7.5 (0.65) - Pr(U | \tilde{U}): AZ 88.6, PS 84.8, FH 62.5, HMM 76.4 (0.98) - Pr(O | \tilde{U}): AZ 9.5, PS 11.5, FH 20.2, HMM 16.1 (0.84) - Pr(E | \tilde{O}): AZ 0.5, PS 1.2, FH 2.9, HMM 1.4 (0.09) - Pr(U | \tilde{O}): AZ 0.3, PS 0.6, FH 0.2, HMM 1.0 (0.10) - Pr(O | \tilde{O}): AZ 99.2, PS 98.2, FH 96.9, HMM 97.6 (0.13) - Note: Bootstrap standard errors based on 500 simulations are reported in parentheses for HMM entries shown. - Average transition probabilities (Pr(j | i) = Pr(Y_t=j | Y_{t-1}=i), in percentage points): - Pr(E | E): R* 95.8, AZ 97.2, PS 98.6, FH 100.5, HMM 98.3 - Pr(U | E): R* 1.3, AZ 1.2, PS 0.7, FH -0.4, HMM 0.7 - Pr(O | E): R* 2.9, AZ 1.6, PS 0.6, FH -0.2, HMM 1.0 - Pr(E | U): R* 24.7, AZ 21.9, PS 16.0, FH 0.8, HMM 15.3 - Pr(U | U): R* 50.7, AZ 60.3, PS 69.0, FH 91.4, HMM 77.7 - Pr(O | U): R* 24.5, AZ 17.8, PS 15.0, FH 7.7, HMM 7.0 - Pr(E | O): R* 5.3, AZ 2.5, PS 0.3, FH -1.4, HMM 1.4 - Pr(U | O): R* 3.1, AZ 2.3, PS 1.8, FH 1.0, HMM 0.9 - Pr(O | O): R* 91.6, AZ 95.1, PS 97.8, FH 100.4, HMM 97.7 - Average duration (in months): - E: R* 23.5, AZ 35.9, PS 73.9, FH -193.2, HMM 58.2 - U: R* 2.0, AZ 2.5, PS 3.2, FH 11.7, HMM 4.5 - O: R* 12.0, AZ 20.6, PS 46.1, FH -242.1, HMM 43.8 - Note: R* denotes reported transition probabilities without misclassification corrections; negative or extreme FH durations illustrate the problem when corrected transition matrices imply absorbing states or implausible durations. --- ### Adjustment methodology for flows (equations and interpretation) - Notation: - ˜N_t: matrix of true labor force flows; (i,j) element = ˜N_{ij,t} = true flows from i to j for i,j ∈ {E,U,O}. - N_t: reported flows matrix; (k,l) element = N_{kl,t} = reported flows from k to l. - B^T: reporting error matrix with entries b_{ij} = Pr(Y=j | ˜Y=i) (probability that a worker whose true status is i reports being in status j). - Relationship between reported and true flows: - Equation (10): N_{kl} = Σ_{i=1}^3 Σ_{j=1}^3 b_{ik} b_{jl} ˜N_{ij} for k,l ∈ {E,U,O}. - Matrix form (11): N_t = B^T ˜N_t B. - If B is invertible, true flows can be recovered (12): ˜N_t = (B^T)^{-1} N_t (B)^{-1}. - Transition probabilities derived from true flows: - ˜P_{ij,t} = ˜N_{ij,t} / Σ_{j=1}^3 ˜N_{ij,t} for i,j ∈ {E,U,O}. - Important caution: - Nothing prevents ˜N from being negative after inversion; negative implied transition probabilities have been observed (e.g., Poterba and Summers (1986) male classification matrix, FH estimates occasionally produce negative implied transition probabilities when applied to 1996–2018 data). - Negative or >100 percent implied probabilities necessitate careful treatment (truncation leads to absorbing-state issues and undefined average durations), motivating the preference for a first order Markov assumption and a latent/HMM approach that yields plausible bounded estimates. --- *Source: Section 3.1 (Costs and Benefits of First Order Markov Assumption) and related excerpts from the provided content unit.* ### 5.4 Adjusted Transition Probabilities ### 5.4 Adjusted Transition Probabilities ### Adjusted transition probabilities — estimation and comparison - Using the adjustment procedures, the paper plots unadjusted and adjusted transition probabilities based on reporting error probabilities by AZ, PS, FH, and the paper’s HMM estimates. - Reporting errors by Feng and Hu (2013) imply negative transition probabilities for some series; probabilities from Feng and Hu (2013) and Poterba and Summers (1986) in some periods fall below 0 or above 100 and thus cannot be compared in subsequent analysis. - The HMM imposes the first order Markov assumption and jointly estimates time varying true labor force transition probabilities, ensuring transition probabilities stay positive and do not exceed 100 percent. - Comparison of estimates: - Probabilities of staying in the same labor force status (Pr(˜Y_{t+1}=i | ˜Y_t=i) for i=1,2,3): - All HMM estimates are higher than corresponding unadjusted and AZ probabilities. - HMM estimates are smaller than FH estimates. - HMM estimates are very similar to PS estimates. - Probability of transitioning from employment to unemployment: - HMM average: 0.7 percent. - HMM is smaller than unadjusted and AZ. - Probability of transitioning from unemployment to employment (job finding probability): - HMM average: around 15 percent. - Unadjusted: 25 percent. - AZ: 22 percent. - PS: 16 percent. - Overall: HMM adjustment revises unadjusted transition probabilities in the same direction as AZ, PS, and FH; degree of adjustment is in general greater than AZ and PS, but smaller than FH. - Measurement-error correction detail: - Under the HMM adjustment, truly unemployed workers have a high probability of incorrectly reporting to be nonparticipant: Pr(Y_t = O | ˜Y_t = U) = 16.1. ### Contribution to unemployment fluctuations — framework and main findings - Continuous-time labor market system governing transitions (notation preserved): - Ė, U̇, Ȯ vector dynamics governed by matrix of λ_{ij} with E+U+O = 1 and λ_{ii} = −∑_{j≠i} λ_{ij}. - Reduced system for s_t ≡ (E,U)' uses ̃F_t and w_t as in equations (14)–(16); steady state ̄s = ̃F^{-1} w. - Method: compute implied steady state unemployment rate and employment-population ratio as functions of transition rates λ_{ij}; quantify contribution of one transition rate by letting it follow its time series while fixing others at their historical average. Results summarized in Table 5. - Key qualitative conclusions: - Consistent with Shimer (2012): fluctuations in job finding rate λ_{UE} explain more fluctuation in the unemployment rate than job separation rate λ_{EU} — "Outs win" — but HMM and AZ adjustments moderate that dominance. - HMM increases the role of "Ins" relative to unadjusted series: for 1996-2007 HMM reports λ_{UE} = 0.53 and λ_{EU} = 0.30, while unadjusted reports λ_{EU} = 0.2. - The fraction of “In” computed as 100× λ_{EU}/(λ_{UE}+λ_{EU}): - without adjustment: 100× 0.2/(0.53+0.2) ≈27.4percent - with HMM adjustment: 100× 0.3/(0.53+0.3) ≈36.1percent - The fraction of employment-population ratio fluctuation explained by λ_{UE}: - Unadjusted: 0.49 - AZ-adjusted: 0.39 - HMM-adjusted (1996-2007): 0.21 - The fraction explained by λ_{EU} rises from 0.14 (no adjustment) to 0.16 (HMM) for 1996-2007. - Elsby, Hobijn, and Sahin (2015) estimate participation margin (λ_{UO}+λ_{OU}) explains approximately 26-36 percent of unemployment dynamics: - This paper finds ~25 percent for unadjusted series. - HMM-adjusted transition rates suggest the participation margin explains approximately 10 percent of unemployment fluctuations. - Interpretation: HMM mutes cyclicality of nonparticipant-to-unemployment (O-U) and unemployment-to-nonparticipation (U-O) transitions, because many reported nonparticipants are corrected to unemployed via error-correction. - Policy-relevance summary: - If "Ins" (job separations) explain a larger share of unemployment fluctuations, policy could focus on discouraging firings (tax incentives) or encouraging adjustment via hours rather than employment headcount. - If "Outs" (job finding) dominate, policy emphasis should be on active labor market policies: training programs, job-search assistance, direct job creation. - Paper’s implication: Outs win but Ins also matter — policymakers should not discard the job separation margin. ### Table 5 — Contributions of transitions to the UR and EP Ratio 1996-2018 (values preserved) - Table layout: columns correspond to Unemployment Rate (1996-2007 and 1996-2018) and Employment-Population Ratio (1996-2007 and 1996-2018); reported series R (no adjustment), AZ (Abowd and Zellner), and HMM. - Entries (λ_{ij} contributions): - λ_{EU}: - Unemployment Rate 1996-2007: R 0.20, AZ 0.24, HMM 0.30 - Unemployment Rate 1996-2018: R 0.20, AZ 0.25, HMM 0.27 - Employment-Population Ratio 1996-2007: R 0.14, AZ 0.11, HMM 0.16 - Employment-Population Ratio 1996-2018: R 0.22, AZ 0.18, HMM 0.12 - λ_{EO}: - Unemployment Rate 1996-2007: R -0.07, AZ -0.11, HMM -0.08 - Unemployment Rate 1996-2018: R -0.05, AZ -0.09, HMM -0.09 - Employment-Population Ratio 1996-2007: R -0.46, AZ -0.47, HMM 0.11 - Employment-Population Ratio 1996-2018: R -0.30, AZ -0.36, HMM -0.05 - λ_{UE}: - Unemployment Rate 1996-2007: R 0.53, AZ 0.57, HMM 0.53 - Unemployment Rate 1996-2018: R 0.46, AZ 0.50, HMM 0.52 - Employment-Population Ratio 1996-2007: R 0.49, AZ 0.39, HMM 0.21 - Employment-Population Ratio 1996-2018: R 0.59, AZ 0.50, HMM 0.23 - λ_{UO}: - Unemployment Rate 1996-2007: R 0.14, AZ 0.12, HMM 0.09 - Unemployment Rate 1996-2018: R 0.12, AZ 0.11, HMM 0.07 - Employment-Population Ratio 1996-2007: R -0.14, AZ -0.15, HMM -0.07 - Employment-Population Ratio 1996-2018: R -0.17, AZ -0.2, HMM -0.04 - λ_{OE}: - Unemployment Rate 1996-2007: R 0.10, AZ 0.16, HMM 0.14 - Unemployment Rate 1996-2018: R 0.09, AZ 0.13, HMM 0.13 - Employment-Population Ratio 1996-2007: R 0.87, AZ 0.89, HMM 0.42 - Employment-Population Ratio 1996-2018: R 0.65, AZ 0.68, HMM 0.45 - λ_{OU}: - Unemployment Rate 1996-2007: R 0.10, AZ 0.01, HMM 0.00 - Unemployment Rate 1996-2018: R 0.15, AZ 0.06, HMM 0.04 - Employment-Population Ratio 1996-2007: R 0.13, AZ 0.28, HMM 0.23 - Employment-Population Ratio 1996-2018: R -0.05, AZ 0.14, HMM 0.19 - λ_{UE} + λ_{EU}: - Unemployment Rate 1996-2007: R 0.73, AZ 0.81, HMM 0.83 - Unemployment Rate 1996-2018: R 0.66, AZ 0.75, HMM 0.79 - Employment-Population Ratio 1996-2007: R 0.63, AZ 0.50, HMM 0.37 - Employment-Population Ratio 1996-2018: R 0.81, AZ 0.68, HMM 0.35 - λ_{OU} + λ_{UO}: - Unemployment Rate 1996-2007: R 0.24, AZ 0.13, HMM 0.09 - Unemployment Rate 1996-2018: R 0.27, AZ 0.17, HMM 0.11 - λ_{EO} + λ_{OE}: - Unemployment Rate 1996-2007: R 0.41, AZ 0.42, HMM 0.53 - Unemployment Rate 1996-2018: R 0.35, AZ 0.32, HMM 0.40 ### Conclusion — implications for measurement and policy - Reassessment of classification errors in the CPS: - Results still support that job finding probability ("outs") plays a more important role than job separation probability ("ins") in explaining unemployment fluctuations, but HMM adjustments moderate the dominance of "Outs". - Importance of the participation margin in explaining unemployment fluctuations falls under HMM adjustment to around 10 percent (this paper) versus previous estimates of 20 to 30 percent (Elsby, Hobijn and Sahin, 2015). - Policy implication distilled: - Policymakers should pay closer attention to the job separation margin than previously thought and less on the participation margin, while recognizing that both ins and outs matter. *Source: “5.4 Adjusted Transition Probabilities” (wpiea2019040).* ### Appendix 3: Validity of First Order Markov Assumptions ### Appendix 3: Validity of First Order Markov Assumptions ### Purpose and approach - Goal: test validity of the first order Markov (FOM) assumption used in the misclassification correction model to measure workers’ flows between employment (E), unemployment (U), and nonparticipation (O) over month t and t+1 in the Current Population Survey. - Sensitivity analysis: simulate a data generating process (DGP) that is a second-order Markov process and estimate a hidden Markov model (HMM) imposing a first-order Markov assumption to assess robustness. ### Simulation design — DGP: second-order Markov process - Assumed DGP: Pr(Ỹt | Ỹt−1, Ỹt−2, Ỹ̄t) = Pr(Ỹt | Ỹt−1, Ỹt−2) for all ̄t ≠ t, t−1. - Defined objects: - Initial distribution Ξ: Ξij ≡ Pr(Ỹt−1 = j, Ỹt−2 = i) for i,j ∈ {E,U,O}. - SecondOrderMarkovTransitionMatrix Ψ: ψijk ≡ Pr(Ỹt = k | Ỹt−1 = j, Ỹt−2 = i). - MeasurementErrorMatrix B_DGP: bDGP_ij = Pr(Yt = j | Ỹt = i). - Data generation steps: 1. Draw (Ỹt−1, Ỹt−2) from Ξ. 2. Draw Ỹt using ψijk conditional on (Ỹt−1, Ỹt−2). 3. Draw observed (Yt, Yt−1, Yt−2) using B_DGP and true (Ỹt, Ỹt−1, Ỹt−2). 4. Repeat for N times with iid sampling; used N = 60000 observations. - Example second-order transition matrix Ψ (DGP) reported as percentages (rows indexed by (Yt−1, Yt); columns Yt+1 = E, U, O): - EE: 98.3 0.7 1.0 - EU: 13.7 79.3 7.0 - EO: 1.8 0.8 97.4 - UE: 97.8 1.1 1.1 - UU: 12.4 81.5 6.1 - UO: 2.5 0.0 97.5 - OE: 98.1 0.2 1.7 - OU: 14.6 78.9 6.5 - OO: 1.4 0.9 97.6 ### Estimation and main results - Estimated an HMM (first-order Markov assumption) on the simulated observed data and compared estimated measurement error matrix B versus true B_DGP. - Table 8 key misclassification probabilities (In percentage points): - B_DGP vs B Estimate (95% CI in brackets) - Pr(E | Ỹ = E): 98.7 vs 98.7 [98.4,99.0] - Pr(U | Ỹ = E): 0.4 vs 0.5 [0.3,0.6] - Pr(O | Ỹ = E): 0.9 vs 0.9 [0.7,1.1] - Pr(E | Ỹ = U): 7.0 vs 6.2 [3.2,9.1] - Pr(U | Ỹ = U): 77.8 vs 78.1 [73.6,82.6] - Pr(O | Ỹ = U): 15.2 vs 15.8 [11.4,20.2] - Pr(E | Ỹ = O): 1.4 vs 1.3 [0.9,1.7] - Pr(U | Ỹ = O): 1.0 vs 1.0 [0.5,1.5] - Pr(O | Ỹ = O): 97.5 vs 97.7 [97,98.3] - Implied first-order transition probabilities for true labor force status (columns labeled Pr(Ỹj | Ỹi)) (In percentage points): - Pr(Ỹ = E | Ỹ = E): 98.3 vs 98.3 [97.9,98.7] - Pr(Ỹ = U | Ỹ = E): 0.8 vs 0.6 [0.4,0.9] - Pr(Ỹ = O | Ỹ = E): 1.0 vs 1.1 [0.8,1.4] - Pr(Ỹ = E | Ỹ = U): 13.9 vs 12.8 [8.8,16.8] - Pr(Ỹ = U | Ỹ = U): 79.5 vs 80.5 [74.8,86.3] - Pr(Ỹ = O | Ỹ = U): 6.6 vs 6.7 [2,11.4] - Pr(Ỹ = E | Ỹ = O): 1.4 vs 1.6 [0.9,2.2] - Pr(Ỹ = U | Ỹ = O): 0.9 vs 1.0 [0.2,1.7] - Pr(Ỹ = O | Ỹ = O): 97.7 vs 97.5 [96.5,98.4] - Confidence intervals: Bootstrap Confidence Interval based on 500 simulations reported in []. - Accuracy in recovering true labor force history distributions: - Weighted mean absolute error (WMAE) between true underlying distribution Pr(Ỹt+1 = k, Ỹt = j, Ỹt−1 = i)DGP and estimated Pr(·)HMM is 0.0014 percentage points. - Average probability of a possible employment path cited as 3.7 percent (used as reference for relative magnitude). ### Sensitivity: deviations in B_DGP (B_DGP_max and B_DGP_min) - Construction: - B_DGP_max: set off-diagonal elements to +2 standard deviations of those in B_DGP (standard errors from HMM estimates on simulated data). - B_DGP_min: set off-diagonal elements to −2 standard deviations of those in B_DGP. - Results (selected findings): - For B_DGP_max, true parameters are within the 95% confidence interval of estimated B_Estimate_max in many cases, though some misclassification probabilities in B_DGP_max are close to but outside the 95% CI of B_Estimate_max. - For B_DGP_min, comparable comparisons reported in Table 10 show similar patterns. - WMAE across all 27 possible labor force history transitions: - WMAE for B_DGP: 0.0014 percentage points (relative to average probability 3.7 percent). - WMAE for max case: 0.001 percentage points. - WMAE for min case: 0.06 percentage points. - Implication: estimated first-order transition probabilities for the true underlying labor force status capture the true transition probabilities implied by the second-order DGP; transitions in the nonparticipation margin (O) could be slightly less robust. ### Conclusion - The first order Markov assumption for the true labor force status is generally robust for the purpose of the flow analysis and misclassification correction: - Estimated measurement error matrix B is close to true B_DGP and often contains true parameters within 95% bootstrap CIs. - Distribution of true labor force history implied by the HMM closely matches the DGP (very small WMAE values). - Deviations in B_DGP (±2 standard deviations) do not materially change the conclusion, though some specific misclassification probabilities—especially involving nonparticipation—can be less precisely recovered. *Source: Appendix 3: Validity of First Order Markov Assumptions (simulation results, tables, and analysis).* --- _Source: https://www.imf.org/-/media/files/publications/wp/2019/wpiea2019040.pdf_