## _wp11151 - section 5. Section 6 summarizes the main ideas of the paper.

## Source details

**Canonical URL:** [_wp11151 - section 5. Section 6 summarizes the main ideas of the paper.](https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2011/_wp11151.pdf)

## Other formats

- [Markdown version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2011/_wp11151.pdf.md)
- [Structured JSON version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2011/_wp11151.pdf.json)

---

### II. Data structures and missing-data patterns (major findings)
- Survey (micro) data vs. time series: differ in origination, structure, interpretation of missingness, and applicability of standard (survey) missing-data techniques.
- Survey micro data: observation-by-variable for one time period; missingness types:
  - Item non-response: one or multiple variables missing for one or multiple observations.
  - Unit non-response: all variables missing for one or multiple observations.
  - Variables not included in the survey: missing for all observations.
- Time series (usually macro data): multiple time periods for one or multiple aggregate observation units (sections). In international statistics sections are most frequently countries. Variables are statistical indicators (e.g., unemployment rate, current account balance, GDP).
- Types of time-series data structures listed:
  - single univariate time series
  - single multivariate time series
  - cross-sectional univariate time series
  - cross-sectional multivariate time series (involves three non-constant dimensions: section, variable, time).
- Relevant missing-pattern types for multivariate cross-sectional time series:
  - Missing items
  - Missing periods
  - Missing variables
  - Missing sections
- Special case: mixed frequencies (e.g., some variables quarterly, others annual) produce systematic missing periods across variables or sections.

### Missing-data mechanisms (key distinctions)
- MCAR (missing completely at random): missingness neither depends on the true (missing) value nor on other observed or non-observed characteristics; missing-data process is ignorable for imputation.
- MAR (missing at random): missingness is independent of the true (missing) value after controlling for other variables; missingness only depends on other variables that can be included in the imputation.
- MNAR (missing not at random): missingness is systematic/informative; distributions among complete and missing observations differ (selection bias). For MNAR the missing-data mechanism cannot be ignored and must be modeled (e.g., jointly model the variable and the probability of missingness) or analyzed via multiple imputation under alternative missing-data scenarios.

### General guidance on imputation vs. deletion
- Imputation: estimate missing/implausible values using collected values or comparable sources; focus on partially missing data due to item non-response.
- List-wise deletion: excludes observation units with any missing values; typically produces large information loss and biased estimates unless MCAR holds.
- Official-statistics perspective: removing incomplete sections/variables/time series is usually not acceptable; some producers may avoid publishing imputed statistics, transferring the challenge to users.

### A. Traditional single-imputation and donor/model approaches (findings and properties)
- Deterministic imputation:
  - Replaces missing values by ad-hoc specified values (e.g., subject-matter expert rules, "Carry Last Value Forward").
  - No statistical principle; distorts marginal and joint distributions; may be acceptable if only a few values are missing.
  - Mirror-value substitution example noted.
- Location-based imputation:
  - Replace missing values with a distribution location parameter (mean, median, mode) computed overall or within subgroups (post-stratification).
  - Risks: high peaks at imputed values, reduced variability, potentially unfeasible values; moving averages or linear interpolation (neighbor count = 2) are time-series variants.
- Distribution-based imputation:
  - Draw imputed values randomly from empirical or parametric distributions estimated from observed values; subgrouping possible.
  - Iterative EM-based procedures are more sophisticated variants.
- Model-based imputation:
  - Use correlations and predictive models (linear, logit/probit for categorical, autoregression for time series).
  - Include as many predictive covariates as available; tends to smooth data but better preserves distributions and reduces bias in aggregates; can account for external shocks.
- Donor-based imputation (hot-/cold-deck, nearest-neighbor):
  - Donor = complete observation similar to recipient based on matching variables.
  - Hot-deck: donors from same dataset; cold-deck: donors from other sources.
  - Selection methods: sequential, random, distance-based, rank-based; random selection common.
  - Nearest-neighbor and k-nearest methods, multi-donor aggregation (weighted average/median) are options.
  - Pros: produce realistic observed-value imputations; cons: subjective choices and potential scarcity of donors when many matching variables are used.

### B. Statistical matching (donor-based imputation)
- Purpose: enrich a recipient dataset with variables available only in a donor dataset by matching on common variables.
- Matching variants:
  - Constrained matching: every recipient and donor included with original sample weights; requires identity of weighted population totals.
  - Unconstrained matching: does not impose weighted-total identity.
  - Equivalence-class matching: subdivide datasets into comparable subsets and match within subsets (cold-deck analogue).
  - Regression-based matching: estimate additional variables in both datasets and match using Mahalanobis distance.
  - Propensity-score matching: estimate probability of being in donor vs. recipient dataset via logistic regression on common variables; match recipients to nearest donor in propensity score.

### C. Multiple imputation and related iterative/simulation methods
- Multiple imputation (Rubin):
  - Impute multiple (usually about five) values per missing cell drawn from an estimate of the posterior distribution, producing multiple completed datasets.
  - Analyze each completed dataset with standard complete-data methods and combine results to incorporate missing-data uncertainty into inference (intervals, p-values).
  - Goal: valid statistical inference, not optimal point prediction of individual missing values.
- Jackknife/bootstrap aggregation approach:
  - Generate K jackknife or bootstrap samples, impute missing values in samples, aggregate imputed values to obtain one imputed value per missing cell.
  - Jackknife: K often equals the number of observations; bootstrap: sampling with replacement.
  - Multiple imputation considered superior to this resampling approach by Rubin (1996).
- EM algorithm and related:
  - EM alternates expectation and maximization until convergence.
  - Data augmentation combines multiple imputation and EM; Gibbs sampling and other Bayesian simulation methods relate to multiple imputation and EM.

### Applicability of techniques to time-series missing patterns (practical guidance)
- Missing-pattern relevance: (i) missing items, (ii) missing periods, (iii) missing variables, (iv) missing sections.
- Mapping of techniques to patterns:
  - Missing items: all missing-data techniques applicable; can be applied variable-wise or inter-variable; section-wise or across-section treatments possible.
  - Missing periods:
    - Single time series: variable-wise or combined approaches.
    - Cross-sectional univariate time series: section-wise or combined section-/cross-sectional approaches.
    - Multivariate cross-sectional time series: variable-wise, section-wise, inter-variable, cross-sectional, or combinations.
  - Missing variables and/or sections: require additional data sources because time is the only shared dimension.
- Technique-specific caveats:
  - List-wise deletion: acceptable only if MCAR; using lower-frequency data only may lose seasonal effects but preserve long-term trends.
  - Deterministic imputation: shares ad-hoc drawbacks and potential bias.
  - Location-based imputation: caution for time series; moving averages and interpolations (e.g., splines) are alternatives but may smooth away peaks and fail for gaps > one period.
  - Distribution-based imputation: requires conditional distributions incorporating lagged variables; effectively becomes model-based.
  - Model-based methods: most frequently adopted for incomplete time series; include autoregressive/moving-average models, state-space models, curve fitting/smoothing, Kalman filters, dynamic Bayesian networks; alternative approaches include neural networks, k-means clustering, seasonal pattern recognition, genetic algorithms.
  - Donor-based/statistical matching for time series:
    - Variable-wise donor selection uses similarity of trajectories (“time series donor”).
    - Inter-variable donor selection uses similarity of additional characteristics (“cross-sectional donor”).
  - Multiple imputation: applicable for all missing patterns in time series; tailored approaches for cross-sectional time series and non-parametric curve fitting exist.
  - Hybrid procedures: common practice; methods should be selected and combined according to the missing-data pattern.

### Evaluation and quality monitoring (principles and indicators)
- Five principal aspects to be reported when publishing analyses based on imputed data:
  - 1. the extent to which imputation was required in a dataset,
  - 2. the performance of the applied method,
  - 3. the accuracy of the imputation results,
  - 4. the variability of statistics based on the imputed dataset, and
  - 5. the plausibility of imputed values.
- Reporting levels: overall dataset, variable-specific, each imputed value; for time series additionally section- and period-specific indicators.
- Plausibility assessment: editing procedures may be applied to the completed dataset; multiple rounds of imputation and plausibility checks may be required; plausibility checks can be integrated in the imputation process.
- Thresholds and comparative usage: no generally valid thresholds; comparative usage recommended (e.g., repeated measures analysis of (co-)variance with imputation procedure as factor; if significant overall difference, pairwise comparisons via t-tests can create a quality ranking).
- Time-series-specific checks: analyze smoothness of an imputed variable (per section) over time to identify implausible peaks introduced by imputation; smoothness checks can use confidence bands of moving averages or kernel estimates.

### A. Degree of missingness (recommended indicators)
- Purpose: measure dataset completeness prior to imputation and quantify reduction after imputation.
- Empirical impacts: increasing missing values per observation or proportion of incomplete observations severely affects (i) the quality of missing value estimates, (ii) the magnitude of discrepancy in means, and (iii) preservation of the covariance structure; increasing sample size can reduce adverse effects.
- Example indicators (ratios and distributions):
  - the number of observations with one or multiple missing values as a ratio of the total number of observations.
  - the frequency distribution of the number of variables to be imputed per observation (section) and appropriate distribution measures (min, max, median).
  - the frequency distribution of the number of periods to be imputed per variable (and/or section) and appropriate distribution measures (applies to time series).
  - the proportion of observations (or sections or periods) missing for specific variables (or sections or periods).
  - the total number of missing values as a ratio of the total number of cells of the dataset.
- For cross-sectional time series, weighted missingness rates can be used (e.g., weights such as population size, GDP, or purchasing power parity).
- Analyze missingness patterns using contingency or correlation matrices to assess interdependency between missingness patterns.

### B. Performance of imputation method (metrics)
- Model-based methods: use regression model quality criteria such as coefficient of determination (R²), information criteria (e.g., Akaike or Bayes), p-values; indicators can be variable-/section-specific and aggregated.
- Donor-based methods: compute distribution parameters of donor usage frequency or values of the distance function between donor and recipient; interpret quality as less donor reuse and closer donor-recipient distance.
- Regression-based and propensity-score matching: use regression quality measures and distance measures between donor and recipient; location and variability measures of the distance are used.
- Multiple imputation: performance indicators of the model estimated for the posterior distribution are relevant (e.g., R² or information criteria); measures of variability of multiple imputed values are performance measures.
- Individual imputed value reporting:
  - Model-based/multiple imputation/regression-based matching: report the standard error or confidence interval for the estimated (=imputed) value.
  - Donor-based methods: provide a reference to the donor used; for distance-based approaches provide the value of the distance function and the number of times the same donor was used.
- Hybrid approaches: provide performance measures of component methods; aggregation into one “hybrid” performance indicator is appealing but usually hard to interpret.

### C. Accuracy of imputation results (typology and validation)
- Four types of imputation accuracy (Chambers, 2000):
  - predictive accuracy or effectiveness: maximal preservation of true values,
  - ranking accuracy: maximal preservation of true ordering relationship in imputed values,
  - distributional accuracy: maximal preservation of (marginal and higher order) distributions of true values,
  - estimation accuracy: maximal preservation of analytic results and conclusions.
- Typology hierarchy: predictive accuracy implies the other three types.
- Intended usage matters:
  - Public release or prediction-model development: predictive and ranking accuracy are crucial.
  - Producing/publishing aggregated estimates: predictive and ranking accuracy less important.
- Rubin’s perspective: aim of (multiple) imputation should be statistically valid inference based on the completed dataset (estimation accuracy) rather than recreation of true values (predictive accuracy).
- Validation approach: cross-validation by treating available values as missing, imputing them, and comparing imputed to true values. Common types: repeated random sub-sampling, k-fold, leaving-one-out (leaving-one-out typically used for imputation validation).
- Accuracy measures by variable type:
  - Nominal variables: proportion of off-diagonal entries in cross-classification of imputed vs. true values; dichotomous variables: area under ROC curve; p > 2 categories can be transformed to p - 1 dichotomous variables.
  - Ordinal variables: require ranking accuracy; use ordinal distance measures.
  - Metric variables: weighted Bravais-Pearson correlation; robust regression without intercept of true on imputed values; coefficient of determination; regression mean square error; ranks-based measures; Kolmogorov-Smirnov, Wilcoxon rank-sum or Mann-Whitney-U tests; dependent t-test for paired samples for mean preservation.

### Variability of statistics based on imputed dataset (bias and variance)
- Key evaluation measures: bias and variance of estimates based on completed data.
- Deriving closed-form variance and bias solutions is complex for realistic patterns and methods; theoretical work limited to simple constellations.
- Neglecting imputation effects leads to underestimated variances and jeopardized validity of confidence statements.
- Recommended approach for complex procedures: simulation.
- Multiple imputation: routine production of several sets of imputed values under different models/assumptions to estimate imputation error and investigate model effects.
- Bootstrap approach (Shao and Sitter, 1996): generate bootstrap samples, apply imputation in each, use distribution of bootstrap estimates for inference; contrasted with multiple imputation in sampling-with-replacement vs. posterior sampling without replacement.

### Conclusion: practice and policy recommendations
- Provide methodological choices for imputation of missing data in cross-country time series and investigate data structures and missingness patterns.
- Offer evaluation procedures based on statistical quality criteria for assessing imputation outcomes.
- Raise awareness of data producers and analysts in international financial institutions about challenges posed by missing data and techniques for handling them.
- Promote usage of sound imputation procedures and stress the importance of accompanying any imputation process with reasonable quality indicators.
- Recommendation: publish a description of the applied missing data technique and quality assessment results together with the data to reduce ambiguity of incomplete data and facilitate analysis and interpretation for advising policy makers.

*Source: _wp11151 - section 5. Section 6 summarizes the main ideas of the paper.*

### section 5. Section 6 summarizes the main ideas of the paper.

### _wp11151 - section 5. Section 6 summarizes the main ideas of the paper.

### II. DATA STRUCTURES AND MISSING DATA PATTERNS
- Missing data techniques commonly used in official statistics focus on filling gaps in survey data.
- Survey data are micro data: multiple variables observed or measured for a sample of observation units from a population at one point in time.
- Gaps in survey data can be classified as:
  - Item non-response: one or multiple variables missing for one or multiple observations. The variables (= items) missing may vary between observations. Item non-response can be dealt with by traditional or multiple imputation and statistical matching (see Section 3).
  - Unit non-response: all variables are missing for one or multiple observations (no data available at all for the respective observation units). Unit non-response is often accounted for by weighting algorithms (not discussed here, see e.g. Little, 1982; Holt, Smith, 1979; Cochran, 1977).
  - Variables not included in the survey: missing for all observations. If available from other data sources, these variables can be added by statistical matching (see Section 3) or record linkage (not discussed here, see e.g. Fellegi, Sunter, 1969; Winkler, 1995; Denk, 2008).

- Figure 1 (illustration described in the source) represents missing data patterns for standard micro (= observation by variable) data and distinguishes:
  - 1 – Item Non-response
  - 2 – Unit Non-response
  - 3 – Missing Variables
  - one time period

- Time series data differ from micro survey data in:
  - the origination of the data,
  - the data structure,
  - the (interpretation of) missing data patterns,
  - the applicability of standard (survey) missing data techniques.

- Differences in structure and scope:
  - Survey (observation by variable) data structure holds data for one time period.
  - Time series contain data for multiple time periods for one or multiple aggregate observation units and for one or multiple observed variables (or aggregate statistical indicators).
  - Time series in official statistics are usually macro data (do not contain data for individual observation units, but for aggregate or macro units, also called sections).
  - In international statistics these aggregate observation units or sections are most frequently countries.
  - Variables in time series are usually statistical indicators such as unemployment rate, current account balance, or GDP.

### Types of time series data structures (as listed)
- 1. single univariate time series: one variable/indicator observed for one observation unit/section over time
- 2. single multivariate time series: multiple variables/indicators observed for one observation unit/section over time
- 3. cross-sectional univariate time series: one variable/indicator observed for multiple sections over time

*Source: _wp11151 - section 5. Section 6 summarizes the main ideas of the paper.*

### 4.      cross-sectional multivariate time series: multiple variables/indicators observed for

### _wp11151 - 4.      cross-sectional multivariate time series: multiple variables/indicators observed for

### Missing-data patterns and structure
- Multivariate cross-sectional time series involves three non-constant dimensions: section, variable, time; consequently all four missing data patterns and their combinations may occur.
- Relevant missing-pattern types:
  - Missing items
  - Missing periods
  - Missing variables
  - Missing sections
- Special case: datasets with mixed frequencies (e.g., some variables quarterly, others annual) produce systematic missing periods across variables or sections and frequently arise from heterogeneous statistical systems across sections.

### Missing-data mechanisms (key distinctions)
- MCAR (missing completely at random): missingness neither depends on the true (missing) value nor on other observed or non-observed characteristics; the missing data process is ignorable for imputation.
- MAR (missing at random): missingness is independent of the true (missing) value after controlling for other variables; missingness only depends on other variables that can be included in the imputation.
- MNAR (missing not at random): missingness is systematic/ informative; distributions among complete and missing observations differ (selection bias). For MNAR, the missing-data mechanism cannot be ignored and must be modeled (e.g., jointly model the variable and the probability of missingness) or analyzed via multiple imputation under alternative missing-data scenarios.

### General guidance on imputation vs. deletion
- Imputation estimates missing/implausible values using collected values or comparable sources; focus is on partially missing data due to item non-response.
- List-wise deletion (complete case analysis) excludes observation units with any missing values and typically produces large information loss and biased estimates unless MCAR holds.
- From an official-statistics perspective, removing incomplete sections/variables/time series is usually not acceptable; some producers may avoid publishing imputed statistics, transferring the challenge to users.

### A. Traditional single-imputation and donor/model approaches (findings and properties)
- Deterministic imputation:
  - Replaces missing values by ad-hoc specified values (e.g., subject-matter expert rules, "Carry Last Value Forward").
  - No statistical principle; distorts marginal and joint distributions; may be acceptable if only a few values are missing.
  - Mirror-value substitution (e.g., replace country A's missing exports to B by country B's imports from A) is an example, though modeling would be preferable.
- Location-based imputation:
  - Replace missing values with a distribution location parameter (mean, median, mode) computed overall or within subgroups (post-stratification).
  - Risks: high peaks at imputed values, reduced variability, potentially unfeasible values, poor treatment of exogenous shocks; moving averages or linear interpolation (neighbor count = 2) are time-series variants.
- Distribution-based imputation:
  - Draw imputed values randomly from empirical or parametric distributions estimated from observed values; can be subgrouped.
  - Advantage: only observable values imputed; multivariate approaches better reflect joint distributions.
  - Iterative EM-based procedures are more sophisticated variants.
- Model-based imputation:
  - Use correlations and predictive models (linear, logit/probit for categorical, autoregression for time series) to predict missing values.
  - Include as many predictive covariates as available; tends to smooth data but better preserves distributions and reduces bias in aggregates; can account for external shocks.
- Donor-based imputation (hot-/cold-deck, nearest-neighbor):
  - Donor = complete observation similar to recipient based on matching variables.
  - Hot-deck: donors from same dataset; cold-deck: donors from other sources.
  - Selection methods: sequential, random, distance-based, rank-based; random selection common.
  - Nearest-neighbor and k-nearest methods, multi-donor aggregation (weighted average/median) are options.
  - Pros: produce realistic observed-value imputations; cons: subjective choices (matching variables, distance measures) and potential scarcity of donors when many matching variables are used.

### B. Statistical matching (as donor-based imputation)
- Purpose: enrich a recipient dataset with variables available only in a donor dataset by matching on common variables.
- Constrained matching: every recipient and donor included with original sample weights; requires identity of weighted population totals in both datasets.
- Unconstrained matching: does not impose weighted-total identity.
- Equivalence-class matching: subdivide datasets into comparable subsets and match within subsets; corresponds to cold-deck imputation for completely missing variables.
- Regression-based matching: estimate additional variables in both datasets (via regressions on matching variables) and match using Mahalanobis distance; comparable to model-based imputation but uses donor values.
- Propensity-score matching: estimate probability of being in donor vs. recipient dataset via logistic regression on common variables; match recipients to nearest donor in propensity score.

### C. Multiple imputation and related iterative/simulation methods
- Multiple imputation (Rubin):
  - Impute multiple (usually about five) values per missing cell drawn from an estimate of the posterior distribution, producing multiple completed datasets.
  - Analyze each completed dataset with standard complete-data methods and combine results to incorporate missing-data uncertainty into inference (intervals, p-values).
  - Goal: valid statistical inference, not optimal point prediction of individual missing values.
  - Evidence shows multiple imputation often preferable to single imputation for inference (e.g., Heitjan and Rubin, 1990).
  - Guidance on the number of imputations: see Graham, Olchowski, Gilreath (2007); methodological developments example: Steele, Wang, Raftery (2010).
- Jackknife/bootstrap aggregation approach:
  - Generate K jackknife or bootstrap samples, impute missing values in samples, aggregate imputed values to obtain one imputed value per missing cell.
  - Jackknife: K often equals the number of observations; bootstrap: sampling with replacement.
  - Multiple imputation is considered superior to this resampling approach by Rubin (1996).
- EM algorithm and related:
  - EM alternates expectation (replace missing values by expected values given parameters) and maximization (estimate parameters maximizing log-likelihood) until convergence.
  - Data augmentation combines multiple imputation and EM; Gibbs sampling and other Bayesian simulation methods relate to multiple imputation and EM.

### Applicability of techniques to time series missing patterns
- Missing-pattern relevance: (i) missing items, (ii) missing periods, (iii) missing variables, (iv) missing sections.
- Missing items: all missing-data techniques applicable; can be applied variable-wise (temporal focus) or inter-variable (cross-variable relationships); section-wise or across-section treatments possible.
- Missing periods:
  - Single (uni- or multivariate) time series: variable-wise or combined variable-/inter-variable approaches.
  - Univariate cross-sectional time series: section-wise or combined section-/cross-sectional approaches.
  - Multivariate cross-sectional time series: variable-wise, section-wise, inter-variable, cross-sectional, or combinations of these.
- Missing variables and/or sections: require additional data sources because time is the only shared dimension.
- Technique-specific applicability and caveats:
  - List-wise deletion: acceptable only if MCAR; e.g., using lower-frequency data only may lose seasonal effects but preserve long-term trends.
  - Deterministic imputation: applicable (e.g., carry-last-value) but shares ad-hoc drawbacks and potential bias.
  - Location-based imputation: use with caution for time series; overall mean often infeasible; moving averages and other interpolations (e.g., splines) are alternatives but may smooth away peaks and fail for gaps > one period.
  - Distribution-based imputation: requires conditional distributions incorporating lagged variables; this effectively becomes model-based.
  - Model-based methods: most frequently adopted for incomplete time series; options include autoregressive/moving-average models, state-space models, curve fitting/smoothing, Kalman filters, dynamic Bayesian networks; alternative classification/optimization approaches include neural networks, k-means clustering, seasonal pattern recognition, genetic algorithms.
    - Neural networks: non-parametric, adaptive, learn complex input-output relations.
    - Cluster analysis (k-means): partition items into homogeneous clusters; assign items to closest cluster center.
    - Pattern recognition/seasonal pattern recognition: identify recurring temporal structures.
    - Genetic algorithms: heuristic optimization via evolution-inspired operations.
  - Donor-based/statistical matching for time series:
    - Variable-wise donor selection uses similarity of trajectories (“time series donor”).
    - Inter-variable (cross-sectional) donor selection uses similarity of additional characteristics (“cross-sectional donor”).
    - Nearest-neighbor methods useful for metric time-series variables; cold-/hot-deck feasible for non-metric variables.
  - Multiple imputation: applicable for all missing patterns in time series; examples of tailored multiple-imputation approaches for cross-sectional time series and non-parametric curve fitting exist.
  - Hybrid procedures: common practice; methods should be carefully selected and combined according to the missing-data pattern.

### Evaluation and quality monitoring (principles)
- Thorough monitoring and evaluation of the imputation process is of high importance and vitally contributes to resulting data quality.
- Imputation quality indicators typically measure (text truncated in source at this point).

*Source: _wp11151 - 4.      cross-sectional multivariate time series: multiple variables/indicators observed for*

### 1.      the extent to which imputation was required in a dataset,

### _wp11151 - 1.      the extent to which imputation was required in a dataset,

### Major quality indicators for imputed data
- Five principal aspects to be reported when publishing analyses based on imputed data:
  - 1.      the extent to which imputation was required in a dataset,
  - 2.      the performance of the applied method,
  - 3.      the accuracy of the imputation results,
  - 4.      the variability of statistics based on the imputed dataset, and
  - 5.      the plausibility of imputed values.
- Reporting levels:
  - Overall indicators for the imputed dataset.
  - Variable-specific indicators.
  - Indicators for each imputed value.
  - For time series: section- and period-specific indicators can be defined.
- Plausibility assessment:
  - Editing procedures (Fellegi, Holt, 1976; de Waal, Pannekoek, Scholtus, 2011) may be applied to the completed dataset (i.e. after imputation).
  - Multiple rounds of imputation and plausibility checks may be required.
  - A plausibility checking mechanism may be integrated in the imputation process to avoid imputing implausible values; in that case, ex post plausibility evaluation may be unnecessary.
- Thresholds and comparative usage:
  - For most quality indicators, no generally valid thresholds discerning high- and low-quality imputation are available.
  - Comparative usage is recommended to enable decisions between different approaches, models, parameterizations, or scenarios.
  - Relative advantage can be calculated; example: evaluate statistical significance of differences between performance indicators via repeated measures analysis of (co-)variance with the imputation procedure as a factor; if significant overall difference, pairwise comparisons via t-tests (Student, 1908) can create a quality ranking of imputation procedures.
- Time series-specific considerations:
  - Adaptations or different interpretations of measures are often required for time series.
  - Analyze smoothness of an imputed variable (per section) over time to identify implausible peaks introduced by imputation, especially if imputation was carried out across sections or across variables instead of over time.
  - Smoothness checks can use confidence bands of moving averages or kernel estimates.

### A. Degree of missingness
- Purpose:
  - Measures indicate dataset completeness prior to imputation.
  - Comparing measures before and after imputation quantifies reduction in degree of missingness and enables comparison across procedures.
- Empirical impacts (experimental studies, e.g. Kaiser (1986)):
  - An increase in missing values per observation, in the proportion of incomplete observations in the dataset, or in both severely affects:
    - (i) the quality of missing value estimates,
    - (ii) the magnitude of the discrepancy in means, and
    - (iii) the preservation of the covariance structure.
  - Increasing sample size can reduce these adverse effects.
- Indicators (mainly ratios of missing values, variables, observations, sections, or time periods with respect to corresponding totals). Examples:
  - 6.      the number of observations with one or multiple missing values as a ratio of the total number of observations. For cross-sectional time series, instead of observations sections with missing values are counted.
  - 7.      the frequency distribution of the number of variables to be imputed per observation (section) and appropriate distribution measures (e.g. min, max, median).
  - 8.      the   frequency distribution of the number of periods to be imputed per variable (and/or section) and appropriate distribution measures. This measure only applies to time series.
  - 9.      the proportion of observations (or sections or periods) missing for specific variables (or sections or periods).
  - 10.      the total number of missing values as a ratio of the total number of cells (i.e. missing or non-missing data points) of the dataset.
- Cross-sectional time series and weighting:
  - For cross-sectional time series, weighted missingness rates can be used (e.g., sections as countries with weights such as population size, GDP, or purchasing power parity) to increase comparability across countries.
- Missingness pattern analysis:
  - Contingency or correlation matrices of missingness patterns of variables (or observations, sections, periods) may assess interdependency between missingness patterns.
  - Analyze missingness patterns themselves statistically to identify relations between variables (or sections or periods) with respect to missingness.

### B. Performance of imputation method
- General:
  - Apart from reduction of missingness, performance criteria are typically method-specific.
  - For some approaches (deterministic, location-, or distribution-based imputation), no method-specific performance evaluation is possible.
- Model-based methods (overall performance indicators):
  - Use standard regression model quality criteria such as coefficient of determination (R²), various information criteria (e.g. Akaike or Bayes), or p-values of regression coefficients.
  - For separate models across variables/sections, indicators are variable-/section-specific and can be aggregated (e.g. simple weighted averaging or composite indicators).
  - Aggregation comparability is limited to specific situations (e.g., same imputation method applied to different vintages).
- Donor-based methods and equivalence class matching:
  - Compute distribution parameters of the usage frequency of individual donors or the values of the distance function between donor and recipient (if applicable).
  - Quality interpreted as: less donor reuse and closer donor-recipient distance imply higher quality.
  - Typical aggregation functions: median and maximum.
  - Calculable at dataset, variable, section, or period level.
- Regression-based and propensity score matching:
  - Use standard regression quality measures and the distance between donor and recipient with respect to estimated matching variables.
  - Location and variability measures of the distance are usually used.
- Multiple imputation:
  - Performance indicators of the model estimated for the posterior distribution are relevant (e.g., R² or information criteria).
  - Measures of the variability of the multiple imputed values are considered performance measures.
- Individual imputed value reporting:
  - For model-based imputation, multiple imputation, or regression-based statistical matching: report the standard error or confidence interval for the estimated (=imputed) value.
  - For donor-based methods: provide a reference to the donor used.
  - For distance-based hot-/cold-deck imputation, nearest-neighbor approaches, or equivalence class statistical matching: additionally provide the value of the distance function between donor and recipient and the number of times the same donor was used in the whole dataset.
- Hybrid approaches:
  - Provide performance measures of the component methods.
  - Aggregating component performance measures into one “hybrid” performance indicator is appealing but usually hard to interpret.

### C. Accuracy of imputation results
- Four commonly discerned types of imputation accuracy (Chambers, 2000):
  - 11.      predictive accuracy or effectiveness: maximal preservation of true values,
  - 12.      ranking accuracy: maximal preservation of true ordering relationship in imputed values,
  - 13.      distributional accuracy: maximal preservation of (marginal and higher order) distributions of true values,

*Source: _wp11151 - 1.      the extent to which imputation was required in a dataset,*

### 14.      estimation accuracy: maximal preservation of analytic results and conclusions.

### 14.      estimation accuracy: maximal preservation of analytic results and conclusions.

### Typology and intended usage
- The typology constitutes a hierarchy: fulfillment of predictive accuracy, which is the strongest type of accuracy, implies the other three types of accuracy.
- The relevance of predictive and ranking accuracy depends on the intended usage of the completed dataset:
  - If the dataset is to be publicly released or used for the development of prediction models, predictive and ranking accuracy are crucial.
  - If the objective is to produce and publish aggregated estimates, these two criteria are less important.
- Rubin (e.g. 1996) claims the aim of (multiple) imputation should be statistically valid inference based on the completed dataset (i.e. estimation accuracy) rather than recreation of true values by optimal point prediction (i.e. predictive accuracy).
- Practical definition: modified estimation accuracy usually measures merely the reproduction of lower order moments (at least mean and variance) of the distributions of true values.
- According to this definition, distributional and estimation accuracy are equivalent for nominal and ordinal variables.

### Validation approach: cross-validation for imputation
- Because true missing values are unknown, accuracy indicators are estimated by:
  - Treating available values as missing,
  - Imputing these fictitiously missing values,
  - Comparing the imputed values to the ignored true values.
- This technique is known as cross-validation.
- Common cross-validation types: repeated random sub-sampling validation, k-fold cross-validation, leaving-one-out cross-validation.
- For validation of imputation results the leaving-one-out approach is typically used:
  - One value is set to missing at a time; theoretically repeated for each value, but in practice repeated over a sample of available values.
- Cross-validation is usually conducted separately for each variable with missing values; overall measures derive from aggregation of variable-specific measures.
- All accuracy measures depend on the variable type.

### Accuracy measures by variable type

- Nominal variables:
  - Measure: proportion of off-diagonal entries for the square table obtained by cross-classifying imputed and true values.
    - If imputation preserves individual values, this indicator is equal or close to zero.
  - Dichotomous variables: area under the receiver-operating characteristic (ROC) curve (Fawcett, 2006).
    - ROC plots sensitivity vs. 1 - specificity.
    - Sensitivity = proportion of correctly imputed “1”s (true positive rate).
    - 1 - specificity = proportion of correctly imputed “0”s (true negative rate).
    - The closer the area under the ROC curve is to 1, the more accurate the imputed values.
  - Nominal variables with p  2 categories can be transformed to a set of p - 1 dichotomous variables; sensitivity, specificity, and area under the ROC curve can be calculated separately for each dummy and aggregated.
  - Preservation of marginal distribution: assessed by a Wald-type test statistic comparing imputed and true distributions across categories (for details see Chambers, 2000).

- Ordinal variables:
  - Imputation should satisfy ranking accuracy in addition to predictive and distributional accuracy.
  - Distributional accuracy: measured as for nominal variables.
  - Predictive and ranking accuracy: measured simultaneously by taking magnitude of imputation errors into account via ordinal distance between imputed and true values (for details see Chambers, 2000).

- Metric variables:
  - Closeness measure: weighted Bravais-Pearson correlation between imputed and true values.
    - Not recommended for highly skewed or non-normal data due to sensitivity to outliers.
  - Alternative: estimates of a (robust) regression model without intercept of true values on imputed values.
    - Predictive accuracy assessment corresponds to testing whether the estimated regression parameter is equal to 1.
    - The coefficient of determination is a related regression-based measure for predictive accuracy.
    - Regression mean square error can be regarded as an inverse measure of predictive accuracy.
  - Preservation of ordering: replace imputed and true values by their ranks and calculate predictive accuracy measures on ranks.
  - Distributional accuracy tests:
    - Kolmogorov-Smirnov test (Massey, 1951) of equality of probability distributions comparing imputed and true value distributions.
    - Wilcoxon rank-sum or Mann-Whitney-U test (Wilcoxon, 1945; Mann, 1947).
  - Estimation accuracy concerning the mean: dependent t-test for paired samples (Student, 1908).

### Variability of statistics based on the imputed dataset
- Statistically most relevant measures in evaluation: bias and variance of estimates based on the completed data.
- Complexity of deriving closed-form solutions for variance and bias increases rapidly with complexity of missing data patterns and imputation methods.
- Theoretical work on direct calculation of variance and bias is generally limited to simple missing-data constellations.
- Neglecting imputation effects leads to underestimated variances and jeopardized validity of confidence statements (Cox, Folsom, 1978).
- Recommended approach for complex procedures: simulation.

- Multiple imputation:
  - Rubin (1987) advocates routine production of several sets of imputed values under different models or sets of assumptions as part of regular data processing.
  - Producing multiple imputations leads to estimates of imputation error and allows investigation of effects of different models (as discussed in the section on multiple imputation).

- Bootstrap approach (Shao and Sitter, 1996):
  - Generate bootstrap samples of complete and incomplete observations; apply imputation procedure to each bootstrap sample.
  - Use distribution of bootstrap estimates for inference.
  - This approach can be regarded as a kind of cross-validation.
  - Contrast with multiple imputation:
    - Multiple imputation samples imputed values from the posterior distribution of the incomplete variable (without replacement).
    - Bootstrap draws samples from the original dataset (with replacement) and imputes missing values for each sample.

### Conclusion: practice and recommendations
- The paper provides methodological choices for imputation of missing data in cross-country time series and investigates data structures and missingness patterns of time series.
- Offers evaluation procedures based on statistical quality criteria for assessing imputation outcomes.
- Aims to raise awareness of data producers and analysts in international financial institutions about challenges posed by missing data and techniques for handling them.
- Promotes usage of sound imputation procedures and stresses importance of accompanying any imputation process with reasonable quality indicators.
- Recommendation: publish a description of the applied missing data technique and quality assessment results together with the data to reduce ambiguity of incomplete data and facilitate analysis and interpretation for advising policy makers.

*Source: _wp11151 - 14.      estimation accuracy: maximal preservation of analytic results and conclusions.*

---


_Source: https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2011/_wp11151.pdf_