## Surrogate Data Models

## Source details

**Canonical URL:** [Surrogate Data Models](https://www.imf.org/-/media/files/publications/wp/2023/english/wpiea2023041-print-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2023/english/wpiea2023041-print-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2023/english/wpiea2023041-print-pdf.pdf.json)

---

### Costs and persistence of economic crises
- Economic crises cause sharp contractions in activity, lower investment, and permanent output losses.
- Empirical estimates cited:
  - "ten years after the 2017 global recession potential output in the United States converged to a level about 12 percentage points below that implied by its crisis trend."
  - "five years following a financial crisis, the gross domestic product was about 9 percentage points lower in OECD countries."
  - Empirical evidence suggests "following a financial crisis output remains permanently depressed one decade later."

### Purpose of early warning systems and ML adoption
- Early warning systems aim to:
  - Identify economic and financial imbalances that increase vulnerability to distress and crisis.
  - Identify economic and financial drivers ahead of a crisis to help policymakers prevent crisis realization via targeted policies.
- Machine learning methods have been rapidly adopted in crisis prediction to:
  - Handle many features (explanatory variables).
  - Capture non-linear effects prevalent before crises.
- Trade-off noted:
  - ML models often increase predictive power but reduce interpretability, limiting their usefulness for policy guidance.

### Proposal: Surrogate Data Models (SDMs) — rationale and advantages
- SDMs are proposed to reduce dimensionality of ML crisis prediction models and enhance interpretability.
- Key characteristics and goals of SDMs:
  - Restrict features to those economic analysts typically monitor and forecast.
  - Translate model results into an economic domain familiar to senior policymakers.
  - Facilitate "what-if" scenario analysis to assess policy measures.
- SDM approach:
  - Approximate the black box ML model using a low-dimensional set of features, which may include variables not present in the original ML feature set.
  - With a low-dimensional SDM feature set, interpretability is achieved via a surrogate explanation method such as SHAP.
- Econometric/statistical justification for dimension reduction:
  - Common dynamics in economic and financial variables can be summarized by a smaller set of latent factors (references to dynamic factor models, high-dimensional VAR reductions, PCA, supervised principal components, and manifold learning).

### Short survey of ML crisis prediction methods (selective)
- Historical methods: probit/logit models and non-parametric signal-extraction models.
- Recent ML-based findings and approaches (selective list):
  - Holopainen and Sarlin (2017): ML methods such as k-nearest neighbors, neural networks, and ensemble learning outperformed logistic regression in out-of-sample forecasting for 15 European countries.
  - Bluwstein et al. (2021): for 17 advanced economies over "1870–2016", used 16 features; all ML models except decision trees outperformed logistic regression; Shapley regressions identified credit growth and the slope of the yield curve as main predictors.
  - Fouliard et al. (2021): predicted systemic financial stress in EU countries and the US three years ahead using online learning; models used "244 features" at quarterly frequency.
  - Hellwig (2021): ML techniques (elastic net, random forests, gradient-boosted trees) delivered superior performance when predictors are many.
  - IMF (2021): suite of ML prediction models for sectoral crises (financial, fiscal, external, real); dataset covered "all 190 IMF member countries"; used many features and data transformations; employed data imputation and synthetic oversampling; feature contributions assessed using the SHAP method.
  - Hacibedel and Qu (2022): ensemble predicting systemic non-financial corporate distress using "40 predictors"; ensemble delivered better and more robust performance.
  - Beutel et al. (2019): using a sample of advanced economies "1971–2016" found ML models sometimes failed to outperform logit in recursive out-of-sample predictions; performance depends on crisis definition, class imbalance, and feature availability.

### Surrogate models and feature importance
- Challenge: large numbers of features hamper interpretability; understanding feature importance is crucial for policy trust and model improvement.
- Surrogate models explained:
  - Global surrogate: explains average behavior of an ML model by training an interpretable model G on (X, Ŷ = F(X)).
  - Local surrogate: explains individual predictions by generating synthetic perturbed samples around an instance and fitting a local interpretable model.
- Local surrogate methods noted: LIME and SHAP.
- SHAP specifics:
  - Based on Shapley values from cooperative game theory.
  - Shapley value formula as stated in the source:
    - φ_j(f) = (1/n) ∑ [ f(S ∪ {x_j}) − f(S) ] ( (n−1 |S| ) )_S.
  - SHAP approximation model:
    - g(z′) = φ_0 + ∑ φ_j z_j′ , where z′ ∈ {0,1}^n is the coalition vector.

### Application to the IMF ML crisis models — objective and approach
- Goal: Test whether SDMs can facilitate crisis prediction for the IMF ML sectoral crisis prediction models (assumed to be the true models underlying crisis dynamics in the real, fiscal, and external sectors).
- SDM purpose: Provide parsimonious, interpretable proxies for the IMF ML crisis risk indices to improve explainability and scenario analysis.

### Feature selection for SDMs
- Selection criteria:
  - Variables with theoretical or intuitive association with crisis realizations.
  - Variables that country economists can project under macro-financial scenarios (e.g., variables in the World Economic Outlook).
  - A small number of features to enhance interpretability (much fewer than VE risk indices).
- Number of selected features per sector: range from 12 to 20.
- Feature categories and inclusion:
  - Business cycle: GDP growth, Inflation, Unemployment rate, Deposit rate, Long-term bond yield, US deposit rate, US long-term bond yield, Investment growth (Investment growth included only in Real).
  - Debt and fiscal: Revenue to GDP ratio, Revenue to GDP ratio change, Deficit to GDP ratio, Deficit to GDP ratio change, Government debt to GDP ratio.
  - External: External debt to GDP ratio, External debt to GDP change, Portfolio investment to GDP ratio, Reserve assets to GDP ratio, Reserve assets to GDP ratio change, Current account to GDP ratio.
  - Global assumptions: Depreciation against US dollar, Oil price growth, Agriculture price growth.
  - Other: GDP per capita.
- Source of variables: IMF historical data and 5-year projections in the World Economic Outlook.

### SDM estimation strategy and model design
- Model family: Random Forest Models (RFMs).
- Combination approach:
  - Global model estimated with data from all IMF member countries (a total of 196 countries).
  - Income-based country-group models estimated separately for: AE, EM, LIC.
  - Combination: SDM risk indices are optimal linear combinations of the global model and income-based country-group models with weights chosen to minimize out-of-sample mean squared error (MSE).
  - Weight estimation imposes positive constraints to reduce estimation error.
- Country-group sizes:
  - AE: 40 countries
  - EM: 97 countries
  - LIC: 59 countries
- Sample periods:
  - Real and fiscal risk index models: 1980 to 2021.
  - External risk index model: 1989 to 2021.
- Cross-validation and hyperparameter selection:
  - Method: gap-block cross-validation (gap size chosen 1 year) with 15-fold cross-validation (validation set length ≈ 2 years).
  - Robustness: varied gap size and validation set between 1 and 3 years to verify cross-validation R-square stability.
  - Resulting hyperparameter choices:
    - Tree depths below 15.
    - Number of predictors in each tree varying from 10 percent to 70 percent of total predictors (from 2 to 14 at most).
    - Number of trees in the forest set equal to 100.
- Combination weight estimation: optimal combination weights estimated based on model predictions on the validation sets.

### Estimated combination weights (by sector)
- Real sector:
  - AE: 1
  - EM: 1
  - LIC: 0.7
- Fiscal sector:
  - AE: 0.6
  - EM: 0.4
  - LIC: 0.8
- External sector:
  - AE: 0.4
  - EM: 0.7
  - LIC: 0.6
- Note: Except for AE and EM models in the real sector, all estimated weights are positive and below 1, indicating diversification effects from model combinations.

### Out-of-sample performance (year 2022)
- Rank Correlation between SDM indices and VE indices (out-of-sample, 2022):
  - Real: 0.41
  - Fiscal: 0.88
  - External: 0.56
- R-squared (out-of-sample, 2022):
  - Real: 0.18
  - Fiscal: 0.77
  - External: 0.34
- Interpretation:
  - SDM performs especially well in the fiscal sector (rank correlation ≈ 0.9 and R-squared 0.77).
  - Performance is weaker in real and external sectors (rank correlations 0.41 and 0.56; R-squared 0.18 and 0.34), likely reflecting absence of financial market variables in the selected features.

### Scenario projections and uncertainty bounds
- Two projection scenarios for 2022–2026:
  1. IMF (2022) 5-year projections for model features (baseline).
  2. An adverse scenario: higher inflation, oil and gas supply shock, further slowdown of the Chinese economy, and further tightening of financial conditions super-imposed on IMF 5-year projections.
- Scenario characteristics averaged across country income groups and reported in tables (variables include Balance of Payments to GDP ratio, Portfolio investment to GDP ratio, External debt to GDP ratio and changes, Depreciation against US dollar, Short-term deposit rate, US Short-term deposit rate, Long-term bond yield, US Long-term bond yield, GDP per capita relative to US, Revenue to GDP ratio and changes, Deficit to GDP ratio and changes, Government debt to GDP ratio, Foreign reserve to GDP ratio, Reserve assets to GDP ratio changes, Unemployment rate, Investment growth, GDP growth, Agriculture price growth, Inflation, Oil price growth).
- Uncertainty bounds on SDM projections:
  - One-standard deviation upper and lower bounds constructed from log residuals in the validation set:
    - Upper bound: multiply projection values by exp(one-standard deviation of the log residuals).
    - Lower bound: divide projection values by the same factor.
  - Country-group averages are calculated from country-specific bounds; plotted area between bounds.

### Goodness-of-fit, limitations, and notable episodes
- General finding: SDMs track general trends of VE crisis risk indices well for all country groups and sectors.
- Notable periods where SDMs underpredict crisis indices:
  - Pre-2008 Global Recession (2006–2008): SDM underpredicts risk indices across all sectors in Advanced Economies and in the fiscal sector for Emerging Markets.
  - 2020 Covid-19 onset: SDMs fail to capture the large surge in real and external sector risk indices for Low-Income Countries.
- Interpretation of limitations:
  - A dataset with only a few features may suffice to capture many crisis dynamics, but selected features cannot capture all information in the full IMF ML models; therefore SDMs may underpredict particularly severe crisis episodes.
  - The absence of financial market variables in the SDM feature set likely contributes to weaker performance in the real and external sectors.

### Explainability: SHAP analysis and feature importance (SDMs)
- SHAP analysis applied to SDMs to assess feature contributions to risk indices:
  - Predictors ranked by sum of absolute SHAP values.
- Key driver relationships:
  - Real sector crises: lower GDP growth and tightening financial conditions (proxied by increase in USD short-term deposit rates) are the most important drivers.
  - Fiscal crises: higher debt levels and weaker government revenue predict fiscal crises.
  - External crises: currency depreciation and worsening current account balance signal future external sector crises.
- Non-linearities detected:
  - Mixtures of blue and red dots across SHAP axes indicate non-linear relationships (e.g., commodity price increases benefit exporters but hurt importers).
- Cross-country average SHAP contributions (time series demeaned and stacked to equal SDM indices plus a constant):
  - 2022 peaks of LICs’ external sector risk mainly driven by recent inflation increases and high government deficits starting from 2020.
  - From 2022 onwards, elevated projected fiscal risks in LICs mainly driven by persistent tightening of global financial conditions arising from higher US long-term yields.
- Regional comparison example:
  - Western hemisphere (WHD): increases in fiscal sector risk mainly driven by inflation.
  - African (AFR) countries: fiscal sector risk increases mainly driven by tightening global monetary conditions proxied by US interest rates.

### Overall assessment from the application
- SDMs with a small set of interpretable features can track crisis risk dynamics well, especially in the fiscal sector.
- SDMs enhance interpretability and scenario analysis by making it easier to disentangle feature contributions (e.g., via SHAP), which can guide policy and country economist decisions.
- Caution: SDMs may miss extreme crisis episodes and suffer where important financial-market features are excluded; users should scrutinize surrogate indices for the real and external sectors.

### Conclusions — main findings, limitations, and policy implications
- Main findings:
  - ML tools are especially useful for supporting economic and financial surveillance, especially crisis prediction.
  - Enhancing the ability of the ML models to guide crisis prevention and mitigation policies requires improving model interpretability and establishing causal relationships.
  - Motivated by the empirical fact that a few latent variables are sufficient to explain the joint dynamics of a high dimensional dataset, the paper proposes using SDMs as dimensionality reduction and interpretability enhancing tools suitable for crisis prediction, prevention, and mitigation.
  - SDMs were tested to explain the output of a suite of IMF ML models forecasting crises in the external, fiscal, and real sectors, using as predictors only a subset of the variables country economists typically monitor on a recurrent basis.
  - The SDMs captured well the crisis dynamics generated by the ML models in general.
- Performance and limitations:
  - In specific episodes, especially during 2007-8, SDMs failed to match the crisis severity (as measured by the ML crisis index), likely due to lack of information not captured by the surrogate dataset.
  - Failures suggest SDMs may omit important information present in the original high-dimensional datasets and thus may understate crisis severity in some historical episodes.
  - Shapley-based contributions singled out main drivers consistent with economic intuition and anecdotal crisis narratives.
- Interpretability and methodology:
  - Simpler surrogate models that rely on the original high-dimensional dataset do not provide substantial interpretability gains.
  - SDMs serve as dimensionality reduction and interpretability-enhancing tools by focusing on a lower-dimensional surrogate dataset derived from the original high-dimensional data.
  - Calculations were performed using the Python API SHAP.
- Policy implications and recommendations:
  - SDMs could be suitable interpretability tools to support crisis prediction, prevention, and mitigation, helping to identify main drivers of model outputs in terms consistent with economic intuition.
  - The use and implementation of SDMs should be explored on a case-by-case basis, acknowledging episode-specific information gaps and model limitations.
  - Improving model interpretability and pursuing causal inference remain priorities to increase the usefulness of ML models in policy-making settings.

*Source: wpiea2023041-print-pdf — IMF Working Paper, "Surrogate Data Models."*

### Introduction............................................................................................................

### Introduction

### Costs and persistence of economic crises
- Economic crises cause sharp contractions in activity, lower investment, and permanent output losses.
- Empirical estimates cited:
  - "ten years after the 2017 global recession potential output in the United States converged to a level about 12 percentage points below that implied by its crisis trend."
  - "five years following a financial crisis, the gross domestic product was about 9 percentage points lower in OECD countries."
  - Empirical evidence suggests "following a financial crisis output remains permanently depressed one decade later."

### Purpose of early warning systems and ML adoption
- Early warning systems aim to:
  - Identify economic and financial imbalances that increase vulnerability to distress and crisis.
  - Identify economic and financial drivers ahead of a crisis to help policymakers prevent crisis realization via targeted policies.
- Machine learning methods have been rapidly adopted in crisis prediction to:
  - Handle many features (explanatory variables).
  - Capture non-linear effects prevalent before crises.
- Trade-off noted:
  - ML models often increase predictive power but reduce interpretability, limiting their usefulness for policy guidance.

### Proposal: Surrogate Data Models (SDMs)
- The paper proposes using surrogate data models (SDMs) to reduce dimensionality of ML crisis prediction models and enhance interpretability.
- Key characteristics and goals of SDMs:
  - Restrict features to those economic analysts typically monitor and forecast.
  - Translate model results into an economic domain familiar to senior policymakers.
  - Facilitate “what-if” scenario analysis to assess policy measures.
- The paper provides a concrete illustration applying SDMs to a suite of IMF ML models predicting sectoral economic crises (IMF, 2021).

### Short survey of ML crisis prediction models (selective)
- Historical methods: probit/logit models and non-parametric signal-extraction models.
- Recent ML-based findings and approaches (selective list):
  - Holopainen and Sarlin (2017): comparison in 15 European countries; ML methods such as k-nearest neighbors, neural networks, and ensemble learning outperformed logistic regression in out-of-sample forecasting.
  - Bluwstein et al. (2021): for 17 advanced economies over "1870–2016", used 16 features; all ML models except decision trees outperformed logistic regression; Shapley regressions identified credit growth and the slope of the yield curve as main predictors.
  - Fouliard et al. (2021): predicted systemic financial stress in EU countries and the US three years ahead using online learning; models used "244 features" at quarterly frequency.
  - Hellwig (2021): traditional econometric models sometimes underperform simple heuristics for fiscal crises; ML techniques (elastic net, random forests, gradient-boosted trees) delivered superior performance when predictors are many.
  - IMF (2021): suite of ML prediction models for sectoral crises (financial, fiscal, external, real); dataset covered "all 190 IMF member countries"; used many features and data transformations; employed data imputation and synthetic oversampling; feature contributions assessed using the SHAP method.
  - Hacibedel and Qu (2022): ensemble of gradient-boosted trees, linear discriminant, and logit Lasso to predict systemic non-financial corporate distress using "40 predictors"; ensemble delivered better and more robust performance across time blocks and country groups.
  - Beutel et al. (2019): using a sample of advanced economies "1971–2016" (a "45-year data sample") found ML models sometimes failed to outperform logit in recursive out-of-sample predictions, including for the "2007-8 global financial crisis"; performance depends on crisis definition, class imbalance, and feature availability.

### Surrogate models and feature importance
- Challenge: large numbers of features in ML crisis models hamper interpretability; understanding feature importance is crucial for policy trust and model improvement.
- Surrogate models are simpler approximation models that mimic complex systems at lower computational cost and can enhance understanding.
- Types of surrogate models:
  - Agnostic vs. model-specific.
  - Global vs. local:
    - Global surrogate: explains average behavior of an ML model by training an interpretable model G on (X, Ŷ = F(X)).
    - Local surrogate: explains individual predictions by generating synthetic perturbed samples around an instance and fitting a local interpretable model.
- Notable local surrogate methods: LIME (Ribeiro et al., 2016b) and SHAP (Lundberg and Lee, 2017).
- SHAP explanation:
  - Based on Shapley values from cooperative game theory (Shapley, 1953).
  - Concept: a feature’s Shapley value is its average marginal contribution over all coalitions of features.
  - Formula as stated in the source:
    - φ_j(f) = (1/n) ∑ [ f(S ∪ {x_j}) − f(S) ] ( (n−1 |S| ) )_S.
  - SHAP approximation model:
    - g(z′) = φ_0 + ∑ φ_j z_j′ , where z′ ∈ {0,1}^n is the coalition vector.

### Surrogate Data Models (SDMs): rationale and advantages
- Standard surrogate models may still struggle when the feature set is large or contains complex data transformations lacking straightforward economic interpretation.
- SDM approach:
  - Approximate the black box ML model using a low-dimensional set of features, which may include variables not present in the original ML feature set.
  - With a low-dimensional SDM feature set, it becomes easier to interpret results even if the SDM itself is not a simple model; interpretability is achieved via a surrogate explanation method such as SHAP.
  - Figure 1 in the source contrasts standard surrogate vs. SDM approaches (schematic depiction).
- Econometric and statistical justification for dimension reduction:
  - Economic behavior often depends on the joint distribution of economic and financial variables, implying common dynamics that can be summarized by a smaller set of latent factors.
  - Methods and findings supporting dimension reduction:
    - Dynamic factor models capture common dynamics via few latent factors (Stock and Watson, 2016).
    - High-dimensional VARs can be reduced to low-dimensional VAR plus idiosyncratic components (Cubadda and Hecq, 2022).
    - Principal component analysis (PCA) and supervised principal components help extract latent factors (Fan et al. 2018; Bair et al., 2006).
    - Manifold learning suggests high-dimensional economic models are reducible to low-dimensional spaces suitable for clustering and early warning (Huang, Kou, and Peng, 2017; Chan-Lau and Wang, 2020).

*IMF WORKING PAPERS — Surrogate Data Models, Introduction.*

### 4. An application to the IMF ML crisis models

### 4. An application to the IMF ML crisis models

### Objective and approach
- Goal: Test whether Surrogate Data Models (SDMs) can facilitate crisis prediction for the IMF ML sectoral crisis prediction models (assumed to be the true models underlying crisis dynamics in the real, fiscal, and external sectors).
- SDM purpose: Provide parsimonious, interpretable proxies for the IMF ML crisis risk indices to improve explainability and scenario analysis.

### Feature selection
- Selection criteria:
  - Variables with theoretical or intuitive association with crisis realizations.
  - Variables that country economists can project under macro-financial scenarios (e.g., variables in the World Economic Outlook).
  - A small number of features to enhance interpretability (much fewer than VE risk indices).
- Number of selected features per sector: range from 12 to 20 (significantly fewer than those in VE risk indices).
- Feature categories and inclusion (as in Table 1):
  - Business cycle: GDP growth, Inflation, Unemployment rate, Deposit rate, Long-term bond yield, US deposit rate, US long-term bond yield, Investment growth (Investment growth included only in Real).
  - Debt and fiscal: Revenue to GDP ratio, Revenue to GDP ratio change, Deficit to GDP ratio, Deficit to GDP ratio change, Government debt to GDP ratio.
  - External: External debt to GDP ratio, External debt to GDP change, Portfolio investment to GDP ratio, Reserve assets to GDP ratio, Reserve assets to GDP ratio change, Current account to GDP ratio.
  - Global assumptions: Depreciation against US dollar, Oil price growth, Agriculture price growth.
  - Other: GDP per capita.
- Source of variables: IMF historical data and 5-year projections in the World Economic Outlook.

### Surrogate data models: estimation strategy
- Model family: Random Forest Models (RFMs).
- Combination approach:
  - Global model estimated with data from all IMF member countries (a total of 196 countries).
  - Income-based country-group models estimated separately for: AE, EM, LIC.
  - Rationale: account for parameter heterogeneity across Advanced Economies (AE), Emerging Markets (EMs), and Low-Income Countries (LICs).
  - Combination: SDM risk indices are optimal linear combinations of the global model and income-based country-group models with weights chosen to minimize out-of-sample mean squared error (MSE).
  - Weight estimation imposes positive constraints to reduce estimation error.
- Country-group sizes: AE: 40 countries; EM: 97 countries; LIC: 59 countries.
- Sample periods:
  - Real and fiscal risk index models: 1980 to 2021.
  - External risk index model: 1989 to 2021.
- Cross-validation and hyperparameter selection:
  - Method: gap-block cross-validation (gap size chosen 1 year) with 15-fold cross-validation (validation set length ≈ 2 years).
  - Robustness: varied gap size and validation set between 1 and 3 years to verify cross-validation R-square stability.
  - Resulting hyperparameter choices:
    - Tree depths below 15.
    - Number of predictors in each tree varying from 10 percent to 70 percent of total predictors (from 2 to 14 at most).
    - Number of trees in the forest set equal to 100 (adding more trees does not improve performance).
- Combination weight estimation: optimal combination weights estimated based on model predictions on the validation sets.

### Estimated combination weights (Table 2)
- Real sector:
  - AE: 1
  - EM: 1
  - LIC: 0.7
- Fiscal sector:
  - AE: 0.6
  - EM: 0.4
  - LIC: 0.8
- External sector:
  - AE: 0.4
  - EM: 0.7
  - LIC: 0.6
- Note: Except for AE and EM models in the real sector, all estimated weights are positive and below 1, indicating diversification effects from model combinations.

### Out-of-sample performance (year 2022) (Table 3)
- Performance metrics reported:
  - Rank Correlation between SDM indices and VE indices (out-of-sample, 2022):
    - Real: 0.41
    - Fiscal: 0.88
    - External: 0.56
  - R-squared (out-of-sample, 2022):
    - Real: 0.18
    - Fiscal: 0.77
    - External: 0.34
- Interpretation:
  - SDM performs especially well in the fiscal sector (rank correlation ≈ 0.9 and R-squared 0.77).
  - Performance is weaker in real and external sectors (rank correlations 0.41 and 0.56; R-squared 0.18 and 0.34), likely reflecting absence of financial market variables in the selected features. Surrogate indices in the real and external sectors should be treated more critically.

### Scenario projections and bounds
- Two projection scenarios shown for 2022–2026:
  1. IMF (2022) 5-year projections for model features (baseline) (blue line in figures).
  2. An adverse scenario: higher inflation, oil and gas supply shock, further slowdown of the Chinese economy, and further tightening of financial conditions super-imposed on IMF 5-year projections (red line in figures).
- Scenario characteristics: averaged across country income groups and reported in Table 4 (Panel A: Advanced economies; Panel B: Emerging market economies; Panel C: Low income economies). (The table lists projected changes relative to 2021 values for many variables across 2022–2026, including Balance of Payments to GDP ratio, Portfolio investment to GDP ratio, External debt to GDP ratio and changes, Depreciation against US dollar, Short-term deposit rate, US Short-term deposit rate, Long-term bond yield, US Long-term bond yield, GDP per capita relative to US, Revenue to GDP ratio and changes, Deficit to GDP ratio and changes, Government debt to GDP ratio, Foreign reserve to GDP ratio, Reserve assets to GDP ratio changes, Unemployment rate, Investment growth, GDP growth, Agriculture price growth, Inflation, Oil price growth).
- Uncertainty bounds on SDM projections:
  - One-standard deviation upper and lower bounds constructed from log residuals in the validation set:
    - Upper bound: multiply projection values by exp(one-standard deviation of the log residuals).
    - Lower bound: divide projection values by the same factor.
  - Country-group averages are calculated from country-specific bounds; plotted area between bounds in figures (light blue).

### Goodness-of-fit, limitations, and notable episodes
- General finding: SDMs track general trends of VE crisis risk indices well for all country groups and sectors.
- Notable periods where SDMs underpredict crisis indices:
  - Pre-2008 Global Recession (2006–2008): SDM underpredicts risk indices across all sectors in Advanced Economies and in the fiscal sector for Emerging Markets.
  - 2020 Covid-19 onset: SDMs fail to capture the large surge in real and external sector risk indices for Low-Income Countries.
- Interpretation of limitations:
  - A dataset with only a few features may suffice to capture many crisis dynamics, but selected features cannot capture all information in the full IMF ML models; therefore SDMs may underpredict particularly severe crisis episodes.
  - The absence of financial market variables in the SDM feature set likely contributes to weaker performance in the real and external sectors.

### Explainability: SHAP analysis and feature importance
- SHAP (Shapley) analysis applied to SDMs to assess feature contributions to risk indices:
  - Figure 5: distribution of Shapley values for SDMs (Real, Fiscal, External). Predictors ranked by sum of absolute SHAP values.
  - Key driver relationships (consistent with priors and economic intuition):
    - Real sector crises: lower GDP growth and tightening financial conditions (proxied by increase in USD short-term deposit rates) are the most important drivers.
    - Fiscal crises: higher debt levels and weaker government revenue predict fiscal crises.
    - External crises: currency depreciation and worsening current account balance signal future external sector crises.
  - Presence of non-linearities: mixtures of blue and red dots across SHAP axes indicate non-linear relationships (e.g., commodity price increases benefit exporters but hurt importers).
- Cross-country average SHAP contributions:
  - Figure 6: variables contributing most to crisis risk indices by sector and year (including 5-year scenario); time series of each variable’s average SHAP are demeaned and stacked to equal SDM indices plus a constant.
  - Examples highlighted:
    - 2022 peaks of LICs’ external sector risk mainly driven by recent inflation increases and high government deficits starting from 2020.
    - From 2022 onwards, elevated projected fiscal risks in LICs mainly driven by persistent tightening of global financial conditions arising from higher US long-term yields.
  - Regional comparison (example): Analysis comparing regional averages of fiscal sector SDM indices between 2022 fall and spring WEO shows different sources of vulnerability:
    - Western hemisphere (WHD): increases in fiscal sector risk mainly driven by inflation.
    - African (AFR) countries: fiscal sector risk increases mainly driven by tightening global monetary conditions proxied by US interest rates.

### Overall assessment
- SDMs with a small set of interpretable features can track crisis risk dynamics well, especially in the fiscal sector.
- SDMs enhance interpretability and scenario analysis by making it easier to disentangle feature contributions (e.g., via SHAP), which can guide policy and country economist decisions.
- Caution: SDMs may miss extreme crisis episodes and suffer where important financial-market features are excluded; users should scrutinize surrogate indices for the real and external sectors.

*Source: IMF Working Paper — Surrogate Data Models, section 4. An application to the IMF ML crisis models.*

### 5. Conclusions

### 5. Conclusions

### Main findings
- Machine learning (ML) tools are especially useful for supporting economic and financial surveillance, especially crisis prediction.
- Enhancing the ability of the ML models to guide crisis prevention and mitigation policies requires improving model interpretability and establishing causal relationships.
- Motivated by the empirical fact that a few latent variables are sufficient to explain the joint dynamics of a high dimensional dataset of economic and financial variables, the paper proposes using surrogate data models (SDMs) as dimensionality reduction and interpretability enhancing tools suitable for crisis prediction, prevention, and mitigation.
- SDMs were tested to explain the output of a suite of IMF ML models forecasting crises in the external, fiscal, and real sectors, using as predictors only a subset of the variables country economists typically monitor on a recurrent basis.
- The SDMs captured well the crisis dynamics generated by the ML models in general.

### Performance and limitations
- In a few specific episodes, especially during 2007-8, SDMs failed to match the crisis severity (as measured by the ML crisis index), likely the result of a lack of information not captured by the surrogate dataset.
- The failures in certain episodes suggest that SDMs may omit important information present in the original high-dimensional datasets and thus may understate crisis severity in some historical episodes.
- An analysis of the Shapley-based contributions of different predictors to crisis severity singled out main drivers consistent with economic intuition and an anecdotal crisis narrative.

### Interpretability and methodology
- Simpler models (a.k.a. surrogate models) that rely on the original high-dimensional dataset do not provide substantial interpretability gains.
- SDMs serve as dimensionality reduction and interpretability-enhancing tools by focusing on a lower-dimensional surrogate dataset derived from the original high-dimensional data.
- Calculations were performed using the Python API SHAP. See the documentation section there for a detailed explanation of how to interpret figures 5 and 6.

### Policy implications and recommendations
- SDMs could be suitable interpretability tools to support crisis prediction, prevention, and mitigation, helping to identify main drivers of model outputs in terms consistent with economic intuition.
- The use and implementation of SDMs should be explored on a case-by-case basis, acknowledging episode-specific information gaps and model limitations.
- Improving model interpretability and pursuing causal inference remain priorities to increase the usefulness of ML models in policy-making settings.

*Source: wpiea2023041-print-pdf - 5. Conclusions*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2023/english/wpiea2023041-print-pdf.pdf_