## tnmea2021003

## Source details

**Canonical URL:** [tnmea2021003](https://www.imf.org/-/media/files/publications/tnm/2021/english/tnmea2021003.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/tnm/2021/english/tnmea2021003.pdf.md)
- [Structured JSON version](/-/media/files/publications/tnm/2021/english/tnmea2021003.pdf.json)

---

### Executive Summary
- Purpose and scope of the Vulnerability Exercise (VE)
  - Cross-country, bottom-up, multi-sectoral exercise identifying country-specific near-term macroeconomic risks.
  - Covers vulnerabilities in the fiscal, external, financial, and non-financial corporate and household sectors.
  - Assessments reflect country-team judgement informed by consistent, cross-country quantitative models and country-specific context.
- Rationale for model updates and coverage improvements
  - New generation of VE models closes gaps in risk and country coverage from previous models.
  - New models:
    - assess risks in the external, fiscal, financial, and real sectors for all IMF member countries;
    - are applied consistently across country groups based on identical, quantitative crisis definitions;
    - are assessed within a single model when effective to improve evenhandedness.
- Use of machine-learning (ML) algorithms
  - ML captures interactions between economic parts, non-linear relationships, and robustness to outliers and diverse country experiences.
  - Focus on prediction and generalization to out-of-sample observations.
  - Non-parametric methods (e.g., random forests) can explore predictive relationships and interactions; signal extraction noted as a robust non-parametric precursor.
- Model evaluation and selection strategy
  - Horse-race format using rigorous out-of-sample performance metrics.
  - Models assess near-term crisis risk in external, financial, fiscal, and real sectors.
  - Findings:
    - Random forest-based models outperform other options in most cases.
    - Signal extraction performs best in other cases.
    - No single algorithm dominates ("no free lunch theorem").
  - Standardization:
    - Crisis definitions based on quantitative criteria from literature.
    - Variable selection based on extensive literature review; wider coverage including global and political factors.
    - All models estimated on annual data to maximize coverage.
- Interpretation and communication
  - Developed transparent methods for communicating ML outputs to country teams.
  - Shapley values highlighted to determine additive contributions of input series (see Annex VI).
  - Tools designed to move from prediction to informing policy-relevant judgements.
- Relationship to other IMF tools
  - VE sectoral risk assessments complement tools such as GFSR, ESR, debt sustainability assessments, external balance assessments, Growth-at-Risk models.
  - VE emphasizes a snapshot across all countries and sectors at a single point in time, crisis risk, cross-country consistency, and model performance.
- Limitations and caveats
  - ML models cannot predict crises where shocks or transmission channels have not been seen before.
  - Data availability restricts Big Data models in macro applications.
  - Country teams’ judgement remains central when idiosyncratic factors are important.

### Methods and models evaluated
- Models tested: signal extraction; logit (classic and Bayesian); penalized logit (RIDGE, LASSO, Elastic Net); tree-based models (random forest (RF) and variations: balanced random forest, RUSBoost, ADABoost, gradient boosting); support vector machines (SVM).
- Neural Networks: tested in limited experiments; not successful due to small dataset sizes.
- Boosting: ADABoost and RUSBoost trained subsequent learners on errors made by previous learners.
- Variable treatment: higher-frequency variables included though models estimated on annual data; missing data for tree-based models imputed using the sample median.
- Performance assessment
  - Primary evaluation: out-of-sample predictive performance using classic backtesting simulating a “real time” forecasting exercise beginning in the early 2000’s and rolling forward estimation and testing periods.
  - Complementary evaluation: cross-fold validation (time-block folds of consecutive years).
  - Main metrics: Area Under the Curve (AUC); also sum-of-errors and probability-oriented approaches where appropriate.
  - Backtesting caveat: classic backtesting may mainly test models against the Global Financial Crisis (GFC) in some sectors; supplemented by cross-fold validation and single-cutoff backtesting.
- Estimation sample and pooling
  - Assessed whether single pooled model across countries yields similar or better performance.
  - Pooling findings:
    - Pooling country income groups often improves predictive performance.
    - Pooling all countries typically improves forecasting for crises in LICs and AEs; provides similar out-of-sample performance for EMs.
    - For external sector risks, modeling by conventional income groups preferred for comparability.

### Communicating results and interpretability
- Interpretability tools:
  - Shapley values used to decompose risk assessments into additive contributions of variables; visualized in waterfall charts.
  - Box plots compare distributions of key indicators across crisis and non-crisis cases using Winsorized whiskers and specified percentile rules.
  - Similarity scores derived from RF node co-occurrence to identify countries with similar financial-sector risk profiles (example: Ireland 2005 similar to United Kingdom, Greece, Spain).
- Missing data practice (VE teams)
  - Common options considered: drop observations, drop variables, simple imputation (mean/median/mode), model-based imputation.
  - VE approach: drop variables with excessive missingness; simple imputation (median) for remaining gaps.
  - Fiscal model rule: variables dropped if more than one-third of observations missing; remaining gaps imputed with variable’s median.
  - Rationale: median robust to outliers and simpler to communicate; using median effectively prevents that variable from contributing to a country’s crisis score for that year.

### External sector model (sudden stops and EMP)
- Crisis definitions
  - Sudden stops: net private capital inflows as % of GDP at least 2 percentage points lower than previous year and two years before, and country approved to tap large IMF financial support.
  - Sudden stops with growth impacts (SSGI): large growth declines when change in growth relative to previous five-year average lies in the lower 10th-percentile.
  - EMP events: weighted average index of annual % depreciation in nominal exchange rate and annual decline in reserves as % of previous year’s GDP; EMP events defined when index lies in the lower 15th-percentile and country approved for large IMF support.
- Data coverage and incidence
  - SSGI sample: 53 EMs, period 1990-2017; 183 sudden stops = 12.3 percent of the sample, and 61 SSGIs = 4.1 percent of the sample.
  - EMP database: 192 countries, 1990-2017; average incidence 6.3 percent.
- Explanatory variables: final model based on 79 variables spanning first-, second-, third-generation crisis model variables, political shocks, current account shocks, contagion effects.
- Model selection and performance
  - Evaluated 1–2 year ahead crises using backtesting starting in 2007 and single-cutoff backtesting; main metric sum of errors, AUC for reference.
  - Findings:
    - SSGI: signal extraction performs best in out-of-sample AUC and sum of errors versus Logit (LASSO, RIDGE, Elastic Net) and tree-based (RF, RUSBoost).
    - EMP: no single consistent best model. Signal extraction best for AEs (by sum of errors), RUSBoost best for EMs, RF best for LICs.
    - Practical implementation: signal extraction for EMP in AEs; RF models within income groups for EMP in EMs and LICs. For EM oil exporters, pooled RF (EMs and LICs including oil exporters) applied due to small sample size.
- Variable importance and drivers
  - SSGIs driven by variables from all sectors; emphasis on asset bubble building/busting (stock and housing prices), global shocks and contagion (TED spread, frequency of AE banking crises), and country-specific external variables (external debt shocks, current account shocks).
  - EMP drivers vary by group:
    - Advanced Economies: external sector important; banking sector external exposure leading factor.
    - Emerging Markets: reserves growth, export growth, domestic credit growth; global shocks/contagion not among key drivers.
    - Low-Income Countries: 5yr inflation important; global shocks (TED spread, US term premium, Fed rate, dollar appreciation) among top predictors.
  - Variable interactions example: foreign liability share matters more for external risk when accompanied by large 5-year cumulative private credit-to-GDP growth.

### Fiscal sector model
- Crisis definition: fiscal crisis if any of four criteria met (with minimum two years gap between crises):
  1. Credit event: default, restructuring, or rescheduling of substantial size (>0.5 percent of GDP p.a.; defaulted nominal amount grows ≥ 10 percent p.a).
  2. Exceptionally large official financing: High-access IMF financial arrangement (≥ 100 percent of quota) or EU program.
  3. Implicit domestic public default: high inflation or steep increase in domestic arrears (thresholds listed).
  4. Loss of market confidence: high sovereign spreads or loss of market access (thresholds differ by AMs, EMs, LICs; example: level of spreads ≥ 1,000 bps for AMs; annual change thresholds listed).
- Dataset: 188 economies, 1980–2017.
  - On average, countries have undergone two fiscal crises since 1980; LICs > 3 crises on average; AEs < 1 crisis on average.
  - Crisis duration: EMs and LICs longest episodes—on average five years.
- Explanatory variables: final model based on 106 variables grouped into 11 categories; total permutations used in analysis extend to 290 variables.
- Model selection and testing
  - Evaluated Logit, Signal Extraction, RF, Elastic Net, gradient boosted trees, ADABoost, RUSBoost, Bayesian Logit, LASSO.
  - Classic back-testing starting in 2000; accuracy assessed with likelihood score and AUC.
  - RF model consistently performs best or among best for pooled sample across income groups.
- Variable importance and drivers
  - RF identifies external public debt (ratio to GDP and to exports) as the most important predictor after crisis history and development level.
  - Other top predictors: GDP per capita, reserve coverage of imports, inflation volatility, quality of bureaucracy.
  - Aggregated importance: country-specific external sector variables and public-debt-related variables most important, followed by real variables.
- Illustrative case
  - Greece 2009: out-of-sample forecast (model estimated through 2007) highlights stock of public debt (external public debt), current account pressures, and debt service as key drivers; model yields a 13 percent probability of default for Greece 2009 based on data through 2007.

### Financial sector model
- Crisis definition: banking sector crises per Laeven and Valencia (2018): significant financial distress plus significant policy interventions.
- Dataset: 117 countries, 1980–2017.
  - Crisis frequencies average one crisis every 30 to 45 years across income groups.
  - Crisis timing varies: 1980s–1990s EMs and LICs frequent; post-2000 crises concentrated in GFC and in AEs.
- Explanatory variables: final model based on 31 variables, including 7 financial sector variables and 9 reflecting financial–real sector interaction. Examples listed exactly as in source: Real 10yr US Yield, 3mo T-bill Rate HP gap, External Bank Liabilities to GDP, Loan to Deposit Ratio, Capital Adequacy Ratio, Avg. Bank Prob. Default, etc.
- Model selection and testing
  - Evaluated SVM, RF (balanced), Logit, RIDGE/LASSO Logit, Signal Extraction, ADABoost, RUSBoost.
  - Classic backtesting estimated through 2000 and then re-estimated adding a year; applied to three out-of-sample years per estimation; complemented by cross-validation on groups of consecutive years.
  - Balanced RF consistently performs best or among best: best in cross-validation and second-best in backtesting.
  - Robustness: similar results for 1- and 3-year horizons and datasets with 84, 160, and 949 variables.
- Variable importance and drivers
  - Balanced RF emphasizes Real 10yr US Yield (proxy for global financial conditions), followed by External Bank Liabilities to GDP, 3m US TBill Gap, gross reserves to GDP, external debt to exports, total debt growth, corporate debt sub-investment grade share, and other indicators.
  - Risk drivers span all sectors; global interest rates and external banking sector exposures play central roles.
- Variable interactions example
  - Contribution of external debt to assessed risk increases markedly when Real US 10-year bond rates are higher; risks rise as external debt exceeds 15 percent with magnification under higher global interest rates.
- Aggregate risk pattern
  - Historical average financial risk index: sustained levels through mid-1990s, lull mid-2000s, spike during GFC, lull afterward; recent uptick driven by global variables, notably 3-month US T-Bill gap exceeding 1% by end-2018.

### Real sector model
- Crisis definition
  - Real sector crises defined using 4 GDP series and 4 thresholds (country annual growth rate; cumulative growth over past three years; growth relative to most recent five-year average; average GDP level relative to previous three-year average), focused on GDP per working-age person.
  - For each series, crises flagged if below 10th percentile within groups: i) all countries, ii) same income group (AEs, EMs, LICs, WEO 2020), iii) income group in 1980 with extra small-population category, iv) same tercile of total sample for year-on-year growth volatility.
  - Sixteen crisis criteria (4 series × 4 thresholds); a country-year recorded as real sector crisis when nine or more indicators signal crisis.
- Historical clustering: early 1980s (LICs and EMs, Latin American debt crisis), early 1990s, late 1990s (Asian Financial Crisis), late 2000s (GFC).
- Dataset: average of two real sector crises per country over sample; average crisis duration two years.
- Explanatory variables
  - GLOBALFINANCIALREAL, EXTERNAL, FINANCIAL - REAL, FISCAL groups; specific variables include FFER Shadow Push Factors, Real 10yr US Yield, Trade-Weighted Dollar Index, Capital Adequacy Ratio, Loan to Deposit Ratio, Private Debt and Loans to GDP, Real Deposit Rate, HP Output Gap, Inflation, PPP Income Relative to US, Reserves as Percent of ARA Metric, Current account balance, % of GDP, Exports of goods and services, Net non-FDI capital inflows, External Debt to Exports, Oil Price, Growth Rate, Equity Price Growth, 5yr House Price Growth, BIS Credit Gap, Total Debt Growth, Debt Revenue, Fiscal Balance to Fiscal Revenue, Interest Rate on New Pub. Debt., Interest Expense to Revenue.
- Training design and data preparation
  - Training windows expanded iteratively (example: second training period extends to 1980-2001 and tests model using crisis observations two years ahead).
  - Hyperparameters tuned at each horizon using AUCs in time-block cross validation on training set.
  - Dataset balanced using SMOTE due to imbalanced crisis episodes.
  - Final real-sector model trained using 1980-2016 data.
- Model performance and selection
  - Random forest models usually outperformed other models.
  - Goodness of fit measures more volatile for signal extraction and logit; SVM performs poorly.
  - RF class selected as core model for predicting real sector crises.
  - Variable importance calculated using Shapley values.
- Variable importance and heterogeneity across country groups
  - Advanced Economies: most important variables are global variables associated with US policy stance, followed by real and financial sectors.
  - Emerging Markets: mixture of global, real, external, fiscal, and financial-real variables.
  - Low-Income Countries: fiscal sector variables important alongside global and real sector variables.
  - Specific high-importance variables listed (as labeled in figures): Pushfactors, 1yr diff; Real 10yr US Yield; 10-yr USD real rate, 1-yr diff; 5yr house price growth; PPP income relative to US; GDP, export weighted, diff (5yr avg); Real output growth; Corp debt sub-inv. grade; Inflation; Short-term nominal deposit rates, 5yr log diff; External debt to exports, diff (5yr avg); CA to GDP ratio, diff (5yr avg); Non-FDI liabilities inflow; Reserves; Capital adequacy ratio; Int. Rate on New Pub. Debt.; Exchange rate, 1yr log diff; HP Output Gap; Terms of Trade, 1yr log diff; REER, log diff; Debt revenue; Fiscal balance revenue, 1yr diff; CA balance, 1yr diff; Oil price, growth rate.
- Nonlinear interactions and interpretability
  - Interactions captured by ML but not by traditional models.
  - Example: for a given level of negative domestic GDP growth, domestic growth becomes a more important predictor of real sector crises when the federal funds shadow rate is higher—tight global financial conditions amplify impact of negative domestic growth.
  - Shapley values used to calculate contributions and visualize nonlinear interactions.
- Illustrative country cases and sequencing
  - Ethiopia (late 1980s to early 1990s): in LICs where financial sectors are less developed and fiscal sectors play larger roles, real sector crises typically preceded by fiscal crises.
  - Greece (2010s): in AEs, real sector crises may come hand-in-hand with financial crises, triggering subsequent crises in real and external sectors.
  - Risk index spikes 1-2 years ahead of corresponding crises across the four sectoral models in both examples.

### Training, robustness, and technical practice
- Hyperparameter tuning via time-block cross validation using AUC on training sets.
- Class imbalance handled with SMOTE (Chawla and others, 2002).
- Missing data imputation choices driven by coverage trade-offs; median imputation used widely for robustness and interpretability.
- Shapley values used broadly for prediction attribution; similarity scores derived from RF node co-occurrence.

### Conclusions and next steps
- ML models represent next generation of VE risk-assessment models.
- Models with best horse race performance will underpin country risk assessments in four sectors.
- ML results complement existing models and provide more comprehensive, granular representation of risk.
- Opportunities and directions:
  - Multi-classification models to assess combinations of crises.
  - Move to bigger data, higher-frequency series, and less structured information for short-horizon crisis risk assessment.
  - Leverage progress in causal identification with ML to measure consequences of crises and identify successful prevention methods.

### Annex I — Signal extraction method (summary)
- Establishes thresholds on an indicator-by-indicator basis.
- Threshold choice: minimize sum of percentage of crises missed and percentage of non-crises falsely flagged (false alarms) — equivalent to maximizing difference between cumulative distribution functions of crisis and non-crisis samples.
- Countries receive a 1 if indicator falls on risky side of threshold and 0 otherwise.
- Indicators aggregated with weights equal to signal-to-noise ratio = (1-z)/z, where z is sum of fractions of false alarms and missed crises.
- Model robust to outliers, accommodates missing data, and resembles boosted trees methods with very short trees.

*IMF staff calculations; Technical Notes and Manuals 21/03 | 2021*

### Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

### Executive Summary

### Purpose and scope of the Vulnerability Exercise (VE)
- The VE is a cross-country, bottom-up, multi-sectoral exercise that identifies country-specific near-term macroeconomic risks.
- Assessments reflect the judgement of country teams informed by consistent, cross-country quantitative models as well as country-specific context.
- The VE is a key element of the Fund’s broader risk architecture and covers vulnerabilities in the fiscal, external, financial, and non-financial corporate and household sectors.

### Rationale for model updates and coverage improvements
- The new generation of VE models closes gaps in risk and country coverage from previous models, while improving consistency and comparability of risk assessments across countries.
- Earlier models evolved organically and assessed advanced economies, emerging markets, and low-income countries separately; the new models:
  - assess risks in the external, fiscal, financial, and real sectors for all IMF member countries;
  - are applied consistently across country groups based on identical, quantitative crisis definitions;
  - are assessed within a single model when effective to improve evenhandedness.

### Use of machine-learning (ML) algorithms
- The new generation of VE models leverages machine-learning (ML) algorithms to better capture:
  - interactions between different parts of the economy;
  - non-linear relationships that are not well measured in “normal times”;
  - robustness to outliers, noise, and the diversity of experiences across countries.
- ML focus: prediction and generalization to out-of-sample observations rather than only in-sample inference.
- Non-parametric ML methods (e.g., random forests) can explore datasets to find predictive relationships and interactions; the signal extraction approach is noted as a robust non-parametric precursor.

### Model evaluation and selection strategy
- Models are evaluated in a horse-race format using rigorous out-of-sample performance metrics.
- The models assess the near-term risk of a crisis in the external, financial, fiscal, and real sectors.
- Findings from the horse races:
  - Random forest-based models outperform other options in most cases.
  - In other cases, the signal extraction approach performs best.
  - No single algorithm dominates in crisis prediction (”no free lunch theorem”).
- Model selection, crisis definitions, and explanatory variables are standardized:
  - Crisis definitions are based on quantitative criteria drawn from the literature rather than determined by IMF country teams.
  - Variable selection is based on an extensive literature review and includes wider coverage (global and political factors).
  - All models are estimated on annual data to maximize coverage.

### Interpretation and communication of ML outputs
- ML techniques can appear as a black box; the paper develops direct, transparent methods for communicating model results to country teams.
- Shapley values are highlighted as a method for determining additive contributions to a risk assessment from a model’s input series (see Annex VI).
- Communication tools are designed to help move from predicting to informing policy-relevant judgements.

### Relationship to other IMF tools and institutional role
- VE sectoral risk assessments complement other IMF tools published in staff reports or multilateral surveillance products such as the Global Financial Stability Report (GFSR) or External Sector Report (ESR).
- Other tools (debt sustainability assessments, external balance assessments, Growth-at-Risk models) sometimes consider longer horizons or offer deeper country-specific analysis; VE emphasizes:
  - a snapshot across all countries and sectors at a single point in time;
  - crisis risk, cross-country consistency, and model performance.

### Limitations and caveats
- ML models cannot predict crisis events in which the shocks or transmission channels have not been seen before.
- Data availability restricts the use of Big Data models in macroeconomic applications.
- Country teams’ judgement remains central, especially when idiosyncratic factors are important.

*IMF staff calculations; Technical Notes and Manuals 21/03 | 2021*

### Annex I), logit models (classic, Bayesian, and penalized versions such as Elastic Net, Ridge,

### tnmea2021003 - Annex I), logit models (classic, Bayesian, and penalized versions such as Elastic Net, Ridge,

### Methods and models evaluated
- Models: signal extraction, logit (classic and Bayesian), penalized logit (RIDGE, LASSO, Elastic Net), tree-based models (random forest (RF) and variations: balanced random forest, RUSBoost, ADABoost, gradient boosting), support vector machines (SVM).
- Neural Networks: tested in limited experiments; not successful due to small dataset sizes.
- Boosting: ADABoost and RUSBoost are examples; boosting trains subsequent learners on errors made by previous learners.
- Tree-based models: discussed further in Annexes II and III; viewed as an extension of signal extraction and shown promising results in crisis forecasting literature.
- Variable treatment: higher-frequency variables included though models estimated on annual data; missing data for tree-based models imputed using the sample median.

### Performance assessment and evaluation procedure
- Primary evaluation: out-of-sample predictive performance using classic backtesting simulating a “real time” forecasting exercise beginning in the early 2000’s and rolling forward estimation and testing periods.
- Complementary evaluation: cross-fold validation (time-block folds of consecutive years) to respect time-series properties and avoid overfitting to recent crises.
- Main metrics: Area Under the Curve (AUC); also sum-of-errors and probability-oriented approaches when appropriate.
- Backtesting caveat: classic backtesting may effectively test models mainly against the Global Financial Crisis (GFC) in some sectors; complemented by cross-fold validation and single-cutoff backtesting in some exercises.

### Estimation sample and pooling
- For each sector, assessment made whether a single pooled model across all countries yields similar or better performance.
- Pooling findings:
  - Pooling country income groups often improves predictive performance.
  - Pooling all countries typically improves forecasting for crises in LICs and AEs and provides similar out-of-sample performance for EMs.
  - For external sector risks, modeling by conventional income groups is preferred for comparability.

### Key model results (general)
- Machine-learning models regularly outperform classical econometric methods.
- Tree-based models often add value in financial and fiscal sectors; signal extraction often robust in other cases.
- Variable interactions are important: interactions between global and domestic risk factors significantly affect crisis risk assessments (examples: external debt, debt growth, external exposure of banking sector interacting with global interest rates; foreign liability growth amplified by rapid domestic credit expansion).

### Communicating results and interpretability
- Interpretability tools: Shapley values used to decompose risk assessments into additive contributions of variables; contributions visualized in waterfall charts.
- Box plots: used to compare distributions of key indicators across crisis and non-crisis cases; visualization uses Winsorized whiskers and specified percentile plotting rules.
- Similarity scores: derived from RF node co-occurrence to identify countries with similar financial-sector risk profiles (example: Ireland 2005 similar to United Kingdom, Greece, Spain).

### External sector model (sudden stops and EMP)
- Crisis definitions:
  - Sudden stops: net private capital inflows as % of GDP at least 2 percentage points lower than previous year and two years before, and country approved to tap large IMF financial support.
  - Sudden stops with growth impacts (SSGI): large growth declines when change in growth relative to previous five-year average lies in the lower 10th-percentile.
  - EMP events: weighted average index of annual % depreciation in nominal exchange rate and annual decline in reserves as % of previous year’s GDP; EMP events defined when index lies in the lower 15th-percentile and country approved for large IMF support.
- Data coverage:
  - SSGI sample: 53 EMs, period 1990-2017; 183 sudden stops = 12.3 percent of the sample, and 61 SSGIs = 4.1 percent of the sample.
  - EMP database: 192 countries, 1990-2017; average incidence 6.3 percent.
- Explanatory variables: final model based on 79 variables spanning first-, second-, third-generation crisis model variables, political shocks, current account shocks, contagion effects.
- Model selection and testing:
  - Models evaluated for 1–2 year ahead crises using backtesting starting in 2007 and single-cutoff backtesting (estimated through 2007 and applied to rest of sample); main metric is sum of errors, with AUC for reference.
- Performance findings:
  - SSGI: signal extraction model performs best in terms of out-of-sample AUC and sum of errors compared with Logit (LASSO, RIDGE, Elastic Net) and tree-based (RF, RUSBoost).
  - EMP: no single model consistent best. Signal extraction best for AEs (by sum of errors), RUSBoost best for EMs, RF best for LICs. Practical implementation: signal extraction for EMP in AEs; RF models within income groups for EMP in EMs and LICs. For EM oil exporters, pooled RF (EMs and LICs including oil exporters) applied due to small sample size.
- Variable importance and drivers:
  - SSGIs driven by variables from all sectors; emphasis on asset bubble building and busting (stock and housing price variables), global shocks and contagion (TED spread, frequency of AE banking crises), and country-specific external variables (external debt shocks, current account shocks).
  - EMP drivers vary by group:
    - Advanced Economies: external sector important; banking sector external exposure leading factor.
    - Emerging Markets: reserves growth, export growth, domestic credit growth; global shocks/contagion not among key drivers.
    - Low-Income Countries: 5yr inflation important; global shocks (TED spread, US term premium, Fed rate, dollar appreciation) among top predictors.
- Variable interactions:
  - RF models reveal interactions, e.g., foreign liability share matters more for external risk when accompanied by large 5-year cumulative private credit-to-GDP growth.

### Fiscal sector model
- Crisis definition: fiscal crisis if any of four criteria met (with minimum two years gap between crises):
  1. Credit event: default, restructuring, or rescheduling of substantial size (>0.5 percent of GDP p.a.; defaulted nominal amount grows ≥ 10 percent p.a).
  2. Exceptionally large official financing: High-access IMF financial arrangement (≥ 100 percent of quota) or EU program.
  3. Implicit domestic public default: high inflation or steep increase in domestic arrears (thresholds listed).
  4. Loss of market confidence: high sovereign spreads or loss of market access (thresholds differ by AMs, EMs, LICs; example: level of spreads ≥ 1,000 bps for AMs; annual change thresholds listed).
- Dataset: 188 economies, 1980–2017.
  - On average, countries have undergone two fiscal crises since 1980; LICs > 3 crises on average; AEs < 1 crisis on average.
  - Crisis duration: EMs and LICs longest episodes—on average five years.
- Explanatory variables: final model based on 106 variables grouped into 11 categories (fiscal, three debt groupings, real sector, two external groupings, volatility, institutions, crisis history, country grouping, others). Total permutations used in analysis extend to 290 variables.
- Model selection and testing:
  - Evaluated Logit, Signal Extraction, RF, Elastic Net, gradient boosted trees, ADABoost, RUSBoost, Bayesian Logit, LASSO.
  - Testing via classic back-testing starting in 2000, re-estimating adding a year of data each iteration; accuracy assessed with likelihood score and AUC.
  - RF model consistently performs best or among best for pooled sample across income groups.
- Variable importance and drivers:
  - RF identifies external public debt (ratio to GDP and to exports) as the most important predictor after crisis history and development level.
  - Other top predictors: GDP per capita, reserve coverage of imports, inflation volatility, quality of bureaucracy.
  - Aggregated importance: country-specific external sector variables and public-debt-related variables most important, followed by real variables.
- Illustrative case:
  - Greece 2009: out-of-sample forecast (model estimated through 2007) highlights stock of public debt (external public debt), current account pressures, and debt service as key drivers; model yields a 13 percent probability of default for Greece 2009 based on data through 2007.

### Financial sector model
- Crisis definition: banking sector crises per Laeven and Valencia (2018): significant financial distress plus significant policy interventions.
- Dataset: 117 countries, 1980–2017.
  - Crisis frequencies average one crisis every 30 to 45 years across income groups.
  - Crisis timing varies: 1980s–1990s EMs and LICs frequent; post-2000 crises concentrated in GFC and in AEs.
- Explanatory variables: final model based on 31 variables, including 7 financial sector variables and 9 reflecting financial–real sector interaction. Variables span Global, Financial, Financial-Real, Real, External, Fiscal categories (examples listed exactly as in source: Real 10yr US Yield, 3mo T-bill Rate HP gap, External Bank Liabilities to GDP, Loan to Deposit Ratio, Capital Adequacy Ratio, Avg. Bank Prob. Default, etc.).
- Model selection and testing:
  - Evaluated SVM, RF (balanced), Logit, RIDGE/LASSO Logit, Signal Extraction, ADABoost, RUSBoost.
  - Testing: classic backtesting estimated through 2000 and then re-estimated adding a year; applied to three out-of-sample years per estimation; complemented by cross-validation on groups of consecutive years.
  - Balanced RF model consistently performs best or among best: best in cross-validation and second-best in backtesting.
  - Robustness: similar results for 1- and 3-year horizons and datasets with 84, 160, and 949 variables.
- Variable importance and drivers:
  - Balanced RF emphasizes Real 10yr US Yield (proxy for global financial conditions), followed by external bank liabilities to GDP, 3m US TBill Gap, gross reserves to GDP, external debt to exports, total debt growth, corporate debt sub-investment grade share, and other indicators (as listed in Figure 21).
  - Risk drivers span all sectors; global interest rates and external banking sector exposures play central roles.
- Variable interactions:
  - Interaction example: contribution of external debt to assessed risk increases markedly when Real US 10-year bond rates are higher; risks rise as external debt exceeds 15 percent with magnification under higher global interest rates.
- Aggregate risk:
  - Historical average financial risk index shows sustained levels through mid-1990s, lull mid-2000s, spike during GFC, lull afterward; recent uptick driven by global variables, notably 3-month US T-Bill gap exceeding 1% by end-2018.

### Real sector model
- Crisis definition: real sector crises defined using 4 GDP series and 4 thresholds (country annual growth rate; cumulative growth over past three years; growth relative to most recent five-year average; average GDP level relative to previous three-year average), focused on GDP per working-age person.
  - For each series, crises flagged if below 10th percentile within groups: i) all countries, ii) same income group (AEs, EMs, LICs, WEO 2020), iii) income group in 1980 with extra small-population category, iv) same tercile of total sample for year-on-year growth volatility.
  - Sixteen crisis criteria (4 series × 4 thresholds); a country-year recorded as a real sector crisis when nine or more indicators signal crisis.
- Historical clustering: four clusters—early 1980s (LICs and EMs, Latin American debt crisis), early 1990s, late 1990s (Asian Financial Crisis), late 2000s (GFC).
- Dataset: average of two real sector crises per country over sample; average crisis duration two years.
- Explanatory variables: global variables (short- and long-term US interest rates, oil price), domestic real sector variables (inflation, output gap), external variables (REER, terms of trade, external debt, reserves), fiscal variables (debt-to-revenue, fiscal deficit); variables constructed as levels, differences, growth rates, and one-sided HP trends.
- Model selection/testing: two-stage selection beginning with logit, signal extraction, SVM, RF in backtesting (1980-2016 sample with first training window 1980-2000 and testing 2001 onwards).

*Source: https://www.imf.org/-/media/files/publications/tnm/2021/english/tnmea2021003.pdf*

### 2002. The second training period extends the window by one year, to 1980-2001, and tests the

### tnmea2021003 - 2002. The second training period extends the window by one year, to 1980-2001, and tests the

### Training design and data preparation
- Training windows are expanded iteratively: the second training period extends the window by one year, to 1980-2001, and tests the model using the crisis observations two years ahead; and so on.
- Hyperparameters are tuned at each time horizon using AUCs in a time-block cross validation exercise on the training set.
- Because crisis episodes in the samples are imbalanced, the data set is balanced using the Synthetic Minority Over-sampling Technique (SMOTE) (Chawla and others, 2002).
- The final model is trained using 1980-2016 data.

### Explanatory variables used in the real-crisis model
- GLOBALFINANCIALREAL
  - FFER Shadow Push Factors
  - Real 10yr US Yield
  - Trade-Weighted Dollar Index
  - Capital Adequacy Ratio
  - Loan to Deposit Ratio
  - Private Debt and Loans to GDP
  - Real Deposit Rate
  - Short-Term Nominal Deposit Rate
  - HP Output Gap
  - Inflation
  - Natural Disasters, Material Impact
  - PPP Income Relative to US
  - Real GDP Growth, Export Weighted
  - Real GDP Growth
- EXTERNAL
  - Reserves as Percent of ARA Metric
  - Current account balance, % of GDP
  - Exports of goods and services
  - Net non-FDI capital inflows, % of GDP
  - Exchange Rate (National Currency per US Dollar)
  - Real Effective Exchange Rate
  - External Debt to Exports
  - Oil Price, Growth Rate
  - Terms of Trade
- FINANCIAL - REAL
  - Equity Price Growth
  - 5yr House Price Growth
  - Corp Debt Sub-Inv. Grade
  - BIS Credit Gap
  - Total Debt Growth
- FISCAL
  - Debt Revenue
  - Fiscal Balance to Fiscal Revenue
  - Interest Rate on New Pub. Debt.
  - Interest Expense to Revenue

### Model performance and selection
- Random forest models usually outperformed other models.
- Goodness of fit measures are more volatile for the signal extraction and logit regression models.
- SVM performs poorly.
- Building on the expanding window analysis, the RF class is selected as the core model for predicting real sector crises.
- Model results: variable importance is calculated using Shapley values.

### Variable importance and heterogeneity across country groups
- Global variables are important for predicting crisis episodes, but importance differs by country group:
  - Advanced Economies (AEs): most important variables are global variables associated with the policy stance in the US, followed by variables in the real and financial sectors.
  - Emerging Markets (EMs): mixture of global, real, external, fiscal, and financial-real variables (as shown in the variable importance plots).
  - Low-Income Countries (LICs): fiscal sector variables are also important besides global and real sector variables.
- Specific high-importance variables cited (as labeled in figures):
  - Pushfactors, 1yr diff
  - Real 10yr US Yield
  - 10-yr USD real rate, 1-yr diff
  - 5yr house price growth
  - PPP income relative to US
  - GDP, export weighted, diff (5yr avg)
  - Real output growth
  - Corp debt sub-inv. grade
  - Inflation
  - Short-term nominal deposit rates, 5yr log diff
  - External debt to exports, diff (5yr avg)
  - CA to GDP ratio, diff (5yr avg)
  - Non-FDI liabilities inflow
  - Reserves
  - Capital adequacy ratio
  - Int. Rate on New Pub. Debt.
  - Exchange rate, 1yr log diff
  - HP Output Gap
  - Terms of Trade (ToT), 1yr log diff
  - REER, log diff
  - Debt revenue
  - Fiscal balance revenue, 1yr diff
  - CA balance, 1yr diff
  - Oil price, growth rate

### Nonlinear interactions and interpretability
- Variable interactions are important and captured by the ML model but not by traditional models.
- Illustration: interaction between domestic real GDP growth and global financial conditions (federal funds shadow rate)
  - For a given level of negative domestic GDP growth, domestic growth becomes a more important predictor of real sector crises when the federal funds shadow rate is higher.
  - This suggests tighter global financial conditions amplify the impact of negative domestic growth on the likelihood of a real sector crisis.
- Shapley values are used to calculate contributions to assessed risk and to visualize these nonlinear interactions.

### Illustrative country cases and sequencing of crises
- Example country risk-index evolutions:
  - Ethiopia (late 1980s to early 1990s): risk indices show that in LICs where financial sectors are less developed and fiscal sectors play a larger role, real sector crises are typically preceded by fiscal crises.
  - Greece (2010s): in AEs, real sector crises could come hand-in-hand with financial crises, triggering subsequent crises in real and external sectors.
- In both Ethiopia and Greece, the risk index spikes 1-2 years ahead of each corresponding crisis across the four sectoral models.

### Conclusions and next steps
- The machine-learning models presented represent the next generation of risk assessment models in the VE.
- Models with the best horse race performance will underpin country risk assessments in the four sectors explored.
- These ML results complement models already in use (Ahuja and others, 2017) and help provide a more comprehensive and granular representation of the risk conjuncture.
- Opportunities and directions:
  - Multi-classification models to assess combinations of crises.
  - Move into bigger data, higher frequency series, and less structured information for short-horizon crisis risk assessment.
  - Leverage recent progress in causal identification with ML techniques to better measure consequences of crises and identify successful prevention methods.

### Annex I — Signal extraction method (summary of approach)
- The signal extraction model establishes thresholds on an indicator-by-indicator basis.
- Threshold choice: minimize the sum of the percentage of crises missed and the percentage of non-crises falsely flagged as a crisis (false alarms). This is equivalent to maximizing the difference between the cumulative distribution functions of the crisis and non-crisis samples.
- Example trade-off: if crises are 5 percent of the sample, missing one crisis is as costly as issuing 19 false-alarms.
- Countries receive a 1 if their value of the indicator falls on the risky side of the threshold and a zero otherwise.
- Indicator results are aggregated with weights given by their signal-to-noise ratio – defined as (1-z)/z, where z is the sum of the fractions of false alarms and missed crises.
- The model is robust to outliers, accommodates missing data, and resembles boosted trees methods with very short trees.

*Source: tnmea2021003 - 2002. The second training period extends the window by one year, to 1980-2001, and tests the model using the crisis observations two years ahead; and so on.*

### ANNEX II  FROM DECISION TREES TO A RANDOM FOREST

### ANNEX II  FROM DECISION TREES TO A RANDOM FOREST

### Decision trees: concept and mechanics
- Decision trees deliver a structured set of yes/no questions to predict a particular outcome (example given: “the likelihood of a crisis in the next two years”).
- Core operation: repeated binary partitioning of the predictor space into two sets, starting with the initial split that decreases the prediction error the most.
- The algorithm “considers every possible split on every possible predictor variable, and chooses the one split on the one variable that best separates the sample into the two most dissimilar subsamples (based on the predicted outcome).”
- Partitions are recursive: each subsequent split only considers the subsample under which it falls.
- Key attraction: transforms complex, non-linear problems with many predictors into an easily understood flowchart of yes/no questions.
- Practical strengths and weaknesses:
  - Well suited to important nonlinearities and interactions.
  - Can cope with missing data.
  - Tend not to work as well if the underlying relationship is linear, though they may reveal aspects not apparent from linear approaches.

### Overfitting and traditional mitigation
- A common problem: decision trees often fit the training sample well but perform poorly out-of-sample (overfitting).
- Traditional mitigation: shorten or “prune” the tree by imposing a penalty for overly long/complex structures; choose degree of complexity using cross-validation techniques.

### Random Forest (RF): core modifications and intuition
- RF modifies decision trees to minimize overfitting by growing numerous unpruned trees and combining their results (reference: Breiman, 2001a).
- First modification: bootstrap aggregation (“bagging”).
  - “An individual tree is built on a random sample of the dataset, roughly two thirds of the total observations—the remaining one-third are referred to as out-of bag (OOB) observations and can be used to gauge the accuracy of the tree.”
  - This is repeated “hundreds or thousands of times.”
  - Prediction for a new instance: feed the instance through each tree and aggregate predictions (example aggregation: “majority vote”).
  - Rationale: unpruned trees are weak models that mix signal and noise; by averaging a large ensemble, the algorithm “is essentially exploiting the law of large numbers to average out the noise, leaving only the signal.”
- Second modification: random subset of predictors at each split.
  - At each split, the algorithm only considers a random subset of predictors (usually “the total number of predictors divided by three”).
  - Purpose: avoid multiple similar trees when predictors are highly correlated or one predictor dominates, thus ensuring diversity among trees.
  - Each tree is grown on a deliberately limited dataset and is weak on its own; combined, the uncorrelated weak models yield a strong aggregate prediction.

### Practical advantages of Random Forests
- Require almost no input preparation: can handle binary, categorical, and numerical predictors without scaling.
- Implicitly incorporate an element of feature selection and provide an indicator of relative variable importance.
- Quick to train and applicable to classification, regression, and cluster analysis.
- Predictive performance is usually impressive; RF often performs well with relatively less time and effort and is frequently used as a benchmark model.

*Annex prepared by Andrew Tiffin.*

### Annex prepared by Andrew Tiffin.

### Annex prepared by Andrew Tiffin

### Shapley Values and contribution attribution
- Illustration (from toy example):
  - Alice Pays / Bob Pays scenarios: $6, $42, $6$36, $0$42, $0 + $6, 2, = $3, $36 + $42, 2, = $39
  - The Shapley Value averages contributions over all conceivable sequences (in this case there are only two) to arrive at the fairest possible allocation.
- Conceptual mapping to prediction attribution:
  - Payoff = the actual prediction for a particular instance less the average prediction for the entire dataset (Sample Average).
  - Players = the values of each variable that fed into that prediction.
  - Shapley Values divide the prediction (payoff) among the variables in a way that fairly represents their contributions across all possible subsets of variables.
  - Example use case: in a crisis prediction model, if a country’s crisis probability is higher than for the sample as a whole, Shapley Values indicate which variables prompted the model to assign a higher probability (compared to the sample average) and quantify each variable’s relative contribution.
  - Shapley values allocate credit for joint work among contributing variables; the value for a particular variable (e.g., external debt) can differ for two countries with the same value of that variable but different interacting variables.

### Missing data: options, trade-offs, and VE practice
- Four general options when faced with missing data:
  1. Drop all observations with missing variables.
  2. Drop all variables with missing observations.
  3. Simple Imputation (e.g., mean, median, or mode).
  4. Model-Based Imputation (e.g., linear regression, k-nearest neighbor, random forest; often iterative chains when multiple variables are incomplete).
- Empirical consequences of complete-case analysis (dropping all incomplete observations):
  - Would eliminate 80% of the sample for the EMP AE model.
  - Would eliminate 89% of the sample for EMP EM.
  - Would eliminate almost 98% for EMP LIC.
  - Would eliminate more than 98% for the financial sector model.
  - Example impact: in the fiscal sector, complete case analysis would reduce the fiscal model’s AUC by 10 percent for low income countries where data availability is most challenging.
- VE teams’ adopted approach and rationale:
  - Most VE models use a combination of the second and third approach (drop variables with excessive missingness; simple imputation for remaining gaps).
  - Fiscal model rule: variables are dropped if more than one-third of their observations are missing; remaining gaps are imputed using the variable’s median value.
  - Median preferred because it is robust to outliers and simpler to communicate.
  - Communication implication: using the sample median for a missing value (e.g., international reserves) effectively prevents that variable from contributing to that country’s crisis score in that year (its value equals the sample and thus is less likely to generate a significant Shapley value).
  - Contrast with model-based imputation: if a model-based approach is used, the predicted value for the missing variable could itself drive the crisis score and require understanding both the primary crisis model and the secondary background model that produced the imputation.
- Guiding principle:
  - No ideal method exists; different approaches have different strengths. Given the nature of available data and the VE objective (to aid desks in assessment of crisis risks), VE teams generally opted for a simpler, more robust approach.
  - Each model faces specific missing-data challenges; details of chosen solutions are provided in respective working papers.
- Notes and cross-references:
  - Footnote 52: For an introduction to this literature, see Azur and others (2011).
  - Footnote 53: For example, the signaling approach chosen for the external model is proof against missing values, but imputation is nonetheless employed to ensure consistent Shapley values.

### Fiscal sector explanatory variables (grouped lists as presented)
- FISCALEXTERNALDEBT-PRIVATE
  - General government expenditures in percent of GDP
  - Net official development assistance in percent of GDP
  - One sided credit gap based on the GDD loans and securities
  - General government primary expenditures in percent of GDP
  - Current account balance in percent of GDP
  - Total Debt, loans and securities, in percent of GDP
  - Overall balance in percent of GDP
  - Export of goods and services in percent of GDP
  - General government primary balance, percent of GDP
  - Import of goods and services in percent of GDP
- DEBT-PUBLIC
  - General government revenues in percent of GDP
  - Personal remittances in percent of GDP
  - General government short-term external debt in percent of GDP
  - Current account without import
  - Public external debt in percent of GDP
- GLOBAL
  - Net foreign direct investment in percent of GDP
  - Public debt in percent of GDP
  - Percent change of crude oil price
  - Other investment, net (loans, deposits, insurance, pensions, trade credits, SDR, percent of GDP
  - Public debt in percent of general government revenue
  - Percent change of Non-fuel price
  - Portfolio investment, net
  - Public external debt to export
  - Percent change of food price
  - Percent change of exchange rate (NC units per U.S. dollar, period average Units)
  - General government interest expenses in percent of GDP
  - US T-Bill rate Percent
  - Exchange Rate, end of period
  - Amortization of external public debt in percent of GDP
  - VIX Index Period Average
  - Average of the last 10 year of the sum of export and import of goods and services
  - Public debt service to revenue in percent
  - VIX Index Period End
  - Percent change in real exchange rate, period average
  - Public debt service to export in percent
  - Percent change of VIX Index Period Average
  - Log of PPP exchange rate
  - Percent change of VIX Index Period End
  - PPP overvalue
- TOTAL DEBT
  - US T-Note 5-year rate Percent, Period Average
  - Percent change in total reserves excluding gold in national currency
  - Total debt in percent of GDP
  - US T-Note 10-year rate Percent, Period Average
  - Percent change in total reserves (number of months of imports)
  - US T-Note 5-year rate Percent, End of Period
  - Percent change in terms of trade (of goods and services) Index
  - US T-Note 10-year rate Percent, End of Period
  - Trading Partner Growth (Real GDP, 2005=100, local currency, Weighted by trade exports to all economies)
  - World real GDP growth, in percent
  - Percent change of trading Partner Import Demand (Imports volume of goods and services, 2005=100, Weighted by trade exports to all)
  - Geometric average of the last 3-year world GDP growth
  - Net official development assistance in percent of GDP
  - Current account balance in percent of GDP
- CRISIS HISTORY
  - Export of goods and services in percent of GDP
  - Number of crisis
  - Import of goods and services in percent of GDP
- VOLATILITY
  - Standard deviation of real GDP growth
  - Number of Crisis, Advanced and Emerging Economies
  - Personal remittances in percent of GDP
  - Standard deviation of the change in terms of trade
  - Number of Crisis, Emerging and Low-Income Economies
  - Current account without import
  - Standard deviation of the percent change in exchange rate (end of period)
  - Number of Crisis, Advanced Economies
  - Net foreign direct investment in percent of GDP
  - Standard deviation of the inflation
  - Number of Crisis, Emerging Economies
  - Other investment, net (loans, deposits, insurance, pensions, trade credits, SDR, percent of GDP)
  - Standard deviation of the percent change in exchange rate (end of period)
  - Number of Crisis, Low Income Economies
  - Portfolio investment, net
- COUNTRY CATEGORY
  - Dummy: Monetary Union
  - Dummy: Advanced economies
  - Dummy: Emerging market economies
  - Dummy: Low income economies
  - Dummy: Small developing state
  - Dummy: Fuel exporter
  - Dummy: Fragile state
  - Dummy: Fuel exporter or VELIC commodity exporter
- REAL INSTITUTIONS, ELECTIONS
  - Percent change in Material impact - all natural disaster hazards
  - Percent change of real GDP per capita
  - Percent change in real GDP
  - Growth deviation from past 5-year average
  - GDP growth rate
  - Years remaining in current chief executive’s term
  - Geometric average of the last 3-year GDP growth
  - Percent change of Consumer Price Index, period average
  - Percent change of CPI, end of period Units
  - R minus G
  - Log GDP per capita (PPP), relative to US
  - Log GDP per capita (PPP)
  - Log GDP in USD
  - Log of population
  - Revised Combined Polity Score (single regime score, runs from 1 (full democracy) to -1 (full autocracy))
  - Checks and balances index
  - Bureaucracy Quality
  - Corruption
  - Legislative election held dummy variable
  - Executive election held dummy variable
  - Percent change in Material impact - all natural disaster hazards

*International Monetary Fund, TNM/21/03 — Annex prepared by Andrew Tiffin*

---


_Source: https://www.imf.org/-/media/files/publications/tnm/2021/english/tnmea2021003.pdf_