## wpiea2021197-print-pdf ## Source details **Canonical URL:** [wpiea2021197-print-pdf](https://www.imf.org/-/media/files/publications/wp/2021/english/wpiea2021197-print-pdf.pdf) ## Other formats - [Markdown version](/-/media/files/publications/wp/2021/english/wpiea2021197-print-pdf.pdf.md) - [Structured JSON version](/-/media/files/publications/wp/2021/english/wpiea2021197-print-pdf.pdf.json) --- ### Introduction and main findings - Recessions associated with banking and financial crises: median cumulative deviation of GDP from trend 3 years after banking crises is 35 percent for high-income economies (HIE) and 14 percent for low- and middle-income economies (Laeven and Valencia, 2018). - Tight financial conditions associated with higher-than-average borrowing costs, lower-than-average house price growth and lower-than-average credit growth (IMF, 2017). - Paper goals: - Assess whether Financial Soundness Indicators (FSIs) can detect tight financial conditions early enough. - Motivate because tight financial conditions lead to lower economic activity whether or not they culminate in crises. - Key preview findings: - Tight financial conditions can be accurately detected in 1 to 3 years ahead with the use of FSIs. - Example: leverage (assets to Tier 1 capital) exceeding 8.3 issues signals that 80 percent of times correctly detect tight financial conditions 4 to 12 quarters ahead. - Combining several variables yields superior accuracy compared to single-variable signals. - Most accurate signals arise when capital adequacy variables are low and earnings and profitability of deposit taker institutions are high. - Capital adequacy variables dominate variables related to non-financial sector indebtedness in terms of signal accuracy. ### Data and definitions - Dependent variable: - IMF (2017) Financial Conditions Index (FCIs) used; FCIs available for 43 economies from 1990 to 2016. - Tight financial conditions defined by converting continuous FCI into binary: binary = 1 whenever FCI reaches the 90th percentile of the country’s FCI distribution. - Rationale: proportionality assumption and alignment with crisis shares in literature (Laeven and Valencia (2018) find 12.5 percent of sample are crises; Lo Duca et al. (2017) find 10.5 percent). - Comparison datasets for crises timing and robustness: - LV denotes Laeven and Valencia (2018) — banking crises only, years 1970-2017. - LD denotes Lo Duca et al. (2017) — European countries, 1970-2016. - FSIs and additional variables: - FSIs split into core (deposit taker institutions) and additional (other financial institutions, non-financial private sector, real estate market). - FSI data cover almost up to 30 years and more than 100 countries; compiled under internationally agreed methodology. - The paper complements FSIs with broad macroeconomic and financial variables from 1990 onward where possible. ### Data transformations and sample - Transformations applied to all variables: - One-quarter and one-year change of level (differences) and/or growth rate. - One-sided Hodrick-Prescott detrending with smoothing parameter 휆휆=1600 for quarterly data; cyclical deviation computed for all variables except those related to growth rates, interest rates or spreads. - Country-specific z-score standardization so thresholds are country-specific. - Additional predictors: inflation rate, real GDP, industrial production, debt service cost, current account, financial account, financial market prices, IFS data. - Feature selection and dimensionality reduction: - Variables with AUC confidence interval not overlapping 0.5 and maximization of overlap across variables yielded 19 variables. - Principal Component Analysis used to reduce these 19 variables to 6 principal components explaining 95% of variance. - Sample: unbalanced panel covering periods from 2000 to 2016 subject to data availability; countries included: ARG, AUS, AUT, BEL, BGR, BRA, CAN, CHL, CHN, COL, CZE, DEU, DNK, ESP, FIN, FRA, GBR, GRC, HUN, IDN, IND, IRL, ISR, ITA, JPN, KOR, MEX, MYS, NLD, NOR, PER, PHL, POL, PRT, RUS, SWE, THA, TUR, USA, VNM, ZAF. ### Empirical approach and evaluation - Objective: produce early warning signals about tightening of financial conditions 4 to 12 quarters ahead. - Cross-validation: k-fold cross-validation with 10 folds, repeated 5 times; reported results are averages across repetitions. - Class imbalance handling: - Dependent variable equals 1 in 10% of cases. - Synthetic Minority Over-Sampling Technique (SMOTE) applied to equalize tranquil and tightness counts. - Accuracy metrics: - Receiver Operating Characteristic (ROC) curve and Area Under the ROC Curve (AUC). - True Positive Rate (TPR) and False Positive Rate (FPR) tradeoffs and confusion matrices. - Threshold τ ∈ (0,1) represents cost of missing a true positive; relative cost = τ / (1−τ). Optimal threshold chosen via tangent on ROC curve. ### Methods used to generate signals (seven/eight methods) - Signal extraction (SigExt): univariate binary signals via threshold grid search. - Classification/continuous-probability methods: Decision Tree (Tree), Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), Support Vector Machine (SVM), k-Nearest Neighbors (KNN), Naïve Bayes (NB), Logistic Regression (LR). - Tuning and model choices described (grid searches, kernel choice, regularization parameters, k choice). ### Key univariate empirical findings - FSIs produce high-accuracy signals relative to macro-financial variables. - Most accurate univariate signals: capital adequacy FSIs, followed by real estate market and earnings & profitability variables. - Top-performing predictor noted: Tier 1 Capital to Assets ratio (Chile example). - Selected cross-validated out-of-sample AUC entries for 4-12 quarters horizon (Table 1, preserved exactly): - Tier 1 Capital to Assets raw: SigExt 0.227, Tree 0.836, LDA 0.772, QDA 0.771, SVM 0.768, KNN 0.830, NB 0.790, LR 0.772 (FSI: Yes) - Regulatory Tier 1 Capital to Risk Weighted Assets raw: SigExt 0.222, Tree 0.852, LDA 0.778, QDA 0.813, SVM 0.778, KNN 0.866, NB 0.847, LR 0.778 (FSI: Yes) - Regulatory Capital to Risk Weighted Assets raw: SigExt 0.212, Tree 0.854, LDA 0.788, QDA 0.787, SVM 0.787, KNN 0.854, NB 0.803, LR 0.788 (FSI: Yes) - Residential Real Estate Loans to Total Loans raw: SigExt 0.637, Tree 0.856, LDA 0.635, QDA 0.635, SVM 0.638, KNN 0.892, NB 0.733, LR 0.635 (FSI: Yes) - Net Open Position in Foreign Exchange to Capital c4: SigExt 0.590, Tree 0.783, LDA 0.589, QDA 0.751, SVM 0.775, KNN 0.806, NB 0.776, LR 0.588 (FSI: Yes) - Liquid Assets to Short Term Liabilities hp: SigExt 0.355, Tree 0.767, LDA 0.645, QDA 0.573, SVM 0.647, KNN 0.781, NB 0.702, LR 0.642 (FSI: Yes) - Tier 1 Capital to Assets z: SigExt 0.341, Tree 0.697, LDA 0.658, QDA 0.657, SVM 0.659, KNN 0.729, NB 0.655, LR 0.658 (FSI: Yes) - Note: AUC for signal extraction below 0 indicates variable generates a signal when it achieves low level. - Example threshold trade-off (Regulatory Tier 1 Capital ratio): - Thresholds of 8 and 10 percent assessed as non-informative. - Threshold of 12 percent correctly classifies impending tight financial conditions with TPR 84 percent and FPR 30 percent. - Confusion matrix and τ sensitivity: - For lowest τ, Decision Tree correctly classifies 30.9 percent of true negatives and 39.0 percent of true positives; 30.1 percent misclassified. - Increasing τ (more tightness-averse) raises false alarms: for τ=0.6, 23.5 percent observations are false positives; for τ=0.9, 38.1 percent observations are false positives. ### Multivariate model findings and method comparison - Multivariate signals (PCA-reduced 6 components) outperform univariate signals. - KNN and Tree dominate other methods in terms of signals accuracy based on ROC curves; KNN and Tree can correctly classify most true positives with relatively low false positive rate of 20 percent. - KNN is the best performing method, consistent with Holopainen and Sarlin (2017). - Logistic regression and linear discriminant analysis generate the least accurate signals in multivariate setting. - Representative multivariate performance summaries (Table 2 and Appendix F examples, preserved exactly): - Table 2 — FCIs, full sample: AUC column and τ-dependent triples reported. Selected entries preserved: - Tree: 0.96 0.75 0.10 0.94 0.29 0.12 0.97 0.12 0.14 0.98 0.12 0.14 0.98 - LDA: 0.83 0.67 0.12 0.68 0.65 0.12 0.69 0.54 0.19 0.77 0.54 0.19 0.77 - QDA: 0.87 0.93 0.04 0.58 0.92 0.08 0.66 0.92 0.08 0.66 0.60 0.52 0.98 - SVM: 0.85 0.63 0.12 0.72 0.58 0.13 0.74 0.54 0.15 0.76 0.54 0.15 0.76 - KNN: 0.97 0.67 0.07 0.91 0.67 0.08 0.92 0.54 0.11 0.95 0.54 0.11 0.95 - NB: 0.92 0.74 0.06 0.75 0.72 0.06 0.77 0.61 0.11 0.81 0.60 0.11 0.82 - LR: 0.83 0.63 0.12 0.67 0.59 0.14 0.70 0.59 0.14 0.70 0.31 0.55 1.00 - Appendix F — Multivariate early warning (examples from Tables 11–12): - Table 11 (LV, full sample) example rows: - Tree: AUC 0.99; for τ 0.30 Tr 0.56 FPR 0.03 TPR 0.98; τ 0.4 Tr 0.56 FPR 0.03 TPR 0.98; τ 0.5 Tr 0.56 FPR 0.03 TPR 0.98; τ 0.6 Tr 0.56 FPR 0.03 TPR 0.98. - KNN: AUC 1.00; for τ 0.30 Tr 0.93 FPR 0.01 TPR 0.99; τ 0.4 Tr 0.92 FPR 0.01 TPR 0.99; τ 0.5 Tr 0.92 FPR 0.01 TPR 0.99; τ 0.6 Tr 0.87 FPR 0.02 TPR 1.00. - Table 12 (LD, full sample) example rows: - Tree: AUC 0.99; for τ 0.30 Tr 0.67 FPR 0.02 TPR 0.96; τ 0.4 Tr 0.67 FPR 0.02 TPR 0.96; τ 0.5 Tr 0.37 FPR 0.03 TPR 0.98; τ 0.6 Tr 0.37 FPR 0.03 TPR 0.98. - KNN: AUC 1.00; for τ 0.30 Tr 0.80 FPR 0.01 TPR 0.99; τ 0.4 Tr 0.70 FPR 0.02 TPR 1.00; τ 0.5 Tr 0.70 FPR 0.02 TPR 1.00; τ 0.6 Tr 0.70 FPR 0.02 TPR 1.00. ### Cross-sample, region, and income-group findings - Results robust for high-income economies (HIE) subsample; AUC values on HIE subsample on average higher than full sample. - FSIs generate early signals for banking crises (LV dataset) and financial crises (LD dataset). - Best FSI categories for banking crises: real estate markets, earnings and profitability, capital adequacy, sensitivity to market risk. - Income-group specific best predictors: - Upper middle-income: real estate markets, government bond interest rates, financial institutions claims on private non-financial sector, real economy variables. - Lower middle-income: capital adequacy, foreign currency denominated debt, deposits-to-loan relation, earnings & profitability. - Low-income: government bond interest rates, foreign liabilities of banking institutions, current account. - Housing price-to-rent and price-to-income ratios provide useful signals. ### Policy-relevant implications and recommendations - FSIs can be used successfully for macro-financial surveillance to detect tightening financial conditions 1 to 3 years ahead. - Authorities should closely monitor capital adequacy and earnings & profitability FSIs for early-warning signals. - Multivariate models combining multiple FSIs and macro variables provide substantially superior accuracy compared to single-variable signals. - Choice of threshold τ should reflect explicit consideration of relative costs (τ / (1−τ)) to select an operating point on the ROC consistent with policymaker risk aversion between missing tightness and issuing false alarms. - Clustering countries (e.g., by per capita income) and estimating models on more homogeneous subsets can increase predictive accuracy where data permit. - Logistic regression appears to be a suboptimal choice relative to classification methods (KNN, Tree) for detecting tightening financial conditions. ### Appendices and data inventories (high-level) - Appendix A lists IMF and BIS variables used (selected entries include Regulatory Capital to Risk Weighted Assets; Regulatory Tier 1 Capital to Risk Weighted Assets; Return on Assets; Liquid Assets to Short Term Liabilities; Residential Real Estate Loans to Total Loans; Price To Rent; Price To Income; Debt Service Ratio; Real House Prices; Household Debt to GDP; Government Bonds Interest Rates; Real GDP; Real Industrial Production). - Appendix B: evolution plots of FSIs that gave accurate signals (Figures 12–19). - Appendix C–E: single-variable predictive accuracy tables across methods and samples (examples and exact AUC values preserved in main text and tables). - Appendix F: multivariate early warning models, ROC curves, confusion matrices, and Tables 11–12 reporting cross-validated out-of-sample AUC and τ-dependent Tr, FPR, TPR values. *Source: wpiea2021197-print-pdf (REFERENCES and APPENDICES excerpts as supplied).* ### REFERENCES ____________________________________________________________29 ### wpiea2021197-print-pdf - REFERENCES ____________________________________________________________29 ### Introduction - Recessions associated with banking and financial crises are deeper and more protracted than those associated with other types of crises; median cumulative deviation of GDP from its trend 3 years after banking crises is 35 percent for high-income economies (HIE) and 14 percent for low- and middle-income economies (Laeven and Valencia, 2018). - Tight financial conditions are associated with higher-than-average borrowing costs, lower-than-average house price growth and lower-than-average credit growth (IMF, 2017). - Main goals of the paper: - Assess whether Financial Soundness Indicators (FSIs) can detect tight financial conditions early enough (left arrow in Figure 1). - Motivate this because tight financial conditions lead to lower economic activity whether or not they culminate in crises (right arrow in Figure 1). - Key preview findings: - Tight financial conditions can be accurately detected in 1 to 3 years ahead with the use of FSIs. - Example: leverage (assets to Tier 1 capital) exceeding 8.3 issues signals that 80 percent of times correctly detect tight financial conditions 4 to 12 quarters ahead. - Combining several variables yields superior accuracy compared to single-variable signals. - Most accurate signals arise when capital adequacy variables are low and earnings and profitability of deposit taker institutions are high. - Capital adequacy variables dominate variables related to non-financial sector indebtedness in terms of signal accuracy. - Contributions: - Uses rarely used country-level FSIs with rich information about financial soundness. - Uses recent data less subject to structural changes after the Global Financial Crisis (GFC). - Focuses on tight financial conditions rather than banking/financial crises per se. ### Literature review - Pre-GFC early warning literature emphasized macroeconomic predictors: low real GDP growth, excessively high real interest rates, high inflation, real exchange rate (Demirgüç-Kunt and Detragiache, 1998). - Borio and Lowe (2002, 2004): gap of credit to GDP from its long-run trend is the most useful early warning indicator, followed by asset price index and real exchange rate. - GFC shifted focus toward variables capturing financial sector imbalances: real estate prices, credit aggregates, equity market indices (Borio and Drehmann, 2009). - Novel measures: aggregate debt service ratio (DSR) as proxy for private sector financial burden (Drehmann and Juselius, 2012). - Methodological literature notes substantial differences in predictive power across statistical methods (Holopainen and Sarlin, 2017). ### Data — Financial conditions - Dependent variable: IMF (2017) Financial Conditions Index (FCIs). - FCIs measure ease of financing using variables such as short-term interest rates, credit growth, equity returns, asset price returns and spreads. - FCIs time series available for 43 economies from 1990 to 2016. - Definition of tight financial conditions in this paper: - Continuous FCI transformed to a binary variable. - Binary variable set to 1 whenever FCI reaches the 90th percentile of the country’s FCI distribution. - Rationale: proportionality assumption and alignment with crisis shares in literature (Laeven and Valencia (2018) find 12.5 percent of sample are crises; Lo Duca et al. (2017) find 10.5 percent). - Comparison datasets for crises timing and robustness checks: - LV denotes Laeven and Valencia (2018) — banking crises only, wide country coverage, years 1970-2017. - LD denotes Lo Duca et al. (2017) — European countries, 1970-2016, combines quantitative criteria and expert judgment. - Empirical note: - There is overlap but also discrepancies between FCI-based distress periods and LV and LD crises databases (Figure 2 example shown for the UK). - FCIs capture periods of tightened financial conditions that may not correspond exactly to crisis events in LV and LD datasets. ### Data — Financial Soundness Indicators (FSIs) and other variables - FSIs provide detailed view of the current financial health and soundness of financial institutions and of their corporate and household counterparts. - FSIs include both individual institution data aggregated by country and indicators that are representative of markets in which financial institutions operate. - FSI data cover almost up to 30 years and more than 100 countries. - FSIs are compiled under internationally agreed methodology and measure internal and external financial sector risk exposure due to consolidation. - FSIs are split into core and additional indicators: core covers deposit taker institutions; additional covers other financial institutions, non-financial private sector, and real estate market (see IMF (2019)). - This paper complements FSIs with a broad set of macroeconomic and financial variables covering years 1990 (when possible) onward to construct early warning models. ### Methodological approach (overview) - Construct binary dependent variable from FCIs (90th percentile threshold) to track periods of financial distress. - Evaluate single-variable and multivariate early warning signals using FSIs and macro-financial variables. - Compare predictive accuracy across multiple crises datasets (FCI-based distress, LV, LD) to assess robustness to crisis timing definitions. - Performance assessment tools referenced in figures and tables: ROC curves, AUC, confusion matrices (Figures 4–11; Tables 1–3 and others). ### Key illustrative results and implications (as presented) - Single-variable example: aggregate leverage (assets to Tier 1 capital) - Leverage > 8.3 issues a signal that correctly detects tight financial conditions 80 percent of times 4 to 12 quarters ahead. - Multivariate signals outperform single-variable signals: - Combining FSIs with macro variables yields superior early warning accuracy. - Policy implications: - Authorities with financial stability mandates can deploy FSIs as a signal extraction tool to detect elevated likelihood of tight financial conditions and to inform surveillance. - Capital adequacy and bank profitability indicators are particularly informative for early detection, and should be emphasized in macro-financial surveillance frameworks. *Source: wpiea2021197-print-pdf - REFERENCES ____________________________________________________________29* ### 2019. While some of these indicators have been studied in the literature (see section II) ### wpiea2021197-print-pdf - 2019. While some of these indicators have been studied in the literature (see section II) ### Data and variable transformations - Two sets of FSIs: core (covers deposit taker institutions) and additional (covers deposit taker institutions, other financial institutions, non-financial private sector, and real estate market). - Transformations applied to all variables: - One-quarter and one-year change of level (differences) and/or growth rate to capture momentum. - One-sided Hodrick-Prescott detrending with smoothing parameter 휆휆=1600 for quarterly data (Ravn and Uhlig, 2002); cyclical deviation computed for all variables except those related to growth rates, interest rates or spreads. - Country-specific z-score standardization so thresholds are country-specific. - Additional predictors: rate of inflation, real GDP, industrial production, debt service cost, current account, financial account, financial market prices, and additional data from International Financial Statistics (IFS). - The complete list of predictors is in appendix A; evolution of selected FSIs is in appendix B. ### Empirical methodology — overview - Goal: produce early warning signals about tightening of financial conditions 4 to 12 quarters ahead. - Methods yield probabilities that financial conditions worsen in horizon of 4 to 12 quarters (except signal extraction which issues binary signals). - Cross-validation: k-fold cross-validation with 10 folds, repeated 5 times; reported results are averages across repetitions. - Feature selection: variables with AUC confidence interval not overlapping 0.5 and maximization of overlap across variables yielded 19 variables; Principal Component Analysis used to reduce these 19 variables to 6 principal components explaining 95% of variance. - Sample characteristics: unbalanced panel covering periods from 2000 to 2016 subject to data availability; countries included: ARG, AUS, AUT, BEL, BGR, BRA, CAN, CHL, CHN, COL, CZE, DEU, DNK, ESP, FIN, FRA, GBR, GRC, HUN, IDN, IND, IRL, ISR, ITA, JPN, KOR, MEX, MYS, NLD, NOR, PER, PHL, POL, PRT, RUS, SWE, THA, TUR, USA, VNM, ZAF. ### Evaluation window and sample treatment - Evaluation horizon: 4 to 12 quarters before a deterioration in the financial system. - Rationale: lower bound 4 quarters because some prudential measures need at least a year to operationalize (example: Countercyclical Capital Buffer). - Stability: increasing horizon from 12 to 16 quarters does not significantly impact results but reduces accuracy. - Crisis/postcrisis bias: crisis and two years after crisis end are discarded (Lo Duca et al., 2017 crises database provides “system back to normal” dates). ### Accuracy measures and class imbalance adjustments - Accuracy metrics: - True Positive Rate (TPR) and False Positive Rate (FPR) tradeoff summarized by Receiver Operating Characteristic (ROC) curve and Area Under the ROC Curve (AUC). - Loss-function and usefulness framework referenced (Alessi and Detken, 2011; Bussiere and Fratzscher, 2008) but requires policymaker preferences. - Noise-to-signal ratio approach often sets fixed prediction share at 60-75 percent (Borio and Lowe, 2002; Borio and Drehmann, 2009). - AUC interpretation caveat: with binary dependent variable equal to 1 in 10% of cases, high AUC can be misleading under class imbalance. - Imbalance correction: Synthetic Minority Over-Sampling Technique (SMOTE, Chawla et al., 2002) applied to generate additional “tightness” observations until tranquil and tightness counts are equal. - Threshold choice: - Let τ ∈ (0,1) be cost of missing a true positive; (1−τ) cost of issuing a false alarm. - Relative cost = τ / (1−τ). Optimal threshold chosen as tangent on ROC curve to inverse relative cost ( (1−τ) / τ ). ### Methods used to generate signals (eight methods) - Signal extraction (SigExt): binary signal s=1 when predictor exceeds threshold; optimal threshold found by grid search; used for univariate models. - Decision tree (Tree): recursive partitioning; node size tuned by grid search over exponentially spaced values 1 through 100; optimal tree chosen within one standard error of minimum cross-validated out-of-sample error. - Discriminant analysis (LDA and QDA): QDA assumes distinct covariance matrices; LDA assumes same covariance matrix; regularization via grid search over 훾훾 and 훿훿 for multivariate DA. - Support Vector Machine (SVM): Gaussian kernel with tuning parameter 훾훾 picked from grid between 0.1 and 1 to minimize cross-validated out-of-sample prediction error. - k-Nearest Neighbors (KNN): output is probability computed as average of dependent binary variables of k neighbors; k set to rounded square root of sample size. - Naïve Bayes (NB): conditional probability density functions multiplied by prior probabilities (fraction of tranquil and tightness observations); prediction by maximum a posteriori rule. - Logistic Regression (LR): pooled data without fixed effects; data standardized with country-specific z-score to avoid driving probability to zero for countries that never experienced tightness. ### Variable selection and dimensionality reduction - Initially selected 19 variables (list included in source). - High correlations noted: Regulatory Capital to Risk Weighted Assets vs Regulatory Tier 1 Capital to Risk Weighted Assets correlation coefficient 0.88; Return on Assets vs Return on Equity correlation coefficient 0.85. - PCA used to reduce to 6 principal components explaining 95% of variance for multivariate models. - Signal extraction not used in multivariate case due to exponential growth of grid search. ### Key empirical findings — univariate signals - FSIs produce high-accuracy signals relative to macro-financial variables. - Most accurate univariate signals are FSIs related to capital adequacy, followed by real estate markets and earnings & profitability variables. - Top-performing predictor: Tier 1 Capital to Assets ratio (example: Chile used to illustrate). - Table 1 (cross-validated out-of-sample AUC for 4-12 quarters horizon) — selected entries preserved exactly: - Tier 1 Capital to Assets raw: SigExt 0.227, Tree 0.836, LDA 0.772, QDA 0.771, SVM 0.768, KNN 0.830, NB 0.790, LR 0.772 (FSI: Yes) - Regulatory Tier 1 Capital to Risk Weighted Assets raw: SigExt 0.222, Tree 0.852, LDA 0.778, QDA 0.813, SVM 0.778, KNN 0.866, NB 0.847, LR 0.778 (FSI: Yes) - Regulatory Capital to Risk Weighted Assets raw: SigExt 0.212, Tree 0.854, LDA 0.788, QDA 0.787, SVM 0.787, KNN 0.854, NB 0.803, LR 0.788 (FSI: Yes) - Residential Real Estate Loans to Total Loans raw: SigExt 0.637, Tree 0.856, LDA 0.635, QDA 0.635, SVM 0.638, KNN 0.892, NB 0.733, LR 0.635 (FSI: Yes) - Net Open Position in Foreign Exchange to Capital c4: SigExt 0.590, Tree 0.783, LDA 0.589, QDA 0.751, SVM 0.775, KNN 0.806, NB 0.776, LR 0.588 (FSI: Yes) - Liquid Assets to Short Term Liabilities hp: SigExt 0.355, Tree 0.767, LDA 0.645, QDA 0.573, SVM 0.647, KNN 0.781, NB 0.702, LR 0.642 (FSI: Yes) - Tier 1 Capital to Assets z: SigExt 0.341, Tree 0.697, LDA 0.658, QDA 0.657, SVM 0.659, KNN 0.729, NB 0.655, LR 0.658 (FSI: Yes) - Note: AUC for signal extraction below 0 indicates variable generates a signal when it achieves low level. - Dependent variable equals 1 in 10% of cases (class imbalance). ### Findings on thresholds and trade-offs - Example: Regulatory Tier 1 Capital ratio: - Thresholds of 8 and 10 percent assessed as non-informative. - Threshold of 12 percent correctly classifies impending tight financial conditions with TPR 84 percent and FPR 30 percent. - Confusion matrix insights (example summaries): - For lowest τ, Decision Tree correctly classifies 30.9 percent of true negatives and 39.0 percent of true positives; 30.1 percent misclassified. - Increasing τ (more tightness-averse, lower threshold) raises false alarms: for τ=0.6, 23.5 percent observations are false positives; for τ=0.9, 38.1 percent observations are false positives. - Lower thresholds imply more false alarms and more true positives; policymaker trade-off emphasized. ### Cross-sample and crisis-type findings - Results robust for high-income economies (HIE) subsample; AUC values on HIE subsample are on average higher than full sample. - Advanced economies compose 80 percent of full sample when dependent variable is FCI. - FSIs also generate early signals for banking crises (Laeven and Valencia, 2018 dataset) and financial crises (Lo Duca et al., 2017 dataset): - Best FSI categories for banking crises: real estate markets, earnings and profitability, capital adequacy, sensitivity to market risk. - For upper middle-income countries: best predictors include real estate markets, interest rates on government bonds, financial institutions claims on private non-financial sector, and real economy variables. - For lower middle-income countries: best predictors include capital adequacy, foreign currency denominated debt, deposits-to-loan relation, and earnings & profitability. - For low-income economies: best predictors include interest rates on government bonds, foreign liabilities of banking institutions, and current account. - Housing price-to-rent and price-to-income ratios also provide useful signals. ### Policy-relevant implications and recommendations - Authorities in charge of financial sector surveillance should monitor capital adequacy FSIs as they generate reliable early warning signals of tightening financial conditions. - Combining multiple variables in multivariate models can significantly improve accuracy relative to univariate signals (next section discusses multivariate models). - For policy threshold choice, explicit consideration of relative costs (τ / (1−τ)) is recommended to select optimal operating point on ROC curve consistent with risk aversion to missing tightness vs issuing false alarms. - Clustering countries (e.g., by per capita income) can increase predictive accuracy; estimating models on homogeneous subsets is advisable where data permit. ### Transition to multivariate models - Single-variable signals exhibit significant false positive and false negative rates even for best predictors (e.g., Tier 1 Capital to Assets). - Multivariate models combining several predictors yield superior accuracy; feature selection reduced dimensionality to 6 principal components for multivariate classification methods. - Signal extraction is not used for multivariate combinations due to computational infeasibility. *Italic: Content derived from wpiea2021197-print-pdf (2019) as supplied.* ### Section IV.C explains which variables are chosen. To reiterate, only variables with AUC of ### wpiea2021197-print-pdf - Section IV.C explains which variables are chosen. To reiterate, only variables with AUC of ### Variable selection and dimensionality reduction - Only variables with AUC of signals higher than 0.5 are chosen. - Another criterion for variable selection is minimization of missing observations: if an observation for a variable is missing it means that all other variables for this time period and country are ignored. - Nineteen variables are chosen. - Because many variables are highly correlated, Principal Component Analysis (PCA) is used to reduce data size and speed up estimation. - The first six principal components explain 95% of variance in the data and these six components are used in the estimation. - Seven different methods are used due to model uncertainty. - Predictive accuracy is evaluated against benchmark—FCIs binary variable showing when financial conditions are excessively tight. - Estimation and cross-validation processes are the same as in the previous section. - The final output of each method is a cross-validated probability of excessively tight financial conditions occurring in 4 to 12 quarters. ### Predictive performance and ROC analysis - KNN and Tree dominate other methods in terms of signals accuracy based on ROC curves showing feasible combinations of FPR and TPR. - At some point lowering the thresholds does not yield a benefit in terms of capturing more tightness periods. - KNN and Tree can correctly classify most true positives with a relatively low false positive rate of 20 percent. - Figure 11 relates ROC curves to confusion matrices for specific τ. KNN and Tree methods feature high accuracy for any thresholds. - The ratio of false positives and false negatives increases modestly as τ increases. - AUC values confirm the ROC analysis. - KNN is the best performing method, consistent with Holopainen and Sarlin (2017). - AUC values for both indicators are not significantly different; however, analysis of ROC curves shows performance differs in different regions of FPR. - KNN clearly dominates Tree for low levels of FPR. - Logistic regression and linear discriminant analysis generate the least accurate signals; this aligns with their performance for a different crisis dataset in Holopainen and Sarlin (2017). - Many early warning studies use logistic regression as a main tool, but this evidence suggests logistic regression is inferior to classification methods for detecting tightening financial conditions. ### Table 2 — Prediction performance of FSIs measured by AUC - FCIs, full sample - τ 0.30 0.4 0.5 0.6 - Relative cost 0.43 0.67 1 1.5 - Method rows report: AUC, then for each τ the triple (Tr, FPR, TPR) in order. - Tree: 0.96 0.75 0.10 0.94 0.29 0.12 0.97 0.12 0.14 0.98 0.12 0.14 0.98 - LDA: 0.83 0.67 0.12 0.68 0.65 0.12 0.69 0.54 0.19 0.77 0.54 0.19 0.77 - QDA: 0.87 0.93 0.04 0.58 0.92 0.08 0.66 0.92 0.08 0.66 0.60 0.52 0.98 - SVM: 0.85 0.63 0.12 0.72 0.58 0.13 0.74 0.54 0.15 0.76 0.54 0.15 0.76 - KNN: 0.97 0.67 0.07 0.91 0.67 0.08 0.92 0.54 0.11 0.95 0.54 0.11 0.95 - NB: 0.92 0.74 0.06 0.75 0.72 0.06 0.77 0.61 0.11 0.81 0.60 0.11 0.82 - LR: 0.83 0.63 0.12 0.67 0.59 0.14 0.70 0.59 0.14 0.70 0.31 0.55 1.00 - Note (as reported): The table reports cross-validated out-of-sample performance measured by AUC for seven methods and evaluation horizon of 4-12 quarters. The second column shows Area Under the ROC curve. Next three columns report optimal threshold (Tr) in terms of probability form the model and False Positive Rate (FPR) and True Positive Rate (TPR) associated with the optimal threshold. The optimal threshold is chosen according to the cost of missing a crisis ( τ). Unbalanced panel data covers periods from 2000 to 2016 subject to data availability. Countries included: ARG, AUS, AUT, BEL, BGR, BRA, CAN, CHL, CHN, COL, CZE, DEU, DNK, ESP, FIN, FRA, GBR, GRC, HUN, IDN, IND, IRL, ISR, ITA, JPN, KOR, MEX, MYS, NLD, NOR, PER, PHL, POL, PRT, RUS, SWE, THA, TUR, USA, VNM, ZAF. ### Key conclusions and policy implications - Financial Soundness Indicators (FSIs) can be successfully used for macro-financial surveillance. - Signals based on FSIs offer accurate and early detection of financial sector distress as proxied by tightening financial conditions. - Indicators that should be closely monitored are related to capital adequacy and earnings and profitability of deposit taker institutions. - High leverage and returns on assets and equity issue early warnings that vulnerabilities are building up in the financial sector. - Generating signals based on single variables is inferior to multivariate model-based signals; multivariate signals offer superior accuracy of detecting tight financial conditions 1 to 3 years ahead. - Accuracy depends on statistical methods used; logistic regression appears to be a suboptimal choice for detecting tightening financial conditions. - Using FSIs in macro-financial surveillance can signal tightness of financial conditions early enough to implement policy measures, potentially mitigating lower and more volatile economic growth. *Source: wpiea2021197-print-pdf - Section IV.C (excerpt).* ### REFERENCES ### wpiea2021197-print-pdf - REFERENCES ### References (selected themes and authors) - Literature on macroprudential policy, financial vulnerabilities, and monetary policy interactions: Adrian (2017); Adrian and Duarte (2018); Adrian & Liang (2018); Adrian, Boyarchenko, & Giannone (2019). - Early warning indicators, banking crises databases, and crisis dating: Alessi & Detken (2011); Antunes et al. (2014); Babecký et al. (2014); Caprio et al. (2005); Chaudron & de Haan (2014); Laeven & Valencia (2013, 2018). - Financial soundness indicators and IMF work: Cihák & Schaeck (2007); Costa Navajas & Thegeya (2013); International Monetary Fund (2019). - Credit cycles, financial accelerator, and long-run lessons: Bernanke, Gertler, & Gilchrist (1996); Drehmann & Juselius (2012, 2014); Jordà, Schularick, & Taylor (2011); Furceri & Mourougane (2012). - Methodological and machine learning contributions to forecasting and early-warning models: Hastie et al. (2005); Holopainen & Sarlin (2017); Chawla et al. (2002) (SMOTE); Van den Berg et al. (2008). ### Appendix A — Data sources (variables from IMF and BIS) - International Monetary Fund variables (as listed): - Regulatory Capital to Risk Weighted Assets. - Regulatory Tier 1 Capital to Risk Weighted Assets. - Interest Margin to Gross Income. - Non-interest Expenses to Gross Income. - Return on Assets, Return on Equity. - Liquid Assets to Short Term Liabilities. - Liquid Assets to Total Assets Liquid Asset Ratio. - Net Open Position in Foreign Exchange to Capital. - Tier 1 Capital to Assets. - Customer Deposits to Total Non-interbank Loans. - Foreign Currency Denominated Liabilities to Total Liabilities. - Gross Asset Position in Financial Derivatives to Capital. - Gross Liability Position in Financial Derivatives to Capital. - Large Exposures to Capital. - Net Open Position in Equities to Capital. - Personnel Expenses to Non-interest Expenses. - Spread Between Highest and Lowest Interbank Rate. - Spread Between Reference Lending and Deposit Rates. - Trading Income to Total Income. - Household Debt to Gross Domestic Product GDP. - Assets to Gross Domestic Product GDP. - Assets to Total Financial System Assets. - Commercial Real Estate Loans to Total Loans. - Residential Real Estate Loans to Total Loans. - Residential Real Estate Prices. - Current Account. - Net Acquisition of Financial Assets. - Index of Equities. - Consumer Price Index. - Portfolio Investment. - Government Bonds Interest Rates. - Treasury Bills Interest Rates. - Real Gross Domestic Product. - Real Industrial Production. - Bank for International Settlements variables (as listed): - Price To Rent. - Price To Income. - Housing Rent. - Real House Prices. - Debt Service Ratio. ### Appendix B — Evolution of Financial Soundness Indicators - Figures depict evolution of variables that gave accurate signals about future financial distress. - For each variable (for two income groups according to World Bank classification) four panels are shown: - Top-left: median evolution over time. - Top-right: number of reporting countries in a given time. - Bottom panels: box plot with 25th, 50th and 75th percentiles; whiskers extend to most extreme non-outlier observations; red dots show outliers. - Figures referenced include: - Figure 12 Regulatory capital to risk weighted assets. - Figure 13 Regulatory Tier 1 Capital to risk weighted assets. - Figure 14 Tier 1 Capital to assets. - Figure 15 Personnel Expenses to Non-interest Expenses. - Figure 16 Liquid Assets to Short Term Liabilities. - Figure 17 Residential Real Estate Loans to Total Loans. - Figure 18 Return on Equity. - Figure 19 Interest Margin to Gross Income. ### Appendix C–E — Single-variable predictive accuracy (AUC reporting) - Note across tables: "The table reports cross-validated out-of-sample performance measured by AUC for eight methods and evaluation horizon of 4-12 quarters. The last column shows whether a variable is a FSIs. Note that AUC levels for signal extraction below 0 means that a variable generates a signal when it achieves low level. Unbalanced panel data covers periods from 2000 to 2016 subject to data availability." - Table 3 (FCIs, HIE sample) — example rows and exact AUC values: - Regulatory Capital to Risk Weighted Assets raw: SigExt 0.166, Tree 0.869, LDA 0.833, QDA 0.829, SVM 0.833, KNN 0.882, NB 0.878, LR 0.834. FSI: Yes. - Regulatory Tier 1 Capital to Risk Weighted Assets raw: SigExt 0.211, Tree 0.874, LDA 0.788, QDA 0.847, SVM 0.840, KNN 0.883, NB 0.860, LR 0.788. FSI: Yes. - Net Open Position in Foreign Exchange to Capital raw: SigExt 0.739, Tree 0.855, LDA 0.733, QDA 0.788, SVM 0.823, KNN 0.892, NB 0.867, LR 0.728. FSI: Yes. - Table 4 (LV crises dataset, full sample) — example rows and exact AUC values: - Residential Real Estate Loans to Total Loans hp: SigExt 0.174, Tree 0.524, LDA 0.802, QDA 0.801, SVM 0.721, KNN 0.672, NB 0.758, LR 0.814. FSI: Yes. - Return on Assets z: SigExt 0.745, Tree 0.525, LDA 0.742, QDA 0.733, SVM 0.696, KNN 0.689, NB 0.746, LR 0.742. FSI: Yes. - Table 5 (LV crises dataset, HIE sample) — example rows and exact AUC values: - Regulatory Tier 1 Capital to Risk Weighted Assets raw: SigExt 0.069, Tree 0.853, LDA 0.931, QDA 0.972, SVM 0.906, KNN 0.971, NB 0.962, LR 0.931. FSI: Yes. - Personnel Expenses to Non-interest Expenses z: SigExt 0.889, Tree 0.568, LDA 0.889, QDA 0.852, SVM 0.889, KNN 0.842, NB 0.892, LR 0.887. FSI: Yes. - Additional tables (Tables 6–10) present AUC results by income group (upper middle-income, lower middle-income, low-income) and for LD crises dataset, preserving the same reporting format and notes. Example: - Table 9 (LD crises dataset, full sample) first row: Regulatory Capital to Risk Weighted Assets raw: SigExt 0.031, Tree 0.927, LDA 0.968, QDA 0.972, SVM 0.969, KNN 0.970, NB 0.971, LR 0.968. FSI: Yes. - Table 10 (LD crises dataset, HIE sample) first row: Regulatory Capital to Risk Weighted Assets raw: SigExt 0.035, Tree 0.953, LDA 0.965, QDA 0.969, SVM 0.965, KNN 0.967, NB 0.965, LR 0.965. FSI: Yes. ### Appendix F — Multivariate early warning models and performance summaries - Figures and diagnostics: - Figure 20 ROC curve—LV crises dataset. - Figure 21 ROC curve—LD crises dataset. - Figure 22 Confusion matrices for LV crises, full sample. - Figure 23 Confusion matrices for LD crises, full sample. - Table 11 (Prediction performance of FSIs measured by AUC - LV, full sample) — structure and selected numeric entries: - Reporting format: τ columns (0.30, 0.4, 0.5, 0.6) with corresponding Relative cost 0.43, 0.67, 1, 1.5. - Methods compared include Tree, LDA, QDA, SVM, KNN, NB, LR with columns: AUC, Tr, FPR, TPR for each τ. - Example row (Tree): AUC 0.99; for τ 0.30 Tr 0.56 FPR 0.03 TPR 0.98; for τ 0.4 Tr 0.56 FPR 0.03 TPR 0.98; for τ 0.5 Tr 0.56 FPR 0.03 TPR 0.98; for τ 0.6 Tr 0.56 FPR 0.03 TPR 0.98. - Example row (KNN): AUC 1.00; for τ 0.30 Tr 0.93 FPR 0.01 TPR 0.99; for τ 0.4 Tr 0.92 FPR 0.01 TPR 0.99; for τ 0.5 Tr 0.92 FPR 0.01 TPR 0.99; for τ 0.6 Tr 0.87 FPR 0.02 TPR 1.00. - Table 12 (Prediction performance of FSIs measured by AUC - LD, full sample) — structure and selected numeric entries: - Same τ and Relative cost columns as Table 11. - Example row (Tree): AUC 0.99; for τ 0.30 Tr 0.67 FPR 0.02 TPR 0.96; for τ 0.4 Tr 0.67 FPR 0.02 TPR 0.96; for τ 0.5 Tr 0.37 FPR 0.03 TPR 0.98; for τ 0.6 Tr 0.37 FPR 0.03 TPR 0.98. - Example row (KNN): AUC 1.00; for τ 0.30 Tr 0.80 FPR 0.01 TPR 0.99; for τ 0.4 Tr 0.70 FPR 0.02 TPR 1.00; for τ 0.5 Tr 0.70 FPR 0.02 TPR 1.00; for τ 0.6 Tr 0.70 FPR 0.02 TPR 1.00. - Note: Tables 11 and 12 report cross-validated out-of-sample performance measured by AUC for seven methods and evaluation horizon of 4-12 quarters. The second column shows Area Under the ROC curve. Next three columns report optimal threshold (Tr) in terms of probability from the model and False Positive Rate (FPR) and True Positive Rate (TPR) associated with the optimal threshold. The optimal threshold is chosen according to the cost of missing a crisis (τ). Unbalanced panel data covers periods from 2000 to 2016 subject to data availability. *Content derived from the REFERENCES and APPENDICES of the PDF: wpiea2021197-print-pdf - REFERENCES* --- _Source: https://www.imf.org/-/media/files/publications/wp/2021/english/wpiea2021197-print-pdf.pdf_