## _wp1448

## Source details

**Canonical URL:** [_wp1448](https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2014/_wp1448.pdf)

## Other formats

- [Markdown version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2014/_wp1448.pdf.md)
- [Structured JSON version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2014/_wp1448.pdf.json)

---

### Introduction and role of the LIC DSF
- Purpose and context:
  - LICs recovering from the sovereign debt crisis of the 1980s and 1990s and from HIPC and MDRI debt relief seek to expand public investment while balancing renewed-debt-crisis risk.
  - The World Bank–IMF Debt Sustainability Framework (DSF), introduced in 2005, is the standardized framework for conducting debt sustainability analysis (DSA) in LICs.
- Structure of a DSA:
  - Analysis of the country’s projected debt burden over the next 20 years.
  - Assessment of vulnerability to shocks and the risk of debt distress.
- Uses of the LIC DSF:
  - Fund’s analysis of macroeconomic stability and IMF financing access.
  - Determining the share of grants and loans in World Bank assistance to LICs.
  - Informing the World Bank Non-Concessional Borrowing Policy and the IMF Debt Limits Policy.

### Mechanics: Debt Threshold Approach (DTA), probit models, and the WCA
- DTA mechanics:
  - Five separate policy-dependent debt thresholds derived from separate probit models, each with a different debt-burden measure (DGDP, DExp, DRev, DsExp, DsRev) and a common set of controls (CPIA, Growth), interacting with a MIC dummy.
  - A loss function in missed crises and false alarms is used to deduce an optimal probability cutoff P*_j for each debt-burden probit; inversion of the probit yields a debt-ratio threshold D_DTA_j.
  - Countries are assigned risk ratings by comparing projected debt-burden measures for 20 years to these thresholds.
- Worst-Case Aggregator (WCA):
  - DSF classifies a country at high risk if any one debt measure breaches its threshold (any Debt_j ≥ D_DTA_j).
  - The WCA avoids constructing an aggregator from data but applies variable-by-variable cutoffs mechanically.
- Role of judgment:
  - Scope for judgment exists, especially for small/temporary or near breaches; mechanical application usually predicts the actual rating, with occasional overruling for minor or near breaches.

### Goodness-of-fit (GOF) diagnostics and ROC interpretation
- ROC interpretation:
  - y-axis: true positives = A/(A+MC). x-axis: false positives = FA/(B+FA).
  - AUROC: uninformative model = 0.5; perfect predictor = 1.
  - The DTA-selected point (α = 1/2) is indicated on ROC curves; the DTA-cutoff discrepancy arises because cutoffs are chosen using country-specific CPIA and Growth but thresholds are applied using grouped CPIA and LIC-average Growth.
- AUROC reported for single debt-variable probits:
  - DGDP: 0.678 (figure label)
  - DExp: 0.749
  - DRev: 0.653
  - DsExp: 0.792
  - DsRev: 0.762

### Key limitations of the DSF (recap)
- Use of grouped CPIA and LIC-average Growth for thresholds loses information from continuous covariates and prevents adding covariates without proliferating thresholds.
- Aggregation ambiguity:
  - Multivariate probit shows none of the debt variables significant in that specification; correlations among debt measures are high enough to be problematic.
  - Five-variable probit has substantially higher log-likelihood than single-variable probits, implying potential gains from better aggregators.

---

### Bias and accuracy of the WCA
- Empirical bias:
  - The WCA is biased and “too conservative” — it yields too many false alarms and too few missed crises given the loss function used.
  - Cause: WCA applies single-variable cutoffs to the aggregated rule, causing excessive crisis calls.
- Quantification (sample frequencies and optimal cutoffs):
  - Individual debt measures call crises between 13 percent (DRev) and 40 percent (DsRev) of the time.
  - The WCA calls crises 56 percent of the time (Table 5).
  - Variable-by-variable cutoffs minimizing single-variable loss: around 10–14 percent (Table 6, column 2).
  - WCA-optimal variable cutoffs: about 18–20 percent (Table 6, column 3); WCA optimal cutoff reported as 19.0% (Table 6 / References).
  - Example threshold implication (medium policy CPIA = 3.5): debt/GDP threshold moves from 30 to 49 percent when moving from biased to WCA-optimal cutoffs (Table 6, column 4 example).
  - Weighting interpretation: the DTA-chosen cutoffs nearly coincide with an α = 0.67 weight (missed crises twice as important as false alarms); under α = 1/2 the WCA is biased toward false alarms.
- Accuracy comparisons:
  - WCA AUROC at DTA point is slightly lower than best individual measure DsExp (DsExp AUROC = 0.792; WCA AUROC = 0.78 reported in References).
  - WCA chosen: Loss = 0.29; AUROC = 0.78. WCA optimal: Loss = 0.28 (References).
- Remedies for bias:
  - Raise debt thresholds to higher values (Table 6) so WCA minimizes intended loss.
  - Alternatively, adopt a higher weight on missed crises in the loss function (e.g., α = 2/3) to justify lower thresholds.
  - Recommend transparency about implicit beliefs and explicit loss-function weights.

---

### The Probability Threshold Approach (PTA) and aggregation alternatives
- PTA concept and operation:
  - Work directly with estimated probabilities P_j from the probit using country-specific CPIA and Growth; call a crisis if P_j ≥ P*j, where P*j is the optimal probability cutoff found by minimizing loss (2).
  - PTA avoids inverting to debt-ratio thresholds, enables country-specific covariates, and facilitates aggregation.
- PTA advantages and GOF gains (example DsExp):
  - For DsExp: AUROC increases from 0.792 (DTA chosen) to 0.814 (PTA optimal); Loss falls from 0.31 (DTA chosen) to 0.25 (PTA optimal) (References).
  - GOF gain decomposition for DsExp:
    - Move from DTA chosen Loss = 0.31 to DTA optimal Loss = 0.26 eliminates DTA cut-off discrepancy.
    - Further move to PTA optimal Loss = 0.25 captures gain from country-specific CPIA and Growth.
- Aggregation framework:
  - Define composite indicators CI_k = sum_{j=1 to 5} w_j × Debt_j and estimate probits with CI_k replacing individual debt measures (interacting with MIC and including Growth and CPIA).
- Equal-weights composite (CI_EW):
  - CI_EW = sum_{j=1 to 5} 0.2 × Debt_j.
  - CI_EW is much more accurate than WCA: AUROC = 0.84; Loss = 0.23; BIC = 333.02 (Table 7).
  - Bootstrap evidence: in 1,000 bootstrap samples only 3 of 1,000 show AUROC(WCA) > AUROC(CI_EW).
- Multivariate-probit composite (CI_MP):
  - CI_MP = sum_{j=1 to 5} β_j × Debt_j with β_j from multivariate probit.
  - CI_MP AUROC = 0.87; Loss = 0.21; but BIC = 372.36 (penalized, statistically less attractive) (Table 7).
  - Weights: DsRev largest weight (0.524); DExp small (0.049); DRev negative (−0.017). Weights imprecisely estimated.
- Parsimonious data-based models (CI_SW, DsRev, DsRev+DExp):
  - Step-wise BIC search over 31 models finds best single-variable model is DsRev; best two-variable model DsRev + DExp.
  - CI_SW (best two-variable model): AUROC = 0.86; Loss = 0.22; BIC = 337.74 (Table 7).
  - Single-variable DsRev model has lowest BIC among considered probit specifications.
- Equal-weight-prior composite (CI_EWP):
  - CI_EWP assigns 0.2 to four variables and 0.55 to DsRev (data reject many deviations from equal weights); AUROC = 0.86; Loss = 0.21; BIC = 338.44 (Table 7).
- Composite indicators summary (Table 7):
  - CI_EW: Loss = 0.23; AUROC = 0.84; BIC = 333.02.
  - CI_MP: Loss = 0.21; AUROC = 0.87; BIC = 372.36.
  - CI_SW: Loss = 0.22; AUROC = 0.86; BIC = 337.74.
  - CI_EWP: Loss = 0.21; AUROC = 0.86; BIC = 338.44.
  - WCA: Loss = 0.29; AUROC = 0.78; BIC = 372.36.

### Practical recommendations on aggregators
- CI_EW is a strong, robust baseline:
  - Simple equal-weight averaging outperforms WCA and avoids overfitting problems associated with CI_MP.
  - CI_EW AUROC = 0.84; CI_MP AUROC = 0.87 but CI_MP BIC is much larger.
- Parsimonious models centered on DsRev (and possibly DExp) perform well with tighter parameter estimates and lower BIC.
- CI_EWP is an intermediate option if one wishes to maintain a prior that all five variables matter (gives 0.55 weight to DsRev, 0.2 to others) but does not produce statistically significant BIC gains over CI_EW.
- Use penalized goodness-of-fit criteria (BIC) to trade off in-sample accuracy against overfitting risk.
- Avoid WCA unless thresholds are adjusted or the loss-function weight on missed crises is explicitly justified.

---

### Model-selection summary, implicit weighting, and policy implications
- Model-comparison conclusions:
  - WCA is biased and statistically inaccurate relative to simple averaging (CI_EW).
  - CI_MP improves AUROC but at a statistically-significant BIC penalty; CI_EW and parsimonious CI_SW models are preferred for parsimony and robustness.
  - No single aggregator is unambiguously best among CI_EW, CI_SW, CI_EWP; choice depends on prior beliefs about variables and tolerance for sample-dependence.
- Implicit weighting in DSF loss implementation:
  - Sample has 12.1 percent crisis observations; 186 tranquil periods and 17 crisis periods.
  - Ratio (B+FA)/(A+MC) = 10.9 implies DSF “equal” weighting scheme actually weights missed crises 10.9 times more than false alarms; conservative WCA doubles that effect, yielding roughly 22 times weight per missed crisis vs per false alarm.
  - Under truly equal weights on observations, optimal cutoffs would be extremely high (between 0.61 and 0.79) and trivial solutions (cutoff = 1) may minimize loss.
- Recommended remedies and policy options:
  - Fix WCA bias by raising individual debt thresholds so that WCA minimizes intended loss.
  - Alternatively, explicitly adopt a higher weight on missed crises if the conservative bias is desirable and justified.
  - Improve accuracy by replacing or complementing WCA with multivariate indicators such as CI_EW or parsimonious models (DsRev, DsRev+DExp).
  - Consider broader reforms (fiscal reaction functions, country-specific frameworks focusing on public investment and LIC features).
- Guidance on judgment and model use:
  - Statistical models are valuable because crises are infrequent and subjective judgment has limited feedback.
  - Good multivariate predictors should be combined with contextual judgment rather than replaced by heuristics.

*Source: _wp1448*

### 1.   Introduction ......................................................................................................

### 1    Introduction

### Context and purpose
- Low-income countries (LICs) are recovering from the prolonged sovereign debt crisis of the 1980s and 1990s and from debt relief under the Highly-Indebted Poor Country (HIPC) initiative and the Multilateral Debt Relief Initiative (MDRI).
- Debt levels are now low in most of these formerly highly-indebted countries; many countries have improved their policy environments and economic performance.
- LICs and development partners seek to expand public investment to build scarce infrastructure and improve health and education while balancing the risk of renewed debt crises.
- The World Bank and the International Monetary Fund (IMF)’s Debt Sustainability Framework (DSF), introduced in 2005, is the standardized framework for conducting debt sustainability analysis (DSA) in LICs.

### Structure and role of the DSF
- A DSA consists of:
  - an analysis of the country’s projected debt burden over the next 20 years, and
  - its vulnerability to shocks and an assessment of the risk of debt distress.
- The LIC DSF is a central tool for:
  - the Fund’s analysis of macroeconomic stability and in determining access to IMF financing,
  - determining the share of grants and loans in World Bank assistance to LICs (countries at high risk of debt distress receive all their World Bank support as grants),
  - informing the World Bank Non-Concessional Borrowing Policy and the IMF Debt Limits Policy.

### Mechanics: probit models, thresholds, and the Debt Threshold Approach (DTA)
- The DSF is a variant of standard methods that use probit regressions to predict crises.
- The DSF is based on five separate policy-dependent debt thresholds.
- Each threshold is derived from estimation of a simple probit model where:
  - the dependent variable is the probability that a country will experience debt distress,
  - each of the five models includes a different measure of debt burden (public external debt as a share of GDP, debt-service as a share of exports, etc.) along with a common set of controls.
- The DSF defines a loss function in terms of missed crises and false alarms and deduces an optimal risk-of-debt-distress cutoff for each debt-burden measure.
- These probability cutoffs imply a threshold value for each debt-burden measure such that, if the measure breaches the threshold value, then the probability of debt distress is above the tolerable level.
- Under the DSF, countries are assigned a risk rating by comparing the debt-burden measures projected out for 20 years to the corresponding thresholds.
- This method is referred to in the text as the “debt threshold approach” (DTA).

### Aggregation of multiple indicators and the DSF worst-case aggregator (WCA)
- Multiple debt-burden measures may imply different debt risk ratings for a given country.
- In the DSF a country is classified at high risk of debt distress if any one of the debt measures breaches its threshold, irrespective of the evolution of the other four.
- This feature is called the DSF worst-case aggregator (WCA).
- The IMF and the World Bank (2004) justify this rule partly on grounds that collinearity among debt-burden measures makes it difficult to combine them into one probit regression.
- The WCA avoids the multicollinearity problem only insofar as it does not seek to establish the parameters of the aggregator (for example, the relative importance of each indicator) from the data.

### Role of judgment versus mechanical application
- There is scope for judgment in applying the DSF, particularly regarding small and temporary breaches and near breaches of thresholds; the most recent IMF-World Bank guidance note emphasizes this use of judgment.
- In practice, the mechanical application usually predicts the actual rating; exceptions commonly reflect cases in which minor or near-breaches by a single variable are overruled rather than wholesale rejection of thresholds or the WCA.
- The analysis that follows in the source assesses the DSF in its mechanical application while acknowledging the role of judgment.

*Source: _wp1448 - 1.   Introduction*

### conclusion.

### conclusion.

### Probability Threshold Approach (PTA) versus Debt Threshold Approach (DTA)
- Section 5 investigates an alternative approach—the “probability threshold approach” (PTA)—which assesses debt risks by comparing the probability of debt distress with probability thresholds rather than comparing debt measures to fixed thresholds as in the DTA.
- Advantages of the PTA:
  - It is more accurate in part because it makes more effective use of the information contained in other predictors of debt distress such as policy and growth.
  - It facilitates aggregation of the information contained in the various debt indicators.
- The PTA is the more standard approach used in most probit early-warning models; the paper uses the label PTA to clarify the distinction with the DTA.

### Aggregation of Debt Indicators and the Weakest-Cell Aggregator (WCA)
- Section 6 analyzes alternative aggregators based on weighted averages of the five debt measures in the DSF and compares them with the WCA and single debt indicators.
- Findings on the WCA:
  - The WCA is biased: it produces too many false alarms for each missed crisis when evaluated by the DSF loss function.
  - The WCA is inaccurate relative to the proposed alternatives; for any given weights on false alarms and missed crises in the loss function, the WCA performs significantly less well than the proposed alternatives in identifying high-risk countries.
  - The bias of the WCA is a matter of construction and can exacerbate uncorrelated measurement error across debt measures.
- Conceptual analogy used: comparing multiple medical indicators (e.g., cancer vs. heart disease) — focusing on the worst indicator (WCA) versus averaging indicators can lead to different predictive accuracy; the paper lets data decide which aggregation is superior.

### Recommended Aggregation Strategy and Model Comparison
- The paper establishes the statistical superiority of some alternative estimators (averaging-type aggregators) and attempts to recommend which alternative estimator has the most appealing properties, recognizing this is a difficult question.
- A loss function defined in terms of missed crises and false alarms is used to compare models and to discuss bias and relative accuracy.
- The DSF’s application of the loss function and some definitional subtleties result in the DSF effectively placing more than 20 times the weight on each missed crisis episode than on false alarms.

### Methodological setup of the DSF and the DTA (recap of framework)
- The IMF-WB LIC DSF estimates the probability of a country experiencing debt distress using separate probit models for each of five debt burden measures; from each probit a threshold is derived and used to assign risk ratings—this is called the debt threshold approach (DTA).
- Sample and estimation details:
  - The probit regressions are estimated on a sample of low- and middle-income countries observed between 1970 and 2007.
  - Equation (1) follows the DSF specification where five stand-alone debt measures Debt_j are included separately into five different regressions, interacting with a dummy for middle-income countries (MIC) and including CPIA and Growth as controls.
- The five debt measures (Debt_j) are:
  1) the present value of external debt over GDP (DGDP),
  2) the present value of external debt over exports (DExp),
  3) the present value of external debt over revenues (DRev),
  4) debt service over exports (DsExp),
  5) debt service over revenues (DsRev).
- CPIA treatment and threshold variation:
  - CPIA scores are grouped into three categories (low, medium, high) and assigned CPIA_G values of 3.25, 3.5, and 3.75 respectively.
  - The Growth variable is replaced by the historic average growth rate for all LICs (Growth), producing three values for the debt burden thresholds D_DTA_j for each debt measure.

### Loss function, cutoff selection, and threshold inversion
- The DSF selects a probability cutoff P*_j for each debt-variable probit by minimizing a loss function L that weights missed crises and false alarms:
  - L = α × (MC / (A + MC)) + (1 − α) × (FA / (B + FA))
  - The DSF follows IMF and World Bank (2012) in setting equal weights to Type 1 and Type 2 errors (α = 1/2) for cutoff selection.
- The chosen probability cutoff P*_j is inverted through the probit relationship to obtain the corresponding policy-dependent debt threshold D_DTA_j via equation (3), which depends on values of CPIA_G and Growth.

### Risk rating rules in the DSF
- Based on the estimated policy-dependent thresholds and projected baseline and stress-test scenarios, countries are assigned one of four risk-of-debt-distress ratings:
  - Low risk: All the debt burden indicators are well below the thresholds.
  - Moderate risk: Debt burden indicators are below the thresholds in the baseline scenario, but stress tests indicate that at least one threshold would be breached if there are external shocks or abrupt changes in macroeconomic policies.
  - High risk: One or more debt burden indicators breach the thresholds on a protracted basis under the baseline scenario.
  - In debt distress: The country is already experiencing difficulties in servicing its debt (e.g., accumulation of arrears).

### Limits of the paper and policy implications
- The paper’s purpose is limited to diagnosing specific shortcomings of the DSF and suggesting corrections rather than providing a full reassessment of DSF-induced policy outcomes.
- Correcting the bias in the DSF would tend to result in lower risk ratings for many countries, on average; whether this is desirable depends on other DSF features (for example, excessively optimistic growth forecasts) that may bias ratings in the opposite direction.
- The authors argue that transparent discussion of various flaws is preferable to hoping that different biases cancel out without scrutiny; some of these broader issues are revisited in the conclusion.

*Source: _wp1448 - conclusion.*

### 3.1    Goodness-of-fit

### _wp1448 - 3.1    Goodness-of-fit

### Goodness-of-fit (GOF) metrics and ROC interpretation
- The DSF approach estimates probits in (1), calculates debt thresholds in (3), and compares debt levels to these thresholds period by period through the estimation sample; when the actual debt indicator is above the threshold, the procedure “calls” a crisis for the next period and compares calls with actual crises.
- The ROC (Receiver Operating Characteristic) curve is used to evaluate how well a probit model classifies crisis and tranquil periods:
  - y-axis: rate of crises correctly called or ‘true positives’ as a share of crisis observations (corresponding to A/(A+MC) in Table 2).
  - x-axis: rate of false alarms or ‘false positives’ (corresponding to FA/(B+FA) in Table 2).
  - Cutoff of 0 implies 100 percent false positives and true positives; cutoff of 100 percent implies zero false alarms and zero true positives.
  - The further the ROC curve is above the 45 degree line the better; AUROC (area under the ROC) measures overall predictive accuracy (uninformative model = 0.5; perfect predictor = 1).
- The DTA-selected point on the ROC (for α = 1/2) is indicated by a green circle in Figure 1; the AUROC for each single-variable probit is reported in the Figure.
- The loss function (2) for each probit regression is shown in Figure 2; the vertical line corresponds to the probability cutoff P*j chosen by the DTA.
- A discrepancy exists between the DTA-chosen cutoff P*j and the point that actually minimizes the loss when using the WCA: the DTA chooses P*j to minimize loss according to predictions from the probit in equation (1) (using country-specific CPIA and growth), whereas the DTA uses equation (3) (grouped CPIA and sample-average growth) to call crises and calculate GOF. This “DTA cut-off discrepancy” can be fixed by modifying the procedure for choosing P*j, but the discrepancy needs to be tracked to assess performance differences across methods.

### Notes on ROC construction in this work
- The AUROC is asymptotically normally distributed.
- The paper draws “alpha” ROCs by calculating the optimal cutoff probability as α (the weight on missed crises) varies from 1 to 0, and then calculating associated GOF; this “alpha” ROC is a convex version of the standard ROC. For the WCA only the “alpha” ROC can be calculated because each debt indicator in the WCA has a different probability cutoff.

*Italicized source line:* _Source: _wp1448 - 3.1    Goodness-of-fit_

---

### 3.2    Limitations of the DSF

### Key shortcomings highlighted
- Limited room for country-specific characteristics:
  - Debt thresholds are calculated using a LIC-average GDP growth rate and three CPIA categories rather than the continuous CPIA variable used in the original probit regression, which ignores information in the continuous variables.
  - Using category averages or sample averages makes inclusion of further explanatory variables or continuous covariates infeasible because it would proliferate debt thresholds.
- Aggregation ambiguity across multiple debt-burden measures:
  - Running a multivariate probit instead of five separate probits shows a problem: in the multivariate specification none of the debt variables is significant (final column of Table 1).
  - Table 4 indicates correlations among the five debt burden measures that are high enough to be problematic, though not necessarily rendering extraction of further information futile.
  - The five-variable probit has a substantially higher log-likelihood than any single-debt-measure probit, implying potential gains from better aggregation methods than those currently used.

*Italicized source line:* _Source: _wp1448 - 3.1    Goodness-of-fit_

---

### 4    The DSF Method of Aggregation: the WCA

### Definition and operational rule
- The DSF worst-case aggregator (WCA): a country in which any one indicator breaches its threshold under the baseline scenario is considered at high risk of debt distress.
- Operationally: a signal is issued (a crisis is called) if and only if Debt_j ≥ D_DTA_j for any j; GOF is then calculated following Table 2 and equation (3).

*Italicized source line:* _Source: _wp1448 - 3.1    Goodness-of-fit_

---

### 4.1    Bias of the WCA

### Empirical bias and its source
- The WCA is biased and “too conservative” in the sense that it yields too many false alarms and too few missed crises given the loss function it is meant to minimize.
- Cause: the WCA lacks its own probability cutoff and instead applies the variable-by-variable cutoffs (chosen to minimize loss for single-variable probits) to the WCA; because any one variable breaching its cutoff triggers a crisis call, the WCA calls crises too often relative to the loss function.

### Quantification of bias (sample frequencies and optimal cutoffs)
- In the sample:
  - Individual debt measures call crises between 13 percent (for DRev) and 40 percent (for DsRev) of the time.
  - The WCA calls crises 56 percent of the time (Table 5).
- ROC analysis for the WCA (Figure 3):
  - The DTA-chosen point (α = 1/2) is shown by a circle; the point that minimizes the loss when using the WCA is shown by an ‘x’.
  - The optimal point corresponds to variable-by-variable cutoffs ranging from 18 to 20 percent for the five debt variables (column 3 of Table 6).
  - This contrasts with variable-by-variable cutoffs of around 10-14 percent that minimize the loss function when applied to each isolated debt measure (column 2).
  - The difference (about 12 percent versus 19 percent) is a measure of the WCA bias: using the debt thresholds associated with column 2 leads to more false alarms than appropriate given equal weights on missed crises and false alarms.
  - A cutoff of about 19 percent would minimize the loss function when using the WCA.
- Implications for debt thresholds (column 4 of Table 6; example):
  - For medium policy CPIA = 3.5, the threshold on debt/GDP goes from 30 to 49 percent when moving from the biased to the unbiased (WCA-optimal) cutoffs.
- Weighting interpretation:
  - The optimal probability cutoff nearly coincides with the chosen one when α = 0.67; i.e., if the weight on missed crises is twice the weight on false alarms, the current DSF thresholds are approximately optimal.
- Remedies:
  - Raise debt thresholds to the higher values reported in Table 6 to yield missed-crises/false-alarms frequencies consistent with α = 1/2.
  - Alternatively, adopt a higher weight on missed crises in the loss function (e.g., α = 2/3) to justify current lower thresholds.
  - Transparency about the implicit beliefs reflected in the WCA and explicit specification of loss-function weights is recommended.

*Italicized source line:* _Source: _wp1448 - 3.1    Goodness-of-fit_

---

### 4.2    Accuracy of the WCA

### Comparative GOF results
- The WCA’s AUROC and loss value are very close to those of the best individual debt measure (DsExp):
  - Figure 4 shows the ROC for DsExp and for the WCA.
  - The AUROC (at the points chosen by the DTA) for the WCA is actually a bit lower than using just DsExp, indicating no in-sample gains in accuracy from the additional information in the WCA.
- Paradox: the multivariate probit had a substantially higher log-likelihood than any single-debt-measure probit (Table 1), suggesting that more sensible aggregators than the WCA may do better than any single debt measure.
- Challenge: the approach of inverting univariate probits to calculate debt thresholds lends itself to the WCA but not to multivariate probits or other aggregators because thresholds for each debt variable would depend on all the others and be complex.

*Italicized source line:* _Source: _wp1448 - 3.1    Goodness-of-fit_

---

### 5    The Probability Threshold Approach (PTA)

### Concept and operationalization
- Express thresholds in terms of probabilities rather than debt ratios:
  - Estimate model (1) and retrieve estimated probabilities of debt distress based on country-specific CPIA scores, growth rates, and debt projections.
  - The tolerable threshold for the probability of debt distress (previously used in (3) to infer a debt burden threshold) becomes the threshold against which projected probabilities are compared.
  - If projected probability P_j ≥ P*j, the country is considered at high risk of debt distress.
- Procedure called the “probability threshold approach” (PTA):
  - Run the five probit models (1) with debt measures inserted one at a time.
  - For each debt measure j, calculate estimated probabilities P_j and iterate to find an optimal P*j that minimizes the same loss function (2) when crises are called if and only if P_j ≥ P*j.

### Advantages over the DSF thresholds/WCA
- The PTA:
  - Enables use of country-specific CPIA scores and continuous covariates (e.g., growth), and allows adding explanatory variables.
  - Avoids the move from probit probabilities to debt burden thresholds, increasing transparency.
  - Facilitates aggregation across indicators in a manner that naturally accommodates country-specific information.
- Comparative GOF example (best single debt variable DsExp):
  - Figure 5 compares the DTA for DsExp with the PTA for the same variable.
  - The AUROC increases from 0.792 to 0.814 when using the PTA for DsExp.
  - The value of the loss function falls (text truncates before reporting the exact loss-function values).

*Italicized source line:* _Source: _wp1448 - 3.1    Goodness-of-fit_

### 0.31 for theDTAto 0.25 with thePTA.  This gain can be decomposed into two components.  The

### How to Aggregate the Debt Indicators

### Decomposition of Fit Gain from DTA to PTA
- Move from 0.31 for the DTA to 0.25 with the PTA. This gain can be decomposed into two components:
  - Reduction in loss to 0.26 from choosing the optimum cutoff while still using the DTA (eliminating the DTA cut-off discrepancy). This is the move from the green circle to the red x on the DTA ROC in Figure 5.
  - Further reduction to 0.25 reflects the gain from using continuous values of CPIA and Growth rather than grouping the CPIA and using the LIC-average value of Growth (the move from the red x on the DTA ROC to the blue circle on the PTA ROC).
- The right-hand panel of Figure 5 illustrates the same points with the loss functions: moving from the green circle (DTA chosen) to the red x (DTA optimal) eliminates the cut-off discrepancy; moving from the red x to the blue circle (PTA optimal) captures the gain from country-specific CPIA and Growth.

### Advantages of the Probability (Probit) Approach
- Solves the DSF problems identified in Section (3):
  - Enables use of country-specific covariates.
  - Facilitates use of standard multivariate methods.
- Probability approach substantially improves goodness-of-fit.
- Facilitates aggregation of information from different debt burden measures: can include more than one debt measure in the probit regression and compare predicted probability to a tolerable threshold.

### Aggregation Framework and Composite Indicator Definition
- Consider composite indicators as specific linear combinations of the five debt indicators:
  - CI_k = sum_{j=1 to 5} w_j × Debt_j
- Performance is judged by including the composite in equation (1) (replacing the debt measures), interacting with the middle-income-country dummy, and keeping both Growth and CPIA, then analyzing goodness-of-fit measures.

### Equal Weights Composite Indicator (CI_EW)
- Definition:
  - CI_EW = sum_{j=1 to 5} 0.2 × Debt_j
- Empirical findings:
  - CI_EW is much more accurate than the WCA: larger AUROC and lower loss function (left-hand panel of Figure 6).
  - The difference in accuracy is statistically significant at the 1 percent level.
  - Bootstrap evidence: drawing 1,000 bootstrap samples, in only 3 of the 1,000 samples is the AUROC of the WCA higher than that of the CI_EW.
  - Even using optimal cutoffs, the AUROC of the WCA is lower than that of the CI_EW with a p-value of 0.075.
- Interpretation:
  - Putting aside bias and data-based weighting questions, the WCA is inaccurate relative to a simple average.

### Multivariate Probit Composite Indicator (CI_MP)
- Definition:
  - CI_MP = sum_{j=1 to 5} β_j × Debt_j, where β_j are coefficients from the multivariate probit (last column of Table 1).
- Empirical findings:
  - CI_MP is more accurate than CI_EW: AUROC rises from 0.84 to 0.87 (Table 7).
  - BIC increases from 333.0 to 372.4 when moving from CI_EW to CI_MP; this increase is statistically significant (bootstrap: in only 18 out of 10,000 samples is the BIC of the CI_MP lower than that of the CI_EW, p-value = 0.018).
  - DsRev has the largest weight; DExp has a small weight; DRev has a negative weight. These weights are imprecisely estimated.
- Interpretation:
  - Improved in-sample accuracy may reflect overfitting given multicollinearity; penalized fit criteria (BIC) favor more parsimonious models.

### More Parsimonious Data-Based Models
- Motivation:
  - We seek models with parameters estimable with more confidence and that provide as good or better goodness-of-fit than CI_MP.
  - Collinearity makes CI_MP weights imprecise and sensitive to sample variation.

#### Step-wise Models
- Approach:
  - Examine BIC for every possible probit model with from one to five debt variables (31 models).
- Findings:
  - Model with lowest BIC has just one debt variable: DsRev.
  - Best two-variable model includes DsRev and DExp.
  - Either of these (DsRev alone, or DsRev+DExp) is significantly better (at the 10 percent level) than full five-variable CI_MP and many other parsimonious models.
  - Five parsimonious models have performance (BIC) statistically indistinguishable from CI_EW at the 10 percent level.
- CI_SW (best two-variable model, DsRev and DExp):
  - AUROC almost identical to CI_MP and CI_EW.
  - BIC is significantly lower (at 1 percent level) than CI_MP, reflecting greater parsimony.
  - Coefficients are tightly estimated and statistically significant conditional on the model.

#### Equal-Weight-Prior Composite Indicator (CI_EWP)
- Rationale:
  - Start from CI_EW as a prior belief that all five debt variables deserve attention.
  - Demand statistical evidence for deviations from equal weights.
- Estimation strategy:
  - Define CI_EW and run probits including CI_EW plus up to four of the five individual debt measures (omit one variable to avoid perfect collinearity). Examine BIC across 30 such models.
- Findings:
  - Data reject less parsimonious models; only two single-variable deviations are competitive with CI_EW: DsRev and DExp.
  - Best (lowest-BIC) model again has DsRev.
  - CI_EWP weights: 0.2 for four debt variables for which there is no clear evidence against equal weights, and 0.55 for DsRev.
  - CI_EWP shows ROC, AUROC, and loss function reported in Figure 6 (panel c) and Table 7 (column 4).
- Model selection conclusions:
  - Choice between CI_EW and CI_EWP is not obvious:
    - CI_EW is a restricted version of CI_EWP and has slightly worse fit (slightly lower AUROC).
    - BIC of CI_EW is lower, suggesting CI_EWP does not improve fit enough to justify extra parameters.
    - Neither difference is statistically significant.

### Model Selection and Practical Recommendations from the Analysis
- Simple equal-weight averaging (CI_EW) outperforms the worst-case approach (WCA) and is a strong baseline:
  - AUROC: CI_EW = 0.84; CI_MP = 0.87 (improvement), but BIC penalizes CI_MP.
  - BIC: CI_EW = 333.0; CI_MP = 372.4.
- Parsimonious models centered on DsRev perform well:
  - Single-variable model with DsRev has the lowest BIC among considered probit specifications.
  - Best two-variable model (DsRev + DExp) has nearly as low a BIC.
- If one prefers a prior that all debt measures matter, CI_EWP gives modest adjustment (0.55 weight to DsRev, 0.2 to others) but does not produce statistically significant gains over CI_EW by BIC.
- Practical guidance:
  - Prefer CI_EW as a robust, simple aggregator when avoiding overfitting and when multicollinearity is a concern.
  - Consider parsimonious models with DsRev (and possibly DExp) when seeking models with tighter parameter estimates and lower BIC.
  - Use penalized goodness-of-fit criteria (e.g., BIC) to trade off in-sample accuracy against the risk of overfitting.

*Source: IMF working paper section: "How to Aggregate the Debt Indicators" — extracted content unit.*

### 6.4    Summary on model selection

### 6.4    Summary on model selection

### Model comparison and selection
- Five aggregators examined for accuracy: WCA, CI_MP, CI_EW, CI_SW, and CI_EWP.
- Clear losers:
  - WCA is less accurate and biased relative to CI_EW.
  - CI_MP has a statistically-significantly higher BIC than the final three models, implying its slightly smaller errors do not justify the larger number of estimated parameters.
- No clear single winner among the remaining models:
  - CI_EW has the lowest BIC and CI_EWP has the highest BIC, but the difference is not statistically significant.
  - The data do not identify which specific CI_SW or CI_EWP specification is reliably best within each family.
- Model parsimony guidance:
  - Any chosen model should have at most two variables singled out.
  - The generally most promising candidates appear to be DsRev and DsExp.
- Model uncertainty:
  - Each of the five debt variables is included in at least one model whose performance cannot be statistically distinguished from other competitive models.
  - Example overlap: CI_SW and CI_EW predict, respectively, 161 and 167 debt distress episodes, with an overlap of 141 events.

### Trade-offs in choosing aggregation approach
- Choice among CI_EW, CI_SW, and CI_EWP approaches depends on:
  - Strength of prior beliefs that all five variables should be included in the model.
  - Tolerance for sample-dependence with respect to specification (variables, parameter values).
- Authors’ judgment favors CI_EW because:
  - It accepts the DSF view that none of the five variables should be neglected.
  - It avoids strong dependence of which specific variables and weights are selected on small sampling variations.
- There is a scenario with no good model:
  - Weak priors that all five variables matter combined with low tolerance for sample-specific specification yields no satisfactory model.

### Revisiting the loss function and implicit weighting
- Loss function originally expressed as in equation (2) with notional equal weights on false alarms and missed crises.
- The conventional implementation multiplies the weight on false alarms α by the number of missed crises as a share of total crises, and (1−α) multiplies the number of false alarms as a share of tranquil periods.
- Sample composition and implied weights:
  - In the authors' sample, only 12.1 percent of observations are crisis observations.
  - There are 186 tranquil periods and 17 crisis periods.
  - The ratio (B+FA)/(A+MC) = 10.9, implying the DSF “equal” weighting scheme actually weights missed crises 10.9 times more than false alarms.
  - In addition, the conservative bias in the WCA doubles the relative weight on missed crises.
  - Combining these factors, the DSF implicitly weighs each missed crisis as roughly 22 times more important than each false alarm.
- Implications of truly equal weights on observations:
  - Setting α′ = (1−α′) for CI_EW yields an optimal cutoff extremely high, between 0.61 and 0.79.
  - The loss is only trivially lower if a cutoff of 1 (which would call no crises) is used.
  - Using cutoffs from earlier sections with truly equal-weighted loss would produce much higher losses.

### Conclusions, diagnostics, and recommended remedies
- Main empirical conclusions:
  - The DSF approach of using debt thresholds and limited use of country-specific covariate values results in a loss of information and reduced prediction accuracy.
  - The WCA has two distinct problems:
    - It is implicitly biased (more conservative) and calls crises more often than justified by the purported loss-function weights.
    - It is statistically inaccurate; alternative simple aggregators can yield a better in-sample combination of missed crises and false alarms.
  - The DSF’s loss function implementation creates an additional implicit conservative bias because weights are applied to crisis observations as a share of total crises and tranquil periods respectively, yielding much larger implicit weight per missed crisis observation.
- Suggested solutions and policy options:
  - Fix the bias in the WCA by raising thresholds on each individual debt indicator so that, when WCA is used, the frequency of false alarms falls and the WCA minimizes the intended loss function.
  - Alternatively, acknowledge that the conservative bias might be desirable if other biases exist (e.g., overly optimistic growth projections or public investment benefits) or if the expected cost of a false alarm is much lower than the cost of a missed crisis—but these arguments require explicit discussion rather than obscuring the bias.
  - Improve accuracy by replacing or complementing WCA with a multivariate indicator:
    - Simply averaging the five debt indicators into a single index (CI_EW) produces forecasts more accurate than the WCA (fewer missed crises and/or fewer false alarms).
  - Consider modest data-driven weighting of the five variables, but recognize in-sample improvements may reflect sample vagaries rather than real gain.
  - Broader reforms could consider alternative approaches, for example:
    - Emphasizing the fiscal reaction function in determining sustainability.
    - Country-specific frameworks that focus on public investment and LIC-specific features.
- Guidance on judgment versus model use:
  - Statistical models remain valuable because crises are infrequent and feedback to judgment is limited; heuristics and biases can generate large systematic errors in subjective judgments.
  - Good multivariate predictors should serve as useful tools and be combined with contextual judgment.

*Source: _wp1448 - 6.4    Summary on model selection*

### References

### _wp1448 - References

### ROC analysis and DTA/PTA performance
- AUROC values reported:
  - DGDP: 0.67 8 (figure label shows "auroc: 0.678")
  - DExp: 0.749
  - DRev: 0.653
  - DsExp: 0.792
  - DsRev: 0.762
- For DsExp (best individual debt measure):
  - PTA optimal: Loss= 0.25; auroc= 0.814
  - DTA chosen: Loss= 0.31; auroc= 0.792
  - DTA optimal: Loss= 0.26
- WCA (weights of composite average) performance:
  - WCA chosen: Loss= 0.29; auroc= 0.78
  - WCA optimal: Loss= 0.28
- Composite indicators compared:
  - CI_ew optimal: Loss= 0.23; auroc= 0.84
  - CI_mp: Loss= 0.21; auroc= 0.87
  - CI_sw: Loss= 0.22; auroc= 0.86
  - CI_ewp: Loss= 0.21; auroc= 0.86
  - WCA: Loss= 0.29; auroc= 0.78

### Optimal cutoffs and loss-function values
- Loss at chosen points (Figure 2):
  - DGDP: Loss at chosen point = 0.35
  - DExp: Loss at chosen point = 0.40
  - DRev: Loss at chosen point = 0.40
  - DsExp: Loss at chosen point = 0.31
  - DsRev: Loss at chosen point = 0.32
- Table 6: Optimal probability cutoffs (minimizing loss function (2), α= (1−α) as used in paper):
  - DGDP: 10.4%
  - DExp: 10.4%
  - DRev: 13.6%
  - DsExp: 14.0%
  - DsRev: 11.2%
  - WCA: 19.0%

### Debt thresholds and policy-group cutoffs
- Table 3: Debt thresholds (PV of PPG external debt in percent of GDP / Exports / Revenue; Debt service in percent of Exports / Revenue)
  - Estimated minimizing the loss function (probability cutoffs): 10.0%, 10.0%, 14.0%, 14.0%, 11.0%
  - Weak policy (CPIA≤3.25): 24 / 13 / 0? / 18 / 5? / 17 / 14 (table layout indicates multiple cells; preserve table-level numeric entries below)
    - Weak policy (CPIA≤3.25): 24 13 0 18 5 17 14
  - Medium policy (3.25< CPIA <3.75): 30 16 32 17 19 16 19
  - Strong policy (CPIA≥3.75): 36 19 62 50 21 18
- IMF and WB 2012 (Table 3, p. 20) probability cutoffs: 14.0%, 13.0%, 15.0%, 15.0%, 14.0%
  - Corresponding thresholds (Weak/Medium/Strong policy rows): 
    - Weak: 28 13 11 84 18 17
    - Medium: 36 17 92 17 20 20
    - Strong: 44 22 62 50 22 24
- Under the current DSF (actual thresholds reported by IMF and World Bank 2012, Table 3, p. 20):
  - Weak policy: 30 10 02 00 15 25
  - Medium policy: 40 15 02 50 20 30
  - Strong policy: 50 20 03 00 25 35
- Notes: The table explains differences arise from slight differences in the loss function weights α and sample.

### Baseline probit regression results (Table 1 summary)
- Sample size: Observations = 529 for all regressions.
- Pseudo-R2 values by specification (columns 1–6): 0.17, 0.18, 0.17, 0.20, 0.22, 0.26
- Log-Likelihood by column: -162.94, -160.04, -161.72, -155.49, -153.19, -145.42
- BIC by column: 357.24, 351.43, 354.79, 342.34, 337.73, 372.36
- AUROC by column: 0.68, 0.75, 0.65, 0.79, 0.76, 0.78
- Loss (eq. 2) by column: 0.34, 0.40, 0.40, 0.31, 0.32, 0.29
- Selected coefficient estimates and significance (standard errors in parentheses):
  - CPIA: -0.454*** (0.116); -0.393*** (0.116); -0.410*** (0.115); -0.353*** (0.116); -0.399*** (0.117); -0.402*** (0.124)
  - Growth: -7.025*** (1.753); -7.107*** (1.792); -7.326*** (1.755); -6.846*** (1.814); -7.104*** (1.786); -6.814*** (1.898)
  - DGDP: 0.333*** (0.097) in column 1; DGDP×MIC entries include 0.013 (0.136) and -0.249 (0.235)
  - DExp: 0.227*** (0.082) in column 2; DExp×MIC includes 0.363** (0.151) and 0.564 (0.561)
  - DRev: 0.281*** (0.087) in column 3; DRev×MIC entries shown
  - DsExp: 0.450*** (0.120) in column 4; DsExp×MIC entries shown
  - DsRev: 0.697*** (0.149) in column 5; DsRev×MIC includes -0.363** (0.168) and -0.090 (0.459)
- Variable definitions provided:
  - CPIA: Country policy and institutional assessment
  - Growth: Real GDP growth
  - DsRev: The ratio of debt service to revenues
  - DsExp: The ratio of debt service to exports
  - DRev: The ratio of the NPV of debt to revenues
  - DExp: The ratio of the NPV of debt to exports
  - DGDP: The ratio of the NPV of debt to GDP
  - MIC: middle-income country dummy

### Correlations and in-sample frequencies
- Pairwise correlations among five debt indicators (Table 4; sample of 529 observations; non standardized debt variables; * significant at 1%):
  - DGDP with DExp: 0.5523*
  - DGDP with DRev: 0.7872*
  - DGDP with DsExp: 0.3892*
  - DGDP with DsRev: 0.5355*
  - DExp with DRev: 0.6792*
  - DExp with DsExp: 0.6689*
  - DExp with DsRev: 0.3003*
  - DRev with DsExp: 0.3692*
  - DRev with DsRev: 0.5721*
  - DsExp with DsRev: 0.6630*
- In-sample frequencies (Table 5): Total observations = 529; other numeric strings reported across table rows (preserved as presented): 
  - Row entries include sequences: 0 4 6 5 3 3 7 1 2 8 3 9 1 7 4 4 1 6 4 9 3 6 9 9 6 3 0 0 1 6 5 2 2 7 2 3 8; 1 6 4 2 7 3 7 3 9 2 5 4 3 2 1 2 4 4 0 1 7 4 7 6 5 8; Total 529 364 165 430 994 597 039 313 631 721 223 3296

### Bias and threshold comparisons (Table 6 and Table 7)
- Table 6: Debt thresholds computed using average LIC GDP growth rate, CPIA = 3.5 (medium policies):
  - Using Pj (optimal cutoffs from column 2):
    - DGDP: Debt threshold = 30
    - DExp: Debt threshold = 16 3? 285 (table cells contain sequences; preserved as shown)
    - DRev: Debt threshold = 21 7 284
    - DsExp: Debt threshold = 19 24
    - DsRev: Debt threshold = 16 23
  - Using P = 19%:
    - DGDP: 49
    - DExp: 85
    - DRev: 84
    - DsExp: 24
    - DsRev: 23
- Table 7: Weights of composite indicators (weights refer to standardized debt variables) and goodness-of-fit (in-sample):
  - CI_EW weights:
    - DGDP: 0.2000
    - DExp: 0.2000
    - DRev: 0.200
    - DsExp: 0.200
    - DsRev: 0.200
    - AUROC: 0.84
    - BIC: 333.02
    - LOSS: 0.23
  - CI_MP weights:
    - DGDP: 0.208
    - DExp: 0.049
    - DRev: -0.017
    - DsExp: 0.119
    - DsRev: 0.524
    - AUROC: 0.87
    - BIC: 372.36
    - LOSS: 0.21
  - CI_SW weights:
    - DGDP: 0.000
    - DExp: 0.169
    - DRev: 0.000
    - DsExp: 0.000
    - DsRev: 0.639
    - AUROC: 0.86
    - BIC: 337.74
    - LOSS: 0.22
  - CI_EWP weights:
    - DGDP: 0.200
    - DExp: 0.200
    - DRev: 0.200
    - DsExp: 0.200
    - DsRev: 0.550
    - AUROC: 0.86
    - BIC: 338.44
    - LOSS: 0.21
  - WCA:
    - AUROC: 0.78
    - BIC: 372.36
    - LOSS: 0.29

### Model selection and BIC rankings (Table 8 and Table 9)
- Table 8: BIC-based ranking (lower numbers are better). Top BIC values shown:
  - CI_EW: 333.0
  - DsRev: 337.7 (appears as best single-variable comparator)
  - CI_SW: 337.7 / 338.4 entries also present
  - Several model BICs listed up to values in the 370s; CI_EW is noted as benchmark.
- Table 9: Guidance for choosing aggregating rule based on prior that all five variables matter and tolerance for sample dependence:
  - High tolerance, Strong prior that all five matter: CI_EWP or CI_EW (CI_MP) (WCA)
  - Low tolerance, Weak prior: CI_SW
  - Note: CI_MP and WCA printed in gray in original table because they are statistically less accurate than CI_EW.

*Italic: Source: _wp1448 - References (figures and tables and accompanying notes).*

---


_Source: https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2014/_wp1448.pdf_