## _wp12122 - 1. Possible Combinations of Predictions and Borrower Behavior

## Source details

**Canonical URL:** [_wp12122 - 1. Possible Combinations of Predictions and Borrower Behavior](https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2012/_wp12122.pdf)

## Other formats

- [Markdown version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2012/_wp12122.pdf.md)
- [Structured JSON version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2012/_wp12122.pdf.json)

---

### I. Introduction — diagnostic frameworks and comparative points
- Central problem: judging whether a borrower will repay a loan; interest in credit ratings and country-risk assessments.
- Diagnostic evaluation frameworks:
  - ROC (Receiver Operating Characteristic) and CAP (Cumulative Accuracy Profile) evaluate systems distinguishing two states (e.g., default vs nondefault).
  - Core idea: trade-off between hits (true positives) and false alarms (false positives) varies with diagnostic threshold.
- Advantages of ROC highlighted:
  - Established rule for choosing diagnostic threshold that maximizes expected net benefits given priors and values of hits/false alarms.
  - Well-developed models for fitting theoretical ROC curves to empirical data.
  - Principal ROC accuracy measure interpretation: area under ROC equals probability that, when selecting one defaulter and one nondefaulter at random, the defaulter has the lower (worse) rating.
  - ROC curve shape is unaffected by prior probabilities; CAP curve changes with proportion of defaulting borrowers.
- Both ROC and CAP:
  - Generate threshold-independent measures of accuracy by considering all possible thresholds.

### II. Illustration: OECD risk ratings predicting IMF recourse (data and setup)
- Data and setup:
  - OECD Country Risk Classifications (eight-point scale 0 (least risky) to 7 (most risky)); list consulted between October 27, 2001 and January 25, 2002.
  - OECD ratings in early 2002 compared with recourse to the IMF during 2002–2010.
  - From 183 countries in IMF World Economic Outlook Database (October 2010), 90 had entered at least one Fund-supported program during 2002–2010.
  - Compiled risk classifications for 161 countries; 82 had recourse to an IMF program during the following nine years, and 79 did not.
- Rationale:
  - IMF recourse used as an observable proxy for distress/default; OECD ratings give larger sample including many developing countries.

### II.A CAP (Cumulative Accuracy Profile) — construction and OECD results
- Construction and definitions:
  - CAP: rank countries from riskiest to safest; for each OECD rating used as threshold, compute hit rate H (proportion of eventual borrowers identified) vs alarm rate M (proportion of all countries identified as future borrowers).
  - Table 1 terminology: hit, false alarm, miss, correct rejection; c denotes ratings threshold; Fd and Fn denote cumulative distribution functions of ratings of borrowers and nonborrowers respectively.
- Observed CAP summary (from Figure 1 and Table 2):
  - Data points correspond to ratings 0 to 7.
  - Using trapezoidal rule joining points by straight lines:
    - S = 0.659 (area proportion under CAP curve).
    - R = S − 0.5 = 0.159.
  - Triangular area Q expression preserved from text: 245.0)16179()1)(1( 2 1 2 1 pQ.
  - Accuracy ratio: AR = R/Q = 0.65.
- Relation to Lorenz and Gini:
  - G = 2S − 1.
  - Relation presented in text: )1(pARG.

- Exact frequencies and cumulative probabilities (from Table 2; preserved formatting and order):
  - (a) Frequencies
    - IMF Program Yes (Y): 3 0 12 5 8 13 50
    - IMF Program No (N): 2 12 12 14 8 4 5 13
    - Sum (T): 2 4 2 13 16 13 12 18 63
  - (b) Probabilities cumulated from right to left
    - H = cum pr(Y): 1.00 .96 .96 .95 .93 .87 .77 .61
    - F = cum pr(N): 1.00 .73 .71 .56 .38 .28 .23 .16
    - M = cum pr(T): 1.00 .85 .84 .76 .66 .58 .50 .39

### II.B ROC (Receiver Operating Characteristic) — construction and OECD results
- Construction and definitions:
  - ROC: shows how hit rate H co-varies with false-alarm rate F (proportion of nonborrowing countries falsely identified as borrowers).
- Observed ROC summary (from Figure 1 and text):
  - Area under the ROC curve when points joined by straight lines is 0.823.
  - Relation between CAP AR and ROC area A: AR = 2A − 1 (Engelmann, Hayden, and Tasche, 2003).
    - Applying to OECD data: AR = 2 × 0.823 − 1 = 0.65 (agrees with CAP calculation to two decimal places).
  - CAP AR computable from ROC curve as ratio R′/Q′ of areas in ROC space; algebra shows identity with CAP AR.

### III. Four properties of ROC analyses not normally available to CAP analyses
- Overview:
  - Authors identify four advantageous ROC properties and show how two (models for fitting/interpreting ROC curves; theory for setting optimal thresholds) can be translated to CAP language. Two advantages cannot be easily transferred: ROC area’s natural interpretation and independence from probability of default.

### III.A Models (fitting ROC and CAP)
- ROC modeling:
  - Detection-theoretic ROC models imply a pair of underlying distributions on a decision variable: f_d(x) conditional on distress/recourse, f_n(x) conditional on non-distress.
  - For risk decreasing with x: H = F_d(c), F = F_n(c), where c is decision threshold.
  - Example fitted normal equal-variance model:
    - Location parameter d′ equals distance between means in units of common standard deviation.
    - Estimated d′ = 1.43 by ordinal regression with IBM SPSS Statistics version 19.
    - Area under normal-model ROC curve A = Φ(d′/√2); for d′ = 1.43, A = Φ(1.43/√2) = 0.844.
- CAP modeling analog:
  - CAP model implies densities f_d(x) and f_{d+n}(x) = p·f_d + (1 − p)·f_n, with p probability of distress.
  - Ordinate of CAP is F_d(c); abscissa is F_{d+n}(c); modeled CAP depicts F_d(c) vs F_{d+n}(c).
  - Example CAP-model parameter d_c: difference between means of f_d(x) and f_{d+n}(x); with underlying difference 1.43, d_c = 0.703 (presented as d_c = 0.70 in figure).
- Note: authors treat fitted normal model for OECD ratings as illustrative, not definitive interpretation.

### III.B Theory of threshold setting (ROC and CAP analogs)
- Need for threshold theory:
  - Practitioners require yes–no thresholds (e.g., investment grade vs noninvestment grade); common rules of thumb ignore priors and unequal costs of misses vs false alarms.
- ROC-based optimal threshold theory:
  - Threshold maximizing expected utility maximizes probability-weighted value of four outcomes. Expected value expressed in Appendix A (equation (1) and (A1)).
  - Threshold that maximizes expected value characterized by likelihood ratio at threshold. Text presents optimality condition in ROC space (preserving text expression style):
    - (f_d(c*)/f_n(c*)) = (1/p) * (V_CR − V_FA)/(V_HM − V_Miss)  (A2 in Appendix A; text-equivalent forms also presented).
  - Interpretation of value terms:
    - CR − FA = value of a correct rejection relative to a false alarm (V_Nn − V_Dn).
    - HM − Miss = value of a hit relative to a miss (V_Dd − V_Nd).
- CAP-space analog:
  - Define CAP likelihood ratio l_C(c) = f_d(c)/f_{d+n}(c).
  - Threshold maximizing expected value corresponds to CAP likelihood ratio given by (A6 in Appendix A). Preserved textual form:
    - l_C(c*) = (f_d(c*) / f_{d+n}(c*)) = (1/p) * (V_CR − V_FA)/(V_HM + V_CR − V_FA − V_Miss)  (A6).
  - Relationship of slopes:
    - Slope of ROC curve at a point equals corresponding likelihood ratio; slope of CAP curve at a point equals CAP likelihood ratio (see Appendix B).
    - Constraint: slope of CAP curve at optimum cannot exceed 1/p (maximum is slope of ideal-performance line).
- Indifference curves and graphical threshold selection:
  - Indifference curves can be plotted in ROC and CAP space to identify optimal thresholds by tangency.
  - Example scenario (from text): correct decision about defaulters three times more valuable than correct decision about non-defaulters; p = 0.5.
    - In CAP space, indifference curves are straight lines with slope 0.5 (from Equation (3)); optimal threshold found where highest indifference curve is tangent to CAP curve.
  - Figure 3 illustration:
    - Solid circle: optimal threshold via indifference-curve tangency.
    - Open circle: sub-optimal threshold chosen by minimizing total rate of errors (maximum vertical distance from positive diagonal).
    - Dashed lines depict indifference curves through those points and tangency lines to fitted CAP/ROC curves.

### III.C Interpretation of area under the curve
- ROC area interpretation (area theorem):
  - ROC area is threshold-independent and equals the probability that a randomly chosen defaulter has a higher risk rating than a randomly chosen nondefaulter.
  - OECD ROC area = 0.82 ⇒ "unbiased accuracy of 82 percent" by the area theorem (text wording preserved).
- CAP area and AR:
  - CAP area is threshold-independent; AR and ROC area A are linearly related: AR = 2A − 1.
  - Area theorem does not apply to CAP accuracy ratio; CAP AR lacks ROC-area’s direct unbiased-percentage interpretation.
- Statistical properties:
  - ROC area index has well-developed statistical properties (e.g., Bamber, 1975); CAP index inherits these via AR = 2A − 1.

### III.D Independence from sample priors
- ROC measures:
  - Measures derived from ROC analysis (area A, d′) do not depend on sample prior probabilities because H and F are ratios with priors in denominators; ROC measures independent of proportions of positives/negatives in test sample.
- CAP dependence and caveats:
  - CAP curve shape depends on proportion of solvent and insolvent borrowers; visual comparisons of CAPs across portfolios with different priors may be misleading.
  - CAP accuracy ratio AR is independent of sample priors because AR = 2A − 1 and A is independent of priors.
  - Practical implication: CAP curves drawn using data from "good times" (low default rates) may mislead about behavior in "bad times" (higher default rates).

### IV. Conclusions — practical recommendations and preferences
- Both CAP's AR and ROC's A provide threshold-independent indices of accuracy, superior to threshold-dependent measures like percent correct.
- AR and A are linearly related: AR = 2A − 1; therefore ROC results and statistical tools apply to CAP AR.
- Advantages of ROC analyses emphasized:
  - Established models for fitting theoretical ROC curves that may reflect underlying rating processes.
  - Established theory for optimal threshold setting using likelihood-ratio conditions that incorporate priors and outcome values.
- CAP improvements:
  - CAP analysis can be augmented with models and threshold-setting theory (as shown) to improve usefulness for evaluating credit ratings.
- Two ROC advantages not easily transferred to CAP:
  - ROC area’s natural interpretation as an unbiased percentage of correct decisions (CAP AR lacks this).
  - ROC curve independence from probability of default (CAP curve depends on default probability).
- Authors’ stated preference: "We therefore prefer the ROC to the CAP."

### Appendix A — setting optimal thresholds in ROC and CAP space (summarized)
- Setup and notation:
  - Four outcomes with values V_Dd, V_Nd, V_Dn, V_Nn; p = Pr(d).
  - Expected value using threshold c (equation (1) and (A1) preserved):
    - E[V(c)] = F_d(c) V_Dd + [1 − F_n(c)] V_Nd p + [1 − F_d(c)] V_Dn (1 − p) + F_n(c) V_Nn (1 − p)  (A1).
- Optimal threshold in ROC space (first-order condition):
  - Differentiation yields first-order optimality condition (likelihood-ratio form). Preserved textual form:
    - (f_d(c*)/f_n(c*)) = (1/p) * (V_CR − V_FA)/(V_HM − V_Miss)  (A2).
- Equivalent condition in CAP space:
  - Relation F_d(c) p + F_n(c) (1 − p) = F_{d+n}(c)  (A3).
  - CAP likelihood-ratio optimality preserved in text as:
    - l_C(c*) = (f_d(c*) / f_{d+n}(c*)) = (1/p) * (V_CR − V_FA)/(V_HM + V_CR − V_FA − V_Miss)  (A6).
- Indifference curves:
  - Expected utility set equal to constant k yields indifference curves in ROC and CAP space; ROC-space expressions depend on V terms and p as shown in Appendix A.

### Appendix B — slope at a point on a CAP curve equals the likelihood ratio
- CAP curve definitions (preserved):
  - Ordinate: ∫_{c}^{∞} f_d(x) dx = F_d(c).
  - Abscissa: ∫_{c}^{∞} f_{d+n}(x) dx = F_{d+n}(c), where f_{d+n} = f_d + f_n weighted by population shares.
- Differentiation yields:
  - d/dc [∫_{c}^{∞} f_d(x) dx] = −f_d(c).
  - d/dc [∫_{c}^{∞} f_{d+n}(x) dx] = −f_{d+n}(c).
- Slope s at the point determined by c equals the ratio of derivatives and hence equals the CAP likelihood ratio:
  - s = f_d(c) / f_{d+n}(c) = l_C(c).

*Content derived from _wp12122 - 1. Possible Combinations of Predictions and Borrower Behavior (source PDF: _wp12122).*

### 1. Possible Combinations of Predictions and Borrower Behavior .............................................6

### 1. Possible Combinations of Predictions and Borrower Behavior .............................................6

### Sections
- 1. Possible Combinations of Predictions and Borrower Behavior .............................................6
- 2. Frequencies of OECD Rating and Corresponding Rates  ......................................................7

### Figures
- 1. CAP and ROC Curves for OECD Risk Ratings and Recourse to IMF .................................6
- 2. Fitted CAP and ROC Curve .................................................................................................10
- 3. Indifference Curves and Optimal Thresholds in CAP and ROC Space ...............................14

### Appendixes
- A. Setting Optimal Thresholds in ROC and CAP Space .........................................................17
- B. Slope at a Point on a CAP Curve Equals the Likelihood Ratio  .........................................20

*Source: _wp12122 - 1. Possible Combinations of Predictions and Borrower Behavior .............................................6*

### References .............................................................................................................

### _wp12122 - References

### I. INTRODUCTION
- Central problem: judging whether a borrower will repay a loan; interest in assessments of credit risk such as credit ratings (Fitch, Moody’s, Standard and Poor’s), credit scores for individuals, and ratings for firms derived from stock prices (see Crouhy, Galai, and Mark, 2000).
- Closely related: ratings of country risk and assessments of the likelihood of fiscal crises (OECD, 2010; Baldacci, Petrova, Belhocine, Dobrescu, and Mazraani, 2011).
- Uses and influence: credit ratings inform lending decisions and are used in rules governing investments by pension funds and collateral accepted by central banks; they have an important and controversial influence on financial markets (IMF, 2010).
- Diagnostic evaluation frameworks:
  - ROC (Receiver Operating Characteristic) and CAP (Cumulative Accuracy Profile) analyses evaluate diagnostic systems distinguishing between two states (e.g., default vs nondefault).
  - Core idea: trade-off between hits (true positives) and false alarms (false positives) varies with diagnostic threshold; good systems have high hit rate for given false-alarm rate.
  - ROC introduced mid-1950s and widely used across fields (Tanner and Swets, 1954; Peterson, Birdsall, and Fox, 1954; Swets, 1986).
  - Financial analysts use ROC to assess credit-ratings systems and crisis indicators (Basel Committee on Banking Supervision, 2005; Engelmann, Hayden, and Tasche, 2003; Sobehart and Keenan, 2001; Van Gool, Verbeke, Sercu, and Baesens, 2011; IMF, 2011).
  - CAP remains the standard in financial practice (Altman and Sabato, 2005; Das, Hanouna, and Sarin, 2009; Flandreau, Gaillard, and Packer, 2010; IMF, 2010; Standard and Poor’s, 2010; Moody’s, 2009).
- Comparative points between ROC and CAP:
  - Both generate threshold-independent measures of accuracy by taking into account all possible thresholds.
  - Advantages of ROC highlighted:
    - Established rule for choosing diagnostic threshold that maximizes expected net benefits given priors and values of hits/false alarms.
    - Well-developed models for fitting theoretical ROC curves to empirical data.
    - Principal ROC accuracy measure has a natural interpretation: area under ROC equals probability that, when selecting one defaulter and one nondefaulter at random, the defaulter has the lower (worse) rating.
    - ROC curve shape is unaffected by prior probabilities; CAP curve changes with proportion of defaulting borrowers.

### II. AN ILLUSTRATION: OECD RISK RATINGS AS PREDICTORS OF BORROWING FROM THE IMF
- Data and setup:
  - OECD Country Risk Classifications intended to estimate likelihood of countries servicing external debt and used to set minimum permissible interest rates on export-credit agency loans.
  - For illustration, OECD ratings made in early 2002 compared with a country’s recourse to the IMF during 2002–2010.
  - OECD classifies countries on an eight-point scale from 0 (least risky) to 7 (most risky). List consulted between October 27, 2001 and January 25, 2002.
  - Of 183 countries listed in IMF’s World Economic Outlook Database for October 2010, 90 had entered at least one Fund-supported program during 2002–2010.
  - Compiled risk classifications for 161 countries, of which 82 had recourse to an IMF program during the following nine years, and 79 did not.
- Rationale: IMF recourse used as an observable proxy (default is rarer), and OECD ratings provide larger sample including many developing countries.

### II.A Cumulative Accuracy Profile (CAP)
- Construction and definitions:
  - CAP: rank countries from riskiest to safest; use each OECD rating as a threshold; examine hit rate H (proportion of eventual borrowers identified) vs alarm rate M (proportion of all countries identified as future borrowers).
  - Hit rate and alarm rate computed from frequency of each rating (see Table 2).
  - Table 1 terminology: hit, false alarm, miss, correct rejection; symbol c denotes ratings threshold; Fd and Fn denote cumulative distribution functions of ratings of borrowers and nonborrowers respectively.
- Observed CAP summary (from Figure 1 and Table 2):
  - Data points correspond to ratings 0 to 7.
  - Using trapezoidal rule when joining points by straight lines:
    - S = 0.659 (area proportion under CAP curve).
    - R = S − 0.5 = 0.159.
  - If probability of recourse to the IMF is denoted p, triangular area Q is given by 245.0)16179()1)(1( 2 1 2 1 pQ (expression from text).
  - Hence accuracy ratio AR = R/Q = 0.65.
- Relationship to Lorenz and Gini:
  - Lorenz curve and Gini coefficient connections: G = 2S − 1, and given AR definition, relation )1(pARG (as presented in text).

- Exact frequencies and cumulative probabilities from Table 2 (preserved formatting in text):
  - (a) Frequencies
    - IMF Program Yes (Y): 3 0 12 5 8 13 50 (corresponding to OECD ratings 0 1 2 3 4 5 6 7 as presented)
    - IMF Program No (N): 2 12 12 14 8 4 5 13
    - Sum (T): 2 4 2 13 16 13 12 18 63 (as presented)
  - (b) Probabilities cumulated from right to left
    - H = cum pr(Y): 1.00 .96 .96 .95 .93 .87 .77 .61
    - F = cum pr(N): 1.00 .73 .71 .56 .38 .28 .23 .16
    - M = cum pr(T): 1.00 .85 .84 .76 .66 .58 .50 .39

### II.B Receiver Operating Characteristic (ROC)
- Construction and definitions:
  - ROC: shows how hit rate H co-varies with false-alarm rate F (proportion of nonborrowing countries falsely identified as borrowers).
  - ROC compares H with F rather than with alarm rate M.
- Observed ROC summary (from Figure 1 and text):
  - Area under the ROC curve when points joined by straight lines is 0.823.
  - Relation between CAP AR and ROC area A: AR = 2A − 1 (Engelmann, Hayden, and Tasche, 2003).
    - Applying to OECD data: AR = 2 × 0.823 − 1 = 0.65 (agrees with CAP calculation to two decimal places).
- CAP accuracy ratio interpretable via ROC space:
  - CAP AR can be computed from ROC curve as ratio R′/Q′ of areas in ROC space; algebra shows identity with CAP AR.

### III. FOUR PROPERTIES OF ROC ANALYSES NOT NORMALLY AVAILABLE TO CAP ANALYSES
- Overview:
  - Authors discuss four advantageous properties of ROC analysis not normally available with CAP as traditionally applied.
  - They show how two of these (models for fitting/interpreting ROC curves; theory for setting optimal thresholds) can be translated to CAP language. Two advantages cannot be easily transferred: natural interpretation of ROC area and independence from probability of default.

### III.A Models
- ROC modeling:
  - Many models exist for fitting ROC curves (Egan, 1975).
  - Every detection-theoretic ROC model implies a pair of underlying distributions on a decision variable: fd(x) conditional on distress/recourse to IMF, fn(x) conditional on non-distress.
  - For risk decreasing with x: H = Fd(c), F = Fn(c), where c is decision threshold.
  - Example fitted normal equal-variance model:
    - Location parameter d′ (accuracy index) equals distance between means in units of common standard deviation.
    - Estimated d′ = 1.43 by ordinal regression with IBM SPSS Statistics version 19.
    - Area under normal-model ROC curve A = Φ(d′/√2) where Φ is standard normal cdf; for d′ = 1.43, A = Φ(1.43/√2) = 0.844.
- CAP modeling analog:
  - Every CAP model implies densities fd(x) and fd+n(x) = p·fd + (1 − p)·fn, with p probability of distress.
  - Ordinate of CAP is Fd(c); abscissa is Fd+n(c); modeled CAP depicts Fd(c) vs Fd+n(c).
  - Example CAP-model parameter dc: difference between means of fd(x) and fd+n(x); with underlying difference 1.43, dc = 0.703 (presented as d_c = 0.70 in figure).
- Note: authors do not interpret the fitted normal model for OECD ratings beyond illustrating fitting.

### III.B Theory of Threshold Setting
- Need for threshold theory:
  - CAP analysis traditionally yields measures of accuracy only; lenders/regulators require yes–no thresholds (e.g., investment grade vs noninvestment grade).
  - Rules of thumb sometimes used (Baldacci et al., 2011): minimize total rate of errors (maximize proportion correct), maximize hit-to-false-alarm ratio (signal-to-noise), or choose ROC point closest to (0,1).
  - Shortcoming: these ignore probability of default and unequal costs of misses vs false alarms.
- ROC-based optimal threshold theory:
  - Threshold maximizing expected utility maximizes probability-weighted value of four outcomes (derived from Equation A1 in Appendix A).
  - Optimal threshold c maximizes expression (preserved exact formula from text):
    - ()[1()]()(1    )[1()](1    ) DdNdDnNn HcpVHc pVFcpVFcpV       
      (expression as printed in source; relies on H, F, p, and values V with subscripts per Table 1).
  - Likelihood-ratio characterization:
    - Slope at a point on ROC equals likelihood ratio of underlying distributions at that threshold.
    - Threshold that maximizes expected value corresponds to ROC likelihood ratio given by (Equation (2) in text):
      - MH H FACR R V V p p cl *) ( R     1 (as presented in text)
      - Where FACR V  denotes value of a correct rejection relative to value of a false alarm (VNn − VDn), and MH V  denotes value of a hit relative to a miss (VDd − VNd).
- CAP analogs:
  - Authors derive analogous expected-value expression in CAP terms (referencing equation A5) and define CAP likelihood ratio lC(x) = fd(x)/fd+n(x).
  - Threshold maximizing expected value corresponds to CAP likelihood ratio given by (Equation (3) in text):
    - FACR MH FACR C VV V p cl     1 *) ( (expression as presented in text)
  - Relationship of slopes:
    - Slope of ROC curve at a point equals corresponding likelihood ratio; slope of CAP curve at a point equals CAP likelihood ratio (see Appendix B).
    - Constraint: slope of CAP curve at optimum cannot exceed 1/p (maximum is slope of ideal-performance line).
- Indifference curves and graphical threshold selection:
  - Indifference curves can be used in both ROC and CAP space to identify optimal thresholds.
  - Example scenario: correct decision about defaulters three times more valuable than correct decision about non-defaulters; p = 0.5.
    - Indifference curves in CAP space are straight lines with slope 0.5 (from Equation (3)); optimal threshold given by tangent of highest indifference curve intersecting CAP curve.
  - Illustration in Figure 3:
    - Solid circle: optimal threshold via indifference-curve tangency.
    - Open circle: sub-optimal threshold selected by rule-of-thumb that minimizes rate of errors (maximum vertical distance from positive diagonal).
    - Shorter-dashed lines: indifference curves through sub-optimal points; longer-dashed lines: indifference curves tangent to fitted CAP/ROC curves.

### III.C Interpretation of Area under the Curve
- ROC area interpretation:
  - Area under ROC is threshold-independent and interpretable via area theorem (Green, 1964; Green and Swets, 1966; Egan, 1975).
  - Theorem: ROC area equals proportion of correct decisions of an unbiased observer choosing between the two alternatives analyzed by the ROC.
    - Equivalently: probability that a randomly chosen defaulter has a higher risk rating than a randomly chosen nondefaulter equals area under rating ROC.
  - OECD ROC area = 0.82 ⇒ unbiased accuracy of 82 percent (text: "The area obtained under the ROC for OECD ratings is 0.82, and so by the area theorem these OECD ratings can be said to have an unbiased accuracy of 82 percent").
- CAP area and AR:
  - CAP area is threshold-independent; AR and ROC area are linearly related (AR = 2A − 1).
  - Area theorem does not apply to CAP accuracy ratio; CAP AR lacks the ROC-area's direct unbiased-percentage interpretation.
- Statistical properties:
  - ROC area index has well-developed statistical properties (e.g., Bamber, 1975); many of these properties are shared by CAP index via linear relation.

### III.D Independence from Sample Priors
- ROC measures independence:
  - Measures derived from ROC analysis (area A, d′) do not depend on sample prior probabilities because H and F are ratios that use priors in denominators; ROC measures independent of proportions of positive/negative instances in test sample (Swets, Dawes, and Monohan, 2000, p. 26).
- CAP dependence and caveats:
  - CAP curve shape depends on proportion of solvent and insolvent borrowers; visual comparisons of CAPs across portfolios may be misleading (Basel Committee on Banking Supervision, 2005, p. 30).
  - CAP accuracy ratio AR is independent of sample priors because AR = 2A − 1 and A is independent of priors.
  - Practical implication: CAP curves drawn using data from good times (low default rates) may mislead about behavior in bad times (higher default rates).

### IV. CONCLUSIONS
- Both CAP's accuracy ratio (AR) and ROC's area under the curve (A) provide threshold-independent indices of accuracy, superior to threshold-dependent measures like percent correct.
- AR and A are linearly related: AR = 2A − 1; thus ROC properties and statistical results can be applied to CAP results.
- Advantages of ROC analyses:
  - Well-documented set of models for fitting theoretical ROC curves to empirical data; these models may reflect underlying rating processes.
  - Established theory of optimal threshold setting (likelihood-ratio-based, incorporates priors and value of outcomes).
- CAP improvements:
  - The authors show how CAP analysis can incorporate models and threshold-setting theory to improve usefulness in evaluating credit ratings.
- Two ROC advantages not easily transferred to CAP:
  - ROC area has natural interpretation as unbiased percentage of correct decisions that AR lacks.
  - ROC curve is independent of probability of default; CAP curve shape depends on default probability. 
- Preference:
  - Authors state: "We therefore prefer the ROC to the CAP."

*Italic: Content derived from _wp12122 - References .............................................................................................................*

### Appendix A. Setting Optimal Thresholds in ROC and CAP Space

### Appendix A. Setting Optimal Thresholds in ROC and CAP Space

### Setup and notation
- A credit rater’s judgment: distressed or default (D) or not (N). The borrower outcome: defaults (d) or does not default (n). Four possible outcomes with values denoted V_Dd, V_Nd, V_Dn, and V_Nn.
- Expected value using threshold c:
  - E[V(c)] = Pr(Dd ∩ c) V_Dd + Pr(Nd ∩ c) V_Nd + Pr(Dn ∩ c) V_Dn + Pr(Nn ∩ c) V_Nn
  - With p = Pr(d), and using conditional probabilities:
    - E[V(c)] = Pr(d; c) V_Dd + Pr(n; c) V_Dn p + Pr(d; c) V_Nd (1 − p) + Pr(n; c) V_Nn (1 − p)  (as in equation (1))
- Expressed in terms of cumulative distribution functions F_d and F_n:
  - E[V(c)] = F_d(c) V_Dd + [1 − F_n(c)] V_Nd p + [1 − F_d(c)] V_Dn (1 − p) + F_n(c) V_Nn (1 − p)  (A1)

### Optimal threshold in ROC space (first-order condition)
- Differentiating (A1) with respect to c and setting derivative to zero yields:
  - p (V_Dd − V_Dn) f_d(c) − (1 − p) (V_Nn − V_Nd) f_n(c) = 0
- Rearranged optimality condition (using likelihood ratio l_R(c) = f_d(c)/f_n(c)):
  - l_R(c*) = (1/p) * (CR − FA)/(HM − V)  — expressed in text as:
    - 1/(*) R CR FA HM V p lc pV      (A2)
  - Interpretation of value terms:
    - CR − FA = value of a correct rejection relative to the value of a false alarm
    - HM = value of a hit; HM − V = value of a hit relative to the value of a miss

- Key formula preserved from text (explicit form):
  - (f_d(c*)/f_n(c*)) = (1/p) * (V_CR − V_FA)/(V_HM − V_Miss)  (expressed in the document as equation (A2) with the likelihood ratio l_R)

### Equivalent condition in CAP space
- Relation between F_d, F_n, and population fraction p:
  - F_d(c) p + F_n(c) (1 − p) = F_dn(c)  (A3)  (rearranged in text)
- Rearranging expected value (A1) and substituting (A3) yields (A5):
  - E[V(c)] = [F_d(c) + F_n(c)] p V_Dd + ... (full expression as in (A5) in source)
- First-order condition solved for CAP-based likelihood ratio yields:
  - l_C(c*) = (f_d(c*) / f_{d+n}(c*)) = (1/p) * (V_CR − V_FA)/(V_HM + V_CR − V_FA − V_Miss)  (A6)
  - Text form (preserving symbols):
    - 1/(*) C CR FA HMCRFA V lc pVV       (A6)
- Interpretation:
  - The left-hand side is a likelihood ratio in CAP space; the right-hand side is a function of p and relative values of outcomes (CR, FA, HM, etc.).

### Indifference curves (ROC and CAP)
- Set expected utility equal to constant k and solve for relationship between hit rate F_d and false-alarm rate F_n.
- ROC-space indifference curve (one for each k), expressed in source notation:
  - F_d(c) = [(1)(1) (k p V_Dd p V_Dn ...)] / [() () ...]  (preserving the structure and dependence on V terms and p as in the source)
- CAP-space indifference curves obtained analogously by expressing hit rate in terms of population F_dn.

### Appendix B. Slope at a Point on a CAP Curve Equals the Likelihood Ratio
- CAP curve definitions:
  - Ordinate: ∫_{c}^{∞} f_d(x) dx = F_d(c)
  - Abscissa: ∫_{c}^{∞} f_{d+n}(x) dx = F_{d+n}(c), where f_{d+n} = f_d + f_n weighted by population shares
- Differentiate each with respect to c:
  - d/dc [∫_{c}^{∞} f_d(x) dx] = −f_d(c)
  - d/dc [∫_{c}^{∞} f_{d+n}(x) dx] = −f_{d+n}(c)
- Slope s at the point determined by c equals the ratio of derivatives and hence equals the likelihood ratio:
  - s = f_d(c) / f_{d+n}(c) = l_C(c)

*Appendix A and B: Setting Optimal Thresholds in ROC and CAP Space — source content as provided*

---


_Source: https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2012/_wp12122.pdf_