Appraising Credit Ratings: Does the CAP Fit Better than the ROC?
IMF Working Papers, May 1, 2012
Source details
- Canonical URL
- Appraising Credit Ratings: Does the CAP Fit Better than the ROC?
Other formats
Bibliographic details
- Authors: R. John Irwin, Timothy C Irwin
- Published: May 1, 2012
- Series: IMF Working Papers
- DOI: https://doi.org/10.5089/9781475503517.001
Summary
- ROC and CAP analysis are alternative methods for evaluating diagnostic systems, including assessments of credit risk.
- ROC analysis is widely used across many fields; CAP analysis is more common in finance.
- The paper compares the two methods using the OECD’s country risk ratings to predict whether a country will have a program with the IMF (an indicator of financial distress).
- Both ROC and CAP generate measures of accuracy that are independent of the choice of diagnostic threshold (such as risk rating).
Comparison of ROC and CAP
- ROC analysis:
- Widely used in many fields.
- Has a natural interpretation of its measure of accuracy.
- Produces curves that are independent of the probability of default.
- Includes established theories for fitting models to data and for setting the optimal diagnostic threshold.
- CAP analysis:
- More common in finance.
- Also generates measures of accuracy independent of diagnostic threshold.
- Lacks the independence from the probability of default that ROC curves provide.
- Could potentially incorporate model-fitting and threshold-setting theories analogous to those used in ROC analysis.
Illustration and Empirical Application
- Empirical illustration uses the OECD’s country risk ratings to assess prediction of whether a country will have an IMF program, treated as an indicator of financial distress.
- Both ROC and CAP are applied to this problem to compare their performance and interpretability in a finance-related diagnostic setting.
Key Findings and Methodological Implications
- Both ROC and CAP analyses are useful because they provide threshold-independent accuracy measures.
- ROC’s advantages:
- Clear interpretation of its accuracy metric.
- Independence from the probability of default.
- Existing theoretical tools for model fitting and optimal threshold selection.
- CAP could be enhanced by adopting ROC-like theoretical tools for fitting and threshold selection.
- Nonetheless, the interpretation and probability-independence features of ROC give it advantages unavailable to CAP analysis as currently used.
By R. John Irwin and Timothy C Irwin — Appraising Credit Ratings: Does the CAP Fit Better than the ROC? (IMF Working Paper).