How Effectively Can Current LLMs Analyze Macrofinancial Issues?
IMF Working Papers, February 27, 2026
Source details
- Canonical URL
- How Effectively Can Current LLMs Analyze Macrofinancial Issues?
Other formats
Bibliographic details
- Authors: Paola Ganum, Tohid Atashbar
- Published: February 27, 2026
- Series: IMF Working Papers
- DOI: https://doi.org/10.5089/9798229038935.001
Summary
- Empirical evaluation of current Large Language Models (LLMs) on macrofinancial coverage in IMF Article IV staff reports using human economists' assessments as a benchmark.
- Reports tested span 2016-2024.
- Evaluations cover both qualitative ratings and binary questions.
- Key headline results for 2024 across advanced GPT models:
- Average accuracy on ratings: 71-75 percent.
- Average exact match rate on binary questions: 76-81 percent.
- Paper explores drivers of LLM performance and discusses key limitations such as optimistic bias.
Methodology
- Benchmark: human economists' assessments of macrofinancial coverage in IMF Article IV staff reports.
- Models tested: several GPT models (advanced GPT models highlighted in results).
- Time coverage of reports assessed: 2016-2024.
- Outcome measures:
- Qualitative ratings (accuracy reported as a percentage range).
- Binary questions (exact match rates reported as a percentage range).
- Open-ended questions evaluated for deeper contextual judgment.
Main Findings
- LLMs can meaningfully assist economists in analyzing macrofinancial issues based on measured accuracy and exact match rates.
- Performance metrics (2024, advanced GPT models):
- Ratings accuracy: 71-75 percent.
- Exact match rate on binary questions: 76-81 percent.
- Behavioral tendencies of models:
- LLMs tend to assign higher ratings than human experts.
- LLM ratings are less dispersed than human expert ratings.
- Limitations in qualitative judgment:
- Models struggle with open-ended questions requiring deep contextual judgment.
Drivers of Performance (as discussed)
- Choice of GPT model affects outcomes (advanced GPT models show the reported high-end performance).
- Task type matters: binary questions yield higher exact match rates than open-ended qualitative assessments.
Key Limitations and Risks
- Optimistic bias: models systematically assign higher ratings relative to human experts.
- Reduced dispersion in assigned ratings relative to human experts may mask uncertainty or nuance.
- Challenges remain in open-ended, context-rich judgment tasks, limiting standalone use for complex macrofinancial analysis.
Policy and Practical Implications
- LLMs are a useful assistive tool for economists but are not a substitute for expert judgment in context-dependent assessments.
- Caution advised where open-ended analysis and deep contextual judgment are required.
- Awareness of optimistic bias and compressed rating dispersion is necessary when integrating LLM outputs into macrofinancial analysis workflows.
Paola Ganum, and Tohid Atashbar. "How Effectively Can Current LLMs Analyze Macrofinancial Issues?", IMF Working Papers 2026, 035 (2026), DOI: https://doi.org/10.5089/9798229038935.001
Content in this bundle
- Working Paper