## Section I. Introduction

## Source details

**Canonical URL:** [Section I. Introduction](https://www.imf.org/-/media/files/publications/wp/2026/english/wpiea2026035-source-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2026/english/wpiea2026035-source-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2026/english/wpiea2026035-source-pdf.pdf.json)

---

### Context and objective
- Rapid development of Generative Artificial Intelligence (gen-AI), notably public release of Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini, marks an inflection point with productivity-enhancing potential, particularly where LLMs are complementary to human work (IMF 2024).
- Objective: empirically evaluate current Generative Pre-trained Transformer (GPT) models’ ability to review macrofinancial coverage in IMF Article IV staff reports, using human economists’ assessments as a benchmark.
- Task characteristics: requires technical knowledge, reasoning, and judgement; LLMs are strong at textual tasks but may face challenges when applying domain-specific technical judgment.

### Research design and scope
- Reports studied: Article IV staff reports over 2016-2024, excluding pandemic years 2020-21.
- Task given to LLMs:
  - Read individual staff reports (standardized pdf format).
  - Answer a structured questionnaire and produce an Excel file per report.
  - Provide ratings from 1 to 4 (intermediate ratings in 0.1 increments possible) and justifications on three areas: (i) macrofinancial analysis and coverage in the baseline, (ii) assessment of systemic risks and vulnerabilities, and (iii) mapping from risks and vulnerabilities to policy advice.
  - Answer 40 binary (Yes/No) questions.
- Benchmark: human economists answered the exact same questionnaire.
- Implementation: automated, programmatic workflow via an API to interact with GPT models hosted on the Microsoft Azure platform.

### Models evaluated and prompting
- GPT models tested: GPT-4o, GPT-4.1 (and GPT-4.1-mini), GPT-o1 (reasoning-optimized), and GPT-5 (medium and high effort).
- Prompting: initial standard prompt (“your task is to read the pdf staff report”) refined with economist justification examples and low/high rating anchors to improve performance.

### Key research questions
- (i) How accurate are LLMs relative to the human economist benchmark?
- (ii) What country/report characteristics are associated with higher LLM and human ratings?
- (iii) How do different GPT models perform on this task?
- (iv) What factors increase the likelihood of a match between human and LLM answers?

### Main findings — performance and behavior
- Overall usefulness:
  - Certain GPT models achieve useful — but still limited — levels of accuracy in analyzing IMF staff reports, especially on structured, fact-based questions.
  - GPT-o1, GPT-4.1, and GPT-5 improve over GPT-4o in justifications and agreement with economists’ assessments; GPT-4.1-mini resembled GPT-4o and compared unfavorably to advanced models.
- Accuracy and exact match rates (relative to human benchmark; as defined in Section III):
  - Average accuracy across models: 71-75% on ratings in 2024 (44-74% in 2022-23).
  - Average exact match rate on binary questions: 76-81% in 2024 (72-79% in 2022-23).
  - Note: "accuracy" = (TP + TN) / (TP + TN + FP + FN).
- Confidence and self-assessment:
  - LLM self-assessed confidence levels around 81%-88% in 2024.
  - Highest confidence in questions typically covered in staff reports (banking sector issues, regulation and supervision policies, financial integrity issues, and use of FSIs); only slightly lower for open-ended questions.
- Stability / reproducibility:
  - Repeated tests with same model, prompt, and temperature settings: 93% consistency in binary answers and 83% accuracy in ratings answers in representative tests; average 88% consistency reported across repeated tests.
- Strengths and weaknesses:
  - Strengths: structured, fact-based, instruction-following tasks; long-context capabilities benefit performance.
  - Weaknesses: open-ended questions and ratings requiring complex judgment; lower probability of matching human answers in these cases.
  - Distributional behavior: advanced models align more closely with human assessments but retain a tendency toward optimism and compressed variance relative to humans.
- Prompt refinement effects:
  - Concrete examples of economist justifications and rating anchors substantially improved LLM accuracy; model advances also contributed.

### Contribution and novelty
- Creation of a novel dataset of LLM answers to a granular questionnaire on IMF Article IV staff reports.
- First systematic quantitative evidence on LLM accuracy for this specific, domain-intensive assessment task.
- Bridges literatures on LLM evaluation and macrofinancial surveillance by applying state-of-the-art LLMs to long-form, semi-structured policy documents and comparing outputs to human expert judgments.

### Box 1 — Differences between the models (summary)
- GPT-4o (omni): Released May 2024; multimodal; design goals: speed and natural interaction; positioned as a capable generalist rather than a deep-reasoner.
- GPT-4.1-mini: Released July 2024; cost-efficient, lightweight; less capable in complex reasoning; performance mirrored GPT-4o.
- OpenAI o1 (reasoning-optimized): includes o1 and o1-mini; trained to "think before answering" with improved multi-step reasoning; outperforms on harder rating tasks at cost of time/tokens.
- GPT-4.1 (long-context): Released April 2025; emphasizes long-context comprehension; API supports up to 1 million tokens; better at digesting long documents.
- GPT-5 (unified router + deeper mode): Released August 2025; unified system that adapts between quick responses and longer “think” modes for complex prompts.

### Methodological metrics and correlations (high-level)
- Metrics used: Accuracy, precision, recall, F1 score, exact match, near match (near match allows ± 1 rating point; differences also categorized in 0.5 ranges).
- Accuracy (ratings) binary classification: positive if rating ≥ 3, negative if < 3.
- Regression approaches:
  - Pooled OLS with year fixed effects for rating regressions (examples: Rating_it = 휹 + 휶1 * X_it + 휷 * macro_it + 휶2 * 풀_i + 휽_t + e_it).
  - Logit and probit regressions for likelihood of match (Match_i = 휹 + 휶 * X_i + e_i).
- Key covariates (X_it): recent FSAP in last 5 years; specialized macrofinancial review; participation by macrofinancial economist; well-articulated view of systemic risk; sum of binary coverage variables; presence of a country program; income-group dummies (AE, EM, LIC); banking crisis dummy (2003-2018); government debt percent of GDP; private sector credit to GDP.
- Estimation details: clustered standard errors at country level; country fixed effects included in some specifications.

### Empirical results across GPT models (selected quantitative findings)
- GPT-4o initial observations:
  - Rating accuracy ranged 19%–59% from 2016 to 2024 (Annex I, Table 1).
  - F1 score reached 72% in 2024.
  - Near-match (within 1 point) ~77% for 2023/2024; within 0.5 points for 44.4% in 2023/2024.
  - Binary exact match estimated ~75% on average for 2024.
- Performance gains with model and prompt refinements:
  - GPT-o1, GPT-4.1, GPT-5 with updated prompts achieved higher accuracy on rating questions than GPT-4o for most years (Annex I, Table 2).
  - Example: 2022 reports accuracy rose from 27% with GPT-4o to 56-59% with GPT-o1 and other advanced models.
- Model accuracy and rating distributions (selected numbers):
  - Overall accuracy for 2023 and 2024 reached 74% and 75%, respectively, in GPT-o1 (Annex I, Table 2).
  - GPT-o1 recall often above 90%; precision reached 71% in 2024; overall F1 = 80% in 2024.
  - Exact match rate for overall rating remained low (~0-5% across models in 2024); near match (0–0.5 point) for 2024: GPT-o1 70%, GPT-4.1 74%, GPT-5 66%; within 1 point: GPT-o1 and GPT-4.1 97%-98%, GPT-5 91%.
- Binary questions performance (exact match rates, 2024 averages):
  - GPT-o1: 80.6%.
  - GPT-4.1: 76.3%.
  - GPT-5: 81.3%.
  - Previous years (2022-2023): around 77% average exact match.
- Confidence scores (0–100 requested per answer; averages reported):
  - GPT-o1 and GPT-4.1 combined: ~81%.
  - GPT-4.1 model: 86%.
  - GPT-5 model: 88.4%.
- Reproducibility in representative sample (35 countries across AEs, EMs, LICs using GPT-4.1):
  - Average match rate between first and newer LLM answer: 93% in binary questions; accuracy 83% for rating questions. GPT-4o: 87% and 88% respectively.

### Correlation and regression highlights (selected coefficients and predicted probabilities)
- Positive correlations with higher LLM overall rating:
  - Recent FSAP, macrofinancial review, participation by specialized economists, well-articulated view of systemic risk, cumulative binary coverage.
- Negative correlations:
  - Being EME or LIC relative to AEs; combined Article IV / Program associated with lower ratings.
- Selected coefficients (GPT-o1 probit match regressions):
  - Model_id = GPT-O1: 0.1491*** (Table 4 excerpts).
  - model_id = 4, GPT-5: 0.1517***.
  - Type = 3, Rating: -2.3261*** indicating much lower probability of a match for rating questions.
  - Complexity = 2, Simple: 0.7267*** indicating higher match probability for simple questions.
- Margins / predicted probabilities (Table 5 / Table 10 excerpts):
  - Topic-specific predicted probabilities of a match (examples):
    - Topic = 1, Analytical tools: 0.843*** (0.00962)
    - Topic = 4, General information: 0.969*** (0.0111)
    - Topic = 5, MacFin Integration: 0.0722*** (0.00806)
  - Type-specific predicted probabilities:
    - Type = 1, Factual questions: 0.803*** (0.00701)
    - Type = 2, Open-ended questions: 0.711*** (0.00937)
    - Type = 3, Rating questions: 0.0720*** (0.00805)
  - Complexity:
    - Complexity = 1, Complex: 0.559*** (0.00765)
    - Complexity = 2, Simple: 0.808*** (0.00673)
  - Observations for probit/logit analyses: 24,206.

### Section V — Discussion of results and LLM issues (key implications)
- Performance, accuracy, and reproducibility:
  - Binary questions (2022-2024): average exact match ~72-81% for advanced models.
  - Ratings questions: F1 ~80% in 2024 with GPT-o1.
  - Reported model-specific accuracy in 2024: o1 75%, GPT-5 72%, GPT-4o 59%, GPT-4.1 71%.
  - Repeated-run practical consistency supports use in supportive review workflows but non-determinism motivates validation frameworks (multiple runs or human review).
- Apparent bias and tendency toward optimism:
  - LLMs tend to assign higher average ratings than human benchmarks, compressing variance across income groups and under-assigning lower (more critical) ratings.
  - Possible sources: training data tone, alignment encouraging non-confrontational responses, keyword/section detection without depth evaluation, absence of ‘grade on a curve’ context.
  - Implication: keep a human in the loop; use LLM outputs as preliminary assessments requiring validation.
- Interpretation of nuanced information:
  - No significant factual fabrication observed, but misinterpretation occurs in open-ended questions and in weighing significance (examples provided for Binary-A28 and Binary-A15).
  - Requiring justifications and direct quotes helps human oversight detect interpretive divergences.
  - Practical conclusion: LLMs are more reliable when required to "show their work"; ambiguity remains for context-sensitive judgments.
- Potential future research and refinements:
  - Further prompt refinements (role-based prompting), calibration/batch benchmarking (simulate grading on a curve), Retrieval-Augmented Generation (RAG) with historical 'gold standard' reports.
  - Develop typology of divergences (factual differences; optimistic rating differences; interpretative differences) and expand data sample (extend back to 2008) to better understand drivers of LLM ratings.

### Annex I — Selected tables and numeric details (excerpted figures from model performance tables)
- GPT-4o (Table 1, ratings accuracy and classification metrics):
  - 2016 — Accuracy 47%, Precision 46%, Recall 100%, F1 score 63%
  - 2017 — Accuracy 28%, Precision 27%, Recall 100%, F1 score 43%
  - 2018 — Accuracy 28%, Precision 28%, Recall 100%, F1 score 44%
  - 2019 — Accuracy 19%, Precision 18%, Recall 92%, F1 score 30%
  - 2022 — Accuracy 27%, Precision 25%, Recall 100%, F1 score 40%
  - 2023 — Accuracy 53%, Precision 50%, Recall 100%, F1 score 66%
  - 2024 — Accuracy 59%, Precision 57%, Recall 100%, F1 score 72%
- Cross-model accuracy in rating questions (Table 2):
  - 2016 — GPT-4o 47%, GPT-o1 50%, GPT-4.1 50%, GPT-5 (default effort) 46%
  - 2019 — GPT-4o 19%, GPT-o1 44%, GPT-4.1 44%, GPT-5 34%
  - 2022 — GPT-4o 27%, GPT-o1 56%, GPT-4.1 59%, GPT-5 44%
  - 2023 — GPT-4o 53%, GPT-o1 74%, GPT-4.1 74%, GPT-5 61%
  - 2024 — GPT-4o 59%, GPT-o1 75%, GPT-4.1 71%, GPT-5 72%
- GPT-o1 (Table 3):
  - 2024 — Accuracy 75%, Precision 71%, Recall 90%, F1 score 80%
- GPT-4.1 (Table 4):
  - 2024 — Accuracy 71%, Precision 68%, Recall 88%, F1 score 77%
- GPT-5 (medium effort; Table 5):
  - 2024 — Accuracy 72%, Precision 67%, Recall 97%, F1 score 79%
- Exact match rates across models (Table 6):
  - 2024 — GPT-4o 5%, GPT-o1 5%, GPT-4.1 0%, GPT-5 0%
- Logit/probit highlights (Table 9 / Table 10):
  - Model_id = GPT-O1: positive and significant association with match (e.g., 0.2595*** in logit column (1)).
  - model_id = GPT-5: 0.2693*** in logit column (1).
  - Type = 3, Rating: large negative association with match probability (e.g., -3.9929*** in logit column (3)).
  - Predicted probability for Type = 3, Rating questions: 0.0716*** (0.00801).

Summary conclusion (from empirical results)
- Advanced GPT models (GPT-o1, GPT-4.1, GPT-5) paired with refined prompts substantially improve accuracy and alignment with human economists on structured, factual macrofinancial assessment tasks, achieving accuracy rates in the range of 76–81% on fact-based binary questions and strong recall/precision improvements by 2024; however, exact agreement on ratings remains low, optimistic bias persists, and open-ended/context-sensitive judgments require human validation — implying LLMs are most effective as supportive tools rather than replacements for expert judgment.

*Source: wpiea2026035-source-pdf — IMF working paper content (Sections I, Box 1, IV, V, Annex I).*

### Section I. Introduction ................................................................................................

### Section I. Introduction

### Context and objective
- Rapid development of Generative Artificial Intelligence (gen-AI), notably public release of Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini, marks an inflection point with productivity-enhancing potential, particularly where LLMs are complementary to human work (IMF 2024).
- This paper empirically evaluates current Generative Pre-trained Transformer (GPT) models’ ability to review macrofinancial coverage in IMF Article IV staff reports, using human economists’ assessments as a benchmark.
- The task requires technical knowledge, reasoning, and judgement: LLMs are strong at textual tasks but may face challenges when applying domain-specific technical judgment.

### Research design and scope
- Reports studied: Article IV staff reports over 2016-2024, excluding pandemic years 2020-21.
- Task given to LLMs:
  - Read individual staff reports (standardized pdf format).
  - Answer a structured questionnaire and produce an Excel file per report.
  - Provide ratings from 1 to 4 (intermediate ratings in 0.1 increments are possible) and justifications on three areas: (i) macrofinancial analysis and coverage in the baseline, (ii) assessment of systemic risks and vulnerabilities, and (iii) mapping from risks and vulnerabilities to policy advice.
  - Answer 40 binary (Yes/No) questions.
- Human economists answered the exact same questionnaire to provide the benchmark.
- Implementation: automated, programmatic workflow via an API to interact with GPT models hosted on the Microsoft Azure platform.

### Models evaluated
- GPT models tested: GPT-4o, GPT-4.1 (and GPT-4.1-mini), GPT-o1 (noted as better at reasoning problems), and GPT-5 (medium and high effort).
- Prompting: initial standard prompt (“your task is to read the pdf staff report”) was refined with specific examples of economists’ justifications and low/high ratings to improve performance.

### Key research questions
- (i) How accurate are LLMs relative to the human economist benchmark?
- (ii) What country/report characteristics are associated with higher LLM and human ratings?
- (iii) How do different GPT models perform on this task?
- (iv) What factors increase the likelihood of a match between human and LLM answers?

### Main findings (performance and behavior)
- Overall usefulness:
  - Certain GPT models achieve useful — but still limited — levels of accuracy in analyzing IMF staff reports, especially on structured, fact-based questions.
  - GPT-o1, GPT-4.1, and GPT-5 offer an improvement over GPT-4o, with improved justifications and more agreement with economists’ assessments.
  - GPT-4.1-mini resembled GPT-4o and compared unfavorably to the more advanced models.
- Accuracy and exact match rates (relative to human benchmark; as defined in Section III):
  - Average accuracy across models: 71-75% on ratings in 2024 (44-74% in 2022-23).
  - Average exact match rate on binary questions: 76-81% in 2024 (72-79% in 2022-23).
  - Note: "accuracy" measures the proportion of correct predictions (true positives and true negatives) by the LLM out of all LLM predictions.
- Confidence and self-assessment:
  - LLM self-assessed confidence levels around 81%-88% in 2024.
  - Confidence is highest in questions about issues typically covered in staff reports (banking sector issues, regulation and supervision policies, financial integrity issues, and use of financial sector indicators (FSIs)); only slightly lower for open-ended questions.
- Stability / reproducibility:
  - Repeated tests with same model, prompt, and temperature settings showed high practical consistency: 93% consistency in binary answers and 83% accuracy in ratings answers.
- Strengths and weaknesses:
  - LLMs perform better on structured, fact-based, and instruction-following tasks and with long-context capabilities.
  - LLMs struggle mostly with open-ended questions and with ratings questions that require more complex judgement; probability of matching human answers is lower in these cases.
  - Distributional behavior: more advanced models (GPT-4.1, GPT-o1, GPT-5) align more closely with human assessments but retain a tendency toward optimism and a somewhat compressed distribution at the highest rating end with older models (GPT-4o).
- Prompt refinement effects:
  - Introducing refinements with concrete examples of economist justifications and rating anchors substantially improved LLM accuracy, alongside advances due to model improvements.

### Contribution and novelty
- Creation of a novel dataset of LLM answers to a granular questionnaire on IMF Article IV staff reports.
- First systematic quantitative evidence on LLM accuracy for this specific, domain-intensive assessment task.
- Bridges literatures on LLM evaluation and macrofinancial surveillance by applying state-of-the-art LLMs to long-form, semi-structured policy documents and comparing outputs to human expert judgments.

*Source: wpiea2026035-source-pdf — Section I. Introduction*

### Box 1. Differences between the models

### Box 1. Differences between the models

### Model descriptions
- GPT-4o (omni)
  - Released in May 2024.
  - Multimodal: natively ingests text, images, audio, and video.
  - Design goals: speed and natural interaction; instruction-following without an explicit, slower “reasoning mode.”
  - Positioned in the study as a capable generalist rather than a deep-reasoner.
- GPT-4 (mini)
  - Released in July 2024.
  - Cost-efficient, lightweight alternative with the same context window and multimodal capabilities as GPT-4o.
  - Generally less capable in complex reasoning tasks than larger counterparts; performance mirrored GPT-4o and was less effective at nuanced macrofinancial evaluation.
- OpenAI o1 (reasoning-optimized)
  - Includes o1 and o1-mini variants.
  - Trained to "think before answering," using additional computational resources to work through multi-step problems.
  - Improved complex reasoning and debugging at the cost of time and tokens (internal reasoning consumes context).
  - Outperforms on harder rating tasks despite not being the fastest model.
- GPT-4.1 (long-context)
  - Released in April 2025.
  - Emphasizes long-context comprehension, instruction following, and tool use.
  - API supports up to 1 million tokens; better at using large context and digesting long documents with fewer misses from chunking.
- GPT-5 (unified router + deeper mode)
  - Released in August 2025.
  - Presented as a unified system that knows when to respond quickly and when to think longer for complex prompts or when asked to “think hard.”

### Methodology I. Metrics
- Metrics compared between LLM outputs and human outputs:
  - Accuracy, precision, recall, F1 score, exact match, near match.
- Near match definition:
  - Allows a ± 1 difference in rating level; differences also categorized in ranges of 0.5 points.
- Accuracy (rating questions) formula:
  - Accuracy = (Number of correct predictions) / (Total number of predictions) = (TP + TN) / (TP + TN + FP + FN)
  - Human and LLM answers categorized as positive (value 1) if rating is above or equal to 3 points, negative (value 0) if below 3.
- Accuracy (binary questions):
  - Measured as exact match rate (equation (1) applied where correct predictions mean exact match).
- Precision (rating questions):
  - Precision = TP / (TP + FP)  (eq. (2))
- Recall (rating questions):
  - Recall = TP / (TP + FN)  (eq. (3))
- F1 score (rating questions):
  - F1 = 2 * (Precision * Recall) / (Precision + Recall)  (eq. (4))
- Rationale for using multiple metrics:
  - Accuracy can be misleading with unbalanced classes; precision measures trustworthiness (avoiding 'hallucinating quality'), recall measures coverage (catching actual positives), and F1 balances the two.

### Methodology II. Correlations
- Regression approaches:
  - Pooled OLS with year fixed effects (휽풕) due to unbalanced panel and varying country sample over time:
    - Rating_it = 휹 + 휶1 * X_it + 휷 * macro_it + 휶2 * 풀_i + 휽_t + e_it  (eq. (5))
  - Logit and probit regressions for likelihood of match between LLM and human answers:
    - Match_i = 휹 + 휶 * X_i + e_i  (eq. (6))
- Variables included in X_it (country report characteristics):
  - Dummy: recent Financial Sector Assessment Program (FSAP) in last 5 years.
  - Dummy: report went through specialized macrofinancial review during writing.
  - Dummy: participation in writing by a macrofinancial economist.
  - Dummy: report has a well-articulated view of systemic risk in year t.
  - Variable: sum of binary values in year t (higher value = more comprehensive coverage of macrofinancial angles).
  - Dummy: country had a program at time t (prior: would tend to decrease rating).
- Time-invariant dummies (Yi):
  - One dummy per income group (AE, EM, LIC).
  - Dummy for whether country experienced a banking crisis (using Laeven and Valencia (2018); defined as 1 if experienced a banking crisis over 2003-2018).
- Macroeconomic controls:
  - Level of government debt in percent of GDP.
  - Private sector credit to GDP level.
- Estimation details:
  - Clustered standard errors at the country level.
  - Country fixed effects included in some specifications; excluded from most with country report dummy variables due to collinearity.

### Section IV. Empirical Results across GPT Models
- Overview:
  - Comparison of LLM assessments against human economists' ratings across models: GPT-4o, GPT-4.1, GPT-o1, GPT-5.
  - Analysis covers both ratings and binary questions.

- I. Initial Observations from the GPT-4o Model
  - Distributional tendency:
    - GPT-4o assigned higher and more frequent positive ratings compared to human reviews; fraction of LLM ratings heavily skewed to the right (Figure 2 top row).
  - Accuracy and F1:
    - Overall accuracy for rating questions hovered between 19% and 59% from 2016 to 2024 (Annex I, table 1).
    - F1 score reached 72% in 2024 but was considerably lower in prior years.
  - Near match performance:
    - For the overall rating in 2024 and in 2023, the LLM’s assessment was within one point of the human rating in about 77% of cases.
    - Within 0.5 points for 44.4% of the reports (2024/2023).
    - For 2022, the within-one-point figure was 43.6%.
  - Binary questions:
    - Estimated exact match rate of 75 percent on average across questions for 2024 (Figure 1, last row, right chart), with similar results for 2022-23 and some heterogeneity across questions.

- II. Performance Gains with Model and Prompt Refinements
  - Upgraded models and updated prompt:
    - Tests moved to GPT-5, GPT-4.1, and GPT-o1 with an updated prompt.
  - Comparative accuracy:
    - Annex I Table 2 indicates GPT-o1, GPT-4.1, and GPT-5 achieved higher accuracy in rating questions than GPT-4o for most years.
    - GPT-5 was not as accurate as GPT-o1 and GPT-4.1 over 2016-23.
    - Example: For reports from 2022, accuracy appeared to jump from 27% with GPT-4o to 56-59% with the o1 and other advanced models.

*Source: Box 1. Differences between the models (excerpt), IMF working paper content unit wpiea2026035-source-pdf*

### 4.1 models (44% in GPT-5). The improvement in recent models was even more pronounced for recent years,

### 4.1 models (44% in GPT-5). The improvement in recent models was even more pronounced for recent years,

### Model accuracy and rating distributions
- Overall accuracy for 2023 and 2024 reached 74% and 75%, respectively in the o1 model (Annex I, Table 2).
- Advanced models showed more alignment with human assessments but retained a tendency toward optimism:
  - GPT-o1 and GPT-5 exhibited an optimistic bias manifested as clustering in the upper-middle range of the scale rather than at the ceiling.
  - Advanced models were less likely than humans to assign the absolute highest ratings (negative bar on the right-hand side of the distribution for GPT-o1) but significantly under-assigned lower ratings, resulting in a higher average.
  - GPT-4.1 was the most balanced among the models tried.
  - More advanced models showed a distribution less compressed at the highest end of the scale compared to GPT-4o results.
- A GPT-4.1-mini run in 2024 resembled GPT-4o, with a skew to the right of LLM ratings.

### Classification metrics (GPT-o1 and recent models)
- Recall (ability to identify reports that humans rated highly) for the GPT-o1 model was consistently very strong, often above 90% (Annex I, Table 3).
- Precision improved significantly in recent years, reaching 71% in 2024 (Annex I, Table 3).
- Overall F1 score showed a strong upward trend, reaching 80% in 2024.
- Annex I table 4 and table 5 show similar results for GPT-4.1 and GPT-5 models, indicating improvement in balancing identification of positive cases and avoiding false positives.

### Model-human agreement: exact and near match
- Exact match rate for overall rating remained low, hovering around 0-5% for all models in 2024 (Annex I, Table 6).
- Near match (difference in ranges of 0.5 points):
  - For 2024, about 70% of cases had a difference between 0 and 0.5 points in GPT-o1.
  - 74% in GPT-4.1 for 2024.
  - 66% in GPT-5 for 2024 (see annex I, figure 5a).
  - About 97%-98% of cases had a difference within 1 point for GPT-o1 and GPT-4.1 (91% for GPT-5).
- Near match results were similar when analyzed by the three areas: macrofinancial coverage in the baseline, risk assessment, and policies.
- LLM differs slightly more from economists’ ratings by 0.5 points or more in emerging market and low income countries (Annex I, Figure 2).

### Binary (Yes/No) questions performance
- Binary questions are typically more factual; LLM performance was strong:
  - Average exact match rate across all binary questions for 2024:
    - 80.6% for GPT-o1 model (Annex I, figure 5a).
    - 76.3% for GPT-4.1 model (Annex I, figure 5a).
    - GPT-5 results at 81.3% on average (Annex I, figure 5a).
  - Performance in previous years:
    - Around 77% average exact match rate for 2022 and 2023 (Annex I, Figure 3).
    - 71.5% in GPT-4.1 model (Annex I, figure 4).
    - 79.2% in GPT-5 (Annex I, figure 5a).
- Binary questions covered systemic risk assessment, FSAP integration, coverage of emerging macrofinancial issues, use of analytical tools, sources of financial sector vulnerabilities, and identified areas for policy recommendations.

### Confidence score mechanism
- Prompt modified to request a confidence score from 0 to 100 for each answer to help triage low-confidence items.
- Average confidence scores observed:
  - Around 81 percent on average for GPT-o1 and GPT-4.1 combined analysis.
  - 86% for the GPT-4.1 model.
  - 88.4% for the GPT-5 model.
- LLM signaled slightly lower confidence for open-ended questions where exact match rates are lower.
- High confidence was shown for factual items typically covered in staff reports (banking sector-related vulnerabilities, financial integrity issues (AML/CFT), regulation and supervision policies, and use of FSIs).

### Correlation and regression findings (factors associated with ratings and matches)
- Regression analysis (Table 2 using GPT-5; Table 3 for human ratings) suggests correlations:
  - Positive correlation between having a recent FSAP and higher LLM overall rating.
  - Being an Emerging Market (EME) or a Low-Income Country (LIC) appeared negatively correlated with the LLM’s rating, aligning with human patterns.
  - Having had a banking crisis over 2003-2018, higher values in binary questions, a well-articulated view of systemic risk, a specialized macrofinancial review, and participation by a financial sector-expert economist seem positively correlated with LLM’s overall rating.
  - Similar regressions for human ratings obtained comparable results.
- Probit regressions (Table 4) estimated factors associated with a higher probability of a match between LLM and human answers; logit results are in Annex I, table 9.
- Selected coefficient highlights from Table 4 (probit match regressions):
  - Model_id = GPT-O1: 0.1491*** (column (1)) and similar positive significant coefficients across columns.
  - model_id = 4, GPT-5: 0.1517*** (column (1)) and similar positive significant coefficients across columns.
  - Topic and group effects (examples):
    - Topic = 5, MacFin Integration: -2.4796***.
    - Topic = 4, General information: 0.8690***.
    - Group = 2, EM: -0.1598*** (column (1)).
    - Group = 3, LIC: -0.1195** (column (1)).
  - Type = 3, Rating: -2.3261*** (column (1)) indicating ratings questions have a much lower probability of a match.
  - Complexity = 2, Simple: 0.7267*** (column (1)) indicating simple questions have higher match probability.
- Margins (Table 5, Probit) and converted predicted probabilities indicate:
  - Factual questions have close to 80% probability of a match in 2024.
  - Simple questions have a similar probability.
  - Ratings questions and complex questions have a lower chance of a match.
  - Questions across topics had similar probabilities in the range of 60%-97%, except ratings questions which have around 7% probability in 2024.

### Summary conclusions
- Empirical results indicate rapidly improving technology aiding macrofinancial analysis:
  - Initial models showed modest performance and a strong optimistic tendency.
  - More advanced models (GPT-o1, GPT-5, GPT-4.1) paired with refined prompts deliver higher accuracy and closer alignment with human experts.
  - Models are particularly effective at objective, binary questions.
  - Some optimistic bias persists, and exact agreement on ratings remains low.
- Implication: most effective use is as a supportive tool for human experts rather than a replacement.

*Source: wpiea2026035-source-pdf (IMF Working Papers content provided in the source PDF).*

### Section V. Discussion of Results and LLM issues

### Section V. Discussion of Results and LLM issues

### I. Performance, Accuracy, and Reproducibility
- Key quantitative performance results:
  - For binary (Yes/No) questions, analysis for the 2022-2024 period points to an average exact match rate of approximately 72-81% for some of the more advanced models tested.
  - The F1 scores for the ratings questions reach around 80% in 2024 with the GPT-o1 model.
  - Reported model-specific accuracy observations (2024): o1 75%, GPT-5 72%, 4o 59%, GPT-4.1 71%.
  - Across 2022–24, recall is often between 90–100%; precision improves into the mid-60s/low-70s by 2024, yielding F1 between 77–80%.
  - Summary statement in Conclusions: advanced models such as GPT-o1, GPT-4.1, and GPT-5 demonstrate accuracy rates in the range of 76–81% on fact-based, binary questions.
- Reproducibility observations:
  - In repeated tests using the same model, prompt, and temperature settings, observed average 88% consistency in answers.
  - Representative sample test details: a sample of 35 AEs, EMs, and LICs; using the GPT-4.1 model, the average match rate between the LLM’s first answer and the LLM’s newer answer was 93% in binary questions and the accuracy was 83% for rating questions. Using the GPT-4o model, those figures are 87% and 88% respectively.
  - Practical implication: outputs were very similar on repeated runs, suggesting a high degree of practical consistency; nonetheless, non-determinism motivates validation frameworks (multiple runs or human review).
- Tabulated predicted probabilities (as reported):
  - Topic = 1, Analytical tools: 0.843*** (0.00962)
  - Topic = 2, Emerging issues: 0.792*** (0.0114)
  - Topic = 3, FSAP integration: 0.841*** (0.0145)
  - Topic = 4, General information: 0.969*** (0.0111)
  - Topic = 5, MacFin Integration: 0.0722*** (0.00806)
  - Topic = 6, Policies: 0.756*** (0.012)
  - Topic = 7, Sectoral Vulnerabitlies: 0.748*** (0.014)
  - Topic = 8, Structural Vulnerabilities: 0.620*** (0.0235)
  - Topic = 9, Systemic Risk: 0.734*** (0.024)
  - Topic = 10, Time-varying Vulnerabilities: 0.809*** (0.0179)
  - Type = 1, Factual questions: 0.803*** (0.00701)
  - Type = 2, Open-ended questions: 0.711*** (0.00937)
  - Type = 3, Rating questions: 0.0720*** (0.00805)
  - Complexity = 1, Complex: 0.559*** (0.00765)
  - Complexity = 2, Simple: 0.808*** (0.00673)
  - Observations: 24,206
  - Standard errors shown in parentheses; significance indicated as *** p<0.01, ** p<0.05, * p<0.1.
- Interpretation:
  - LLMs can replicate portions of manual review for structured, fact-based questions and may offer efficiency gains, but are best treated as supportive tools rather than final arbiters.

### II. Apparent Bias and a Tendency Toward Optimism
- Main finding:
  - Across years, country income groups, and model types, LLMs tended to assign higher average ratings for macrofinancial integration than the human economist benchmark.
- Evidence and pattern:
  - Rating-distribution differences show positive bars in the upper-middle range (LLM assigns these scores more frequently) and negative bars at the lower end (LLM less likely to give highly critical assessments).
  - The LLM mimics human pattern of higher ratings for Advanced Economies (AEs) relative to Emerging Markets (EMs) and Low-Income Countries (LICs), but the gap is narrower in the LLM’s assessments (compression of variance).
- Possible sources of optimism:
  - Artifact of training data containing a higher proportion of neutral or positive-toned text.
  - Alignment processes that encourage helpful, non-confrontational responses.
  - LLMs weighting the mere presence of certain keywords or sections without fully evaluating depth or quality.
  - LLM evaluates each document in isolation and lacks an internal mechanism to “grade on a curve.”
- Implications:
  - If used without oversight, LLM optimism could create misleading impressions of report quality and fail to flag weaker macrofinancial analysis.
  - Supports keeping a human in the review loop; treat LLM output as preliminary assessment requiring human validation and contextual interpretation.
  - Prompt refinements and more advanced models improved accuracy but did not fully eliminate the optimistic tendency.

### III. Interpretation of Nuanced Information
- Observed behaviors:
  - No significant instances of outright factual fabrication were observed, but there were cases of misinterpretation in open-ended questions.
  - LLMs sometimes generated justifications referencing text correctly yet not aligning with the contextual intent of the question.
- Specific binary-question examples:
  - Binary-A28 (coverage of financial sector policies not in predefined list): LLM frequently answered "yes" citing fintech, deposit insurance, payment systems, financial inclusion, crypto regulation, or gender in finance. Human economists often answered "no" because they did not consider that wording to qualify as “substantive discussion of other financial policies in the report.”
  - Binary-A15 (coverage of climate-related financial risks): GPT-4o sometimes returned positive if any mention of climate susceptibility existed; newer models distinguish better and can correctly assign zero when climate policies are discussed extensively but not the financial-sector risks. Unmatched cases for newer models tended to be false negatives (LLM answers “No” while economist considered it covered).
- Diagnosis:
  - LLM “brittleness” in highly specialized contexts: can identify mentions but struggle to weigh significance or intended meaning within assessment framework.
  - Justification requirement in prompts is valuable: providing direct quotes and rationales allows human oversight to detect interpretative differences and enables validation.
- Practical conclusion:
  - LLMs are more reliable when required to "show their work"; ambiguity remains in open-ended, context-sensitive judgments, supporting LLMs as assistants rather than substitutes for expert judgment.

### IV. Potential Future Research and Refinements
- Suggested avenues for further work:
  - Further Prompt Refinements:
    - Explore role-based prompting (e.g., instruct LLM to adopt an expert economist role) to better align outputs with human expectations.
  - Calibration and Contextual Benchmarking:
    - Simulate 'grading on a curve' by batch-processing multiple reports to allow relative ranking, or use Retrieval-Augmented Generation (RAG) to provide a dynamic library of 'gold standard' historical reports as comparative references.
  - A Closer Look at Error Types:
    - Create a typology of divergences (i) factual differences, (ii) optimistic rating differences, (iii) different interpretations of open-ended questions, to develop targeted strategies and identify less "LLM-friendly" survey questions.
    - Decompose differences between LLM and economists to assess whether LLMs can provide a useful second opinion.
  - Expanding the Data Sample:
    - Extend assessment period back to 2008 and before to create a continuous panel dataset for regression analysis to understand drivers of LLM ratings (noting human ratings are not available over that earlier period).
- Overarching goal:
  - Improve LLM utility as an analytical assistant via methodological refinements, careful prompt design, model selection, and human–machine collaboration systems.

*Source: IMF working paper — Section V. Discussion of Results and LLM issues*

### Annex I. Additional Tables and Figures

### Annex I. Additional Tables and Figures

### Model performance: GPT-4o (Table 1)
- 2016 — Accuracy 47%, Precision 46%, Recall 100%, F1 score 63%
- 2017 — Accuracy 28%, Precision 27%, Recall 100%, F1 score 43%
- 2018 — Accuracy 28%, Precision 28%, Recall 100%, F1 score 44%
- 2019 — Accuracy 19%, Precision 18%, Recall 92%, F1 score 30%
- 2022 — Accuracy 27%, Precision 25%, Recall 100%, F1 score 40%
- 2023 — Accuracy 53%, Precision 50%, Recall 100%, F1 score 66%
- 2024 — Accuracy 59%, Precision 57%, Recall 100%, F1 score 72%

### Cross-model accuracy in rating questions (Table 2)
- 2016 — GPT-4o 47%, GPT-o1 50%, GPT-4.1 50%, GPT-5 (default effort) 46%
- 2017 — GPT-4o 28%, GPT-o1 46%, GPT-4.1 46%, GPT-5 (default effort) 36%
- 2018 — GPT-4o 28%, GPT-o1 49%, GPT-4.1 49%, GPT-5 (default effort) 38%
- 2019 — GPT-4o 19%, GPT-o1 44%, GPT-4.1 44%, GPT-5 (default effort) 34%
- 2022 — GPT-4o 27%, GPT-o1 56%, GPT-4.1 59%, GPT-5 (default effort) 44%
- 2023 — GPT-4o 53%, GPT-o1 74%, GPT-4.1 74%, GPT-5 (default effort) 61%
- 2024 — GPT-4o 59%, GPT-o1 75%, GPT-4.1 71%, GPT-5 (default effort) 72%

### Model performance: GPT-o1 (Table 3)
- 2016 — Accuracy 50%, Precision 48%, Recall 94%, F1 score 64%
- 2017 — Accuracy 46%, Precision 33%, Recall 94%, F1 score 48%
- 2018 — Accuracy 49%, Precision 36%, Recall 100%, F1 score 53%
- 2019 — Accuracy 44%, Precision 24%, Recall 92%, F1 score 38%
- 2022 — Accuracy 56%, Precision 35%, Recall 100%, F1 score 52%
- 2023 — Accuracy 74%, Precision 65%, Recall 93%, F1 score 77%
- 2024 — Accuracy 75%, Precision 71%, Recall 90%, F1 score 80%

### Model performance: GPT-4.1 (Table 4)
- 2016 — Accuracy 57%, Precision 52%, Recall 88%, F1 score 65%
- 2017 — Accuracy 54%, Precision 35%, Recall 81%, F1 score 49%
- 2018 — Accuracy 63%, Precision 44%, Recall 100%, F1 score 61%
- 2019 — Accuracy 55%, Precision 28%, Recall 92%, F1 score 43%
- 2022 — Accuracy 59%, Precision 37%, Recall 100%, F1 score 54%
- 2023 — Accuracy 74%, Precision 66%, Recall 91%, F1 score 76%
- 2024 — Accuracy 71%, Precision 68%, Recall 88%, F1 score 77%

### Model performance: GPT-5 (medium effort) (Table 5)
- 2016 — Accuracy 46%, Precision 46%, Recall 100%, F1 score 63%
- 2017 — Accuracy 36%, Precision 30%, Recall 100%, F1 score 46%
- 2018 — Accuracy 38%, Precision 32%, Recall 100%, F1 score 48%
- 2019 — Accuracy 34%, Precision 22%, Recall 100%, F1 score 36%
- 2022 — Accuracy 44%, Precision 30%, Recall 100%, F1 score 46%
- 2023 — Accuracy 61%, Precision 55%, Recall 95%, F1 score 69%
- 2024 — Accuracy 72%, Precision 67%, Recall 97%, F1 score 79%

### Exact match rates across GPT models (Table 6)
- 2016 — GPT-4o 6%, GPT-o1 12%, GPT-4.1 12%, GPT-5 0%
- 2017 — GPT-4o 3%, GPT-o1 5%, GPT-4.1 5%, GPT-5 0%
- 2018 — GPT-4o 2%, GPT-o1 2%, GPT-4.1 2%, GPT-5 2%
- 2019 — GPT-4o 2%, GPT-o1 2%, GPT-4.1 2%, GPT-5 0%
- 2022 — GPT-4o 1%, GPT-o1 1%, GPT-4.1 1%, GPT-5 0%
- 2023 — GPT-4o 6%, GPT-o1 4%, GPT-4.1 0%, GPT-5 0%
- 2024 — GPT-4o 5%, GPT-o1 5%, GPT-4.1 0%, GPT-5 0%

### Figures (descriptive listing)
- Figure 1. Differences in Distribution of ratings in GPT-4.1-mini model – 2024
- Figure 2. Near match rate by income group and area of assessment in GPT-O1 Model – 2024
- Figure 3. Exact match rate in binary questions in GPT-O1 Model – 2023 and 2022
- Figure 4. Exact match rate in binary questions in GPT-4.1 Model – 2023 and 2022
- Figure 5a. Match rate in Binary questions, Distribution of Differences, Confidence in GPT-5 Model (medium effort) – 2023 and 2024
- Figure 5b. Match rate in Binary questions, Distribution of Differences, Confidence in GPT-5 Model (high effort) – 2023 and 2024

### Factors associated with LLM average overall ratings (Tables 7 and 8, 2016–2024)
- Selected significant coefficients (GPT-4.1 model, Table 7; robust standard errors in parentheses):
  - Recent FSAP: 0.4650*** (0.058) in column (1)
  - Combined Article IV / Program: -0.2560*** (0.072) in column (1)
  - Macrofinancial review: 0.4499*** (0.064) in column (1)
  - Participation of specialized economists: 0.2494*** (0.063) in column (1); 0.1034*** (0.035) in column (2)
  - Credit-to-GDP, percent: 0.0000*** (0.000) in column (1)
  - Bank Crisis: 0.4034*** (0.062) in column (1)
  - Cumulative Binary values: 0.0815*** (0.004) in column (1); 0.0535*** (0.005) in column (2)
  - Well-articulated view of Systemic Risk: 1.3336*** (0.117) in column (1); 0.6002*** (0.124) in column (2)
  - Income Group = 2, EME: -0.4408*** (0.056) in column (1)
  - Income Group = 3, LIC: -0.8546*** (0.058) in column (1)
- Model fit and sample notes (selected):
  - Observations range across columns (e.g., 543, 539, 494, 456)
  - R-squared values vary widely (e.g., 0.209, 0.031, 0.833, 0.820, 0.644)
  - Country FE: Yes/No depending on column; Year FE: Yes in most columns
  - No. countries: values such as 182, 181, 140, 132, 154
- Selected significant coefficients (GPT-o1 model, Table 8; robust standard errors in parentheses):
  - Recent FSAP: 0.2261*** (0.036) in column (1)
  - Combined Article IV / Program: -0.1460*** (0.051) in column (1)
  - Macrofinancial review: 0.2062*** (0.040) in column (1)
  - Participation of specialized economists: 0.2052*** (0.041) in column (1); 0.0686** (0.028) in column (2)
  - Cumulative Binary values: 0.0435*** (0.003) in column (1); 0.0287*** (0.003) in column (2)
  - Well-articulated view of Systemic Risk: 0.6035*** (0.051) in column (1); 0.2373*** (0.044) in column (2)
  - Income Group = 2, EME: -0.1873*** (0.038) in column (1)
  - Income Group = 3, LIC: -0.3975*** (0.043) in column (1)
  - Observations and R-squared reported similarly to Table 7 (e.g., Observations 542, 538; R-squared 0.121, 0.033)
- Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1

### Logit regression results for 2024 (Table 9)
- Sample size and clustering:
  - Observations 24,206; No. countries 133; Robust standard errors in parentheses
- Model effects (selected coefficients and significance):
  - Model_id = GPT-4o: -0.0325 (0.041) in column (1) — not significant
  - Model_id = GPT-O1: 0.2595*** (0.046) in column (1)
  - model_id = 4, GPT-5: 0.2693*** (0.035) in column (1)
  - Topic = 2, Emerging: -0.3475*** (0.090) in column (4)
  - Topic = 4, General information: 1.7677*** (0.386) in column (4)
  - Topic = 5, MacFin Integration: -4.2674*** (0.147) in column (4)
  - Type = 2, Open-ended: -0.5083*** (0.053) in column (3)
  - Type = 3, Rating: -3.9929*** (0.124) in column (3)
  - Complexity = 2, Simple: 1.2079*** (0.043) in column (1)
- Interpretation: GPT-O1 and GPT-5 (specified model_id entries) show positive and statistically significant associations with the probability of a match relative to the omitted baseline; question Type and Topic categories show large negative or positive associations depending on category.

### Logit margin results: predicted probabilities of a match (Table 10)
- Topic-specific predicted probabilities (standard errors in parentheses):
  - Topic = 1, Analytical tools: 0.843*** (0.00963)
  - Topic = 2, Emerging issues: 0.791*** (0.0114)
  - Topic = 3, FSAP integration: 0.841*** (0.0144)
  - Topic = 4, General information: 0.969*** (0.0112)
  - Topic = 5, MacFin Integration: 0.0717*** (0.00801)
  - Topic = 6, Policies: 0.757*** (0.0119)
  - Topic = 7, Sectoral Vulnerabilities: 0.747*** (0.014)
  - Topic = 8, Structural Vulnerabilities: 0.620*** (0.0236)
  - Topic = 9, Systemic Risk: 0.734*** (0.024)
  - Topic = 10, Time-varying Vulnerabilities: 0.809*** (0.0179)
- Type-specific predicted probabilities:
  - Type = 1, Factual questions: 0.803*** (0.00701)
  - Type = 2, Open-ended questions: 0.711*** (0.00935)
  - Type = 3, Rating questions: 0.0716*** (0.00801)
- Complexity:
  - Complexity = 1, Complex: 0.559*** (0.00765)
  - Complexity = 2, Simple: 0.808*** (0.00675)
- Observations 24,206. Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1

*Annex I. Additional Tables and Figures.*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2026/english/wpiea2026035-source-pdf.pdf_