## IMF Working Paper: Enhancing IMF Economics Training: AI Powered Analysis of Qualitative Learner Feedback

## Source details

**Canonical URL:** [IMF Working Paper: Enhancing IMF Economics Training: AI Powered Analysis of Qualitative Learner Feedback](https://www.imf.org/-/media/files/publications/wp/2024/english/wpiea2024166-print-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2024/english/wpiea2024166-print-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2024/english/wpiea2024166-print-pdf.pdf.json)

---

### 1. Introduction — IMF training and online expansion
- Since 1981, IMF economists have taught over 7,000 courses worldwide and trained over 200,000 country officials in face-to-face and live virtual classrooms.
- The IMF Online Learning (OL) Program began in 2013 and has expanded exponentially.
- As of May 2024:
  - the IMF offered over 100 Massive Open Online Courses (MOOCs) in six languages on the edX platform;
  - the program has registered more than 210,000 active participants;
  - more than one-third of participating government officials are from Sub-Saharan Africa.

### 1. Introduction — motivation and workflow
- Motivation for automated analysis:
  - Open-ended qualitative feedback is collected alongside quantitative survey data to identify reception and improvement areas.
  - Manual review does not scale as course offerings and participant numbers grow.
  - The paper combines a pre-trained Large Language Model (LLM) with human-labelled text responses in a supervised classification task to process open-ended survey responses.
- Manual labelling and supervised learning workflow:
  - Learning experts created a framework and manually labelled thousands of data points.
  - A pre-trained encoder-only LLM (BERT) was appended with a simple classification head; the classifier uses sequence embeddings to predict expert-defined categories.
  - Training procedure:
    - For each survey question a separate model was trained with a random 90 percent training / 10 percent testing split.
    - Training ran for a maximum of 30 epochs with early stopping based on the best F1 score within the first 5 epochs.
    - Mini-batch size was 16 and the Adam optimizer was used with a decreasing learning rate schedule.
  - Preprocessing steps prior to fine-tuning:
    - Drop entries indicating empty responses and nonsensical inputs containing only one unique character (0.4 percent of non-empty responses).
    - For responses with more than one code (<5 percent of observations), only the first code was retained.
    - Remove characters that are not words, white spaces, or digits; convert all responses to lowercase.
  - Model implementation specifics:
    - The BertForSequenceClassification implementation from the Hugging Face Transformers library with “bert-large-uncased” was used.

### 1. Introduction — efficiency and operational gains
- Manual coding workload examples:
  - Six learning experts reviewed comments from six courses (totaling 185 surveys and over 700 individual comments) to build themes.
  - A group of learning experts spent an estimated 90 hours to classify over 15,000 comments from 89 courses between June 2022 and Feb 2023.
  - About 4 percent of comments went through a group audit for consensus; less than 5 percent of comments received more than one label in practice.
- Training data sizes after preprocessing (Table 3):
  - Strengths: 3,517
  - Weaknesses: 3,794
  - Missing Topics: 1,509
  - Application: 3,661
  - Other Comments: 1,370
  - Total: 13,851
- Operational speed-up:
  - The process that previously required about 30 hours of manual effort every trimester is executed in 5 minutes using the BERT classifier and a simple front end (user interface outputs classified responses and associated confidence scores).

### 1. Introduction — survey context
- Post-course (Level 1) survey:
  - Administered using Cvent’s survey functionality and includes closed-ended and open-ended questions (Appendix I).
  - Survey response rate is over 20 percent, yielding responses from over 7,000 participants per trimester in six languages.
  - Language composition of text responses: most in English; French 10 percent; Spanish 6 percent; Portuguese, Arabic, and Russian about 1 percent each.
  - Most open-ended responses are short with an average length of 13 words.
- Illustrative insights from automation:
  - Shorter and modular learning content better caters to IMF online learners, with flexibility particularly appreciated by female participants.
  - Qualitative responses provide evidence that adapting courses to non-English languages increases efficiency and reach of training.

### 5. Model Performance — predictive accuracy and benchmarks
- Purpose: Evaluate classifier accuracy and compare to benchmarks (Mode baseline, Keyword baseline, Human reliability).
- Metrics used: accuracy, precision, recall, and F1 score (weighted averages according to class size).
- Benchmarks defined:
  - Mode baseline: predicts the most frequent class from training data.
  - Keyword baseline: matches expert-provided keywords after stem processing; evaluated both by share containing any keyword and by highest number of matching keywords.
  - Human reliability: inter-coder F1 measured between pairs of human experts on 100 responses for Strengths and 100 for Weaknesses.
- Key model performance results (test sample sizes shown in parentheses):
  - Strengths (n = 389)
    - BERT: Accuracy 0.905; F1 0.823; Precision 0.827; Recall 0.844
    - Mode: Accuracy 0.781; F1 0.686; Precision 0.611; Recall 0.781
    - Keyword: F1 0.189
    - Human reliability: F1 0.822
  - Weaknesses (n = 422)
    - BERT: Accuracy 0.903; F1 0.775; Precision 0.774; Recall 0.795
    - Mode: Accuracy 0.359; F1 0.189; Precision 0.129; Recall 0.359
    - Keyword: F1 0.223
    - Human reliability: F1 0.762
  - Missing Topics (n = 168)
    - BERT: Accuracy 0.929; F1 0.811; Precision 0.820; Recall 0.805
    - Mode: Accuracy 0.604; F1 0.455; Precision 0.365; Recall 0.604
    - Keyword: F1 0.215
    - Human reliability: -
  - Application (n = 407)
    - BERT: Accuracy 0.811; F1 0.652; Precision 0.658; Recall 0.657
    - Mode: Accuracy 0.442; F1 0.271; Precision 0.195; Recall 0.442
    - Keyword: F1 0.095
    - Human reliability: -
  - Other Comments (n = 153)
    - BERT: Accuracy 0.948; F1 0.824; Precision 0.827; Recall 0.846
    - Mode: Accuracy 0.537; F1 0.375; Precision 0.288; Recall 0.537
    - Keyword: F1 0.328
    - Human reliability: -
- Keyword match share (share of responses that contain at least one relevant keyword):
  - Strengths 32.7 (percent)
  - Weaknesses 28.8 (percent)
  - Missing Topics 14.4 (percent)
  - Application 16.6 (percent)
  - Other Comments 29.7 (percent)
- Main findings:
  - The finetuned BERT model substantially outperforms Mode and Keyword baselines on accuracy and F1 across all question types.
  - The model’s agreement with collective human coding is indistinguishable from inter-coder agreement between two independent human coders (example: Strengths — BERT F1 0.823 vs. Human reliability F1 0.822).
  - Keyword-based methods produce substantially lower match rates and F1 scores.

### 5. Model Performance — accuracy across languages
- Training data language composition: English majority, Spanish 7 percent, French 5 percent (Portuguese responses dropped below 0.2 percent).
- Rationale: Multilingual pre-training enables embeddings learned from English to transfer to other languages.
- Per-question accuracy by language (training / test counts shown):
  - Strengths
    - Accuracy (training): En 0.91; Fr 0.92; Sp 0.81; All 0.90
    - Accuracy (test): En 0.92; Fr 0.75; Sp 0.79; All 0.91
    - # of responses (training): En 3,056; Fr 187; Sp 264; All 3,507
    - # of responses (test): En 342; Fr 16; Sp 29; All 387
    - Pearson’s chi-squared p-value: 0.01
  - Weaknesses
    - Accuracy (training): En 0.88; Fr 0.81; Sp 0.83; All 0.87
    - Accuracy (test): En 0.91; Fr 0.90; Sp 0.78; All 0.90
    - # of responses (training): En 3,344; Fr 180; Sp 259; All 3,783
    - # of responses (test): En 379; Fr 20; Sp 23; All 422
    - Pearson’s chi-squared p-value: 0.13
  - Missing Topics
    - Accuracy (training): En 0.90; Fr 0.92; Sp 0.93; All 0.91
    - Accuracy (test): En 0.93; Fr 0.92; Sp 1.00; All 0.93
    - # of responses (training): En 1,279; Fr 107; Sp 121; All 1,507
    - # of responses (test): En 147; Fr 12; Sp 9; All 168
    - Pearson’s chi-squared p-value: 0.69
  - Application
    - Accuracy (training): En 0.81; Fr 0.84; Sp 0.80; All 0.81
    - Accuracy (test): En 0.81; Fr 0.83; Sp 0.86; All 0.81
    - # of responses (training): En 3,215; Fr 180; Sp 256; All 3,651
    - # of responses (test): En 360; Fr 18; Sp 29; All 407
    - Pearson’s chi-squared p-value: 0.73
  - Other Comments
    - Accuracy (training): En 0.94; Fr 0.89; Sp 0.96; All 0.94
    - Accuracy (test): En 0.93; Fr 1.00; Sp 1.00; All 0.95
    - # of responses (training): En 1,161; Fr 85; Sp 121; All 1,367
    - # of responses (test): En 117; Fr 18; Sp 17; All 152
    - Pearson’s chi-squared p-value: 0.28
  - All questions pooled
    - Accuracy (training): En 0.88; Fr 0.87; Sp 0.85; All 0.87
    - Accuracy (test): En 0.89; Fr 0.88; Sp 0.86; All 0.89
    - # of responses (training): En 12,055; Fr 739; Sp 1,021; All 13,815
    - # of responses (test): En 1,345; Fr 84; Sp 107; All 1,536
    - Pearson’s chi-squared p-value: 0.64
- Main findings:
  - For most questions, variation in model performance across languages is not statistically significant.
  - Exception: Strengths category where French and Spanish classification in the test sample is less successful than English (p=0.01).
  - Pooling all survey questions yields nearly identical predictive accuracy across languages (p=0.64).

### 5. Model Performance — uncertainty, calibration, and human review prioritization
- Motivation: Use model confidence to guide selective human review and assess uncertainty for downstream analysis.
- Empirical finding: Trained classifiers tend to be “underconfident” for the predicted (top) class — estimated probability curves lie above the 45-degree line.
- Margin analysis (distance between top two classes) provides additional signal:
  - Logistic regression results (dependent variable = Prob(Correct Class)):
    - Confidence (SoftMax score) coefficients (standard errors):
      - Strengths: 6.61*** (0.32)
      - Weaknesses: 6.64*** (0.27)
      - Missing Topics: 7.32*** (0.64)
      - Application: 7.05*** (0.34)
      - Other Comments: 8.85*** (0.92)
    - Margin (normalized difference between highest and second highest SoftMax scores) coefficients (standard errors):
      - Strengths: 0.94*** (0.29)
      - Weaknesses: 2.45*** (0.32)
      - Missing Topics: 0.64 (0.42)
      - Application: 0.56*** (0.20)
      - Other Comments: 0.06 (0.73)
    - Constants:
      - Strengths: -3.65*** (0.25)
      - Weaknesses: -4.67*** (0.29)
      - Missing Topics: -3.33*** (0.38)
      - Application: -3.54*** (0.23)
      - Other Comments: -5.34*** (0.71)
    - Note: Standard errors in parentheses. ***p < 0.01, **p < 0.05, *p < 0.1.
  - Interpretation:
    - Margin coefficient is consistently positive and highly significant in three out of five question types (Strengths, Weaknesses, Application), indicating larger margins are associated with higher correctness probability.
    - Significance of margin implies imperfect calibration: margin carries information beyond SoftMax confidence.
- Calibration approach and results:
  - Platt-scaling (logistic mapping) applied to SoftMax scores to produce calibrated probabilities.
  - Average calibrated probabilities (after Platt-scaling) for most likely class:
    - Correct Predictions:
      - Strengths 0.92
      - Weaknesses 0.90
      - Missing Topics 0.93
      - Application 0.84
      - Other Comments 0.95
      - All 0.90
    - Incorrect Predictions:
      - Strengths 0.72
      - Weaknesses 0.68
      - Missing Topics 0.75
      - Application 0.71
      - Other Comments 0.81
      - All 0.71
    - Differences (Correct − Incorrect):
      - Strengths 0.20
      - Weaknesses 0.22
      - Missing Topics 0.18
      - Application 0.13
      - Other Comments 0.14
      - All 0.19
  - Main finding: Calibrated probabilities are, on average, substantially higher for correct predictions (average 0.90) than for incorrect predictions (average 0.71).
  - Practical implication: Use calibrated probabilities and margin measures to prioritize human review for low-confidence or low-margin predictions.

### 6.2 Self-Reported Learning Time and Language Barriers — summary findings
- Relationship between developer time estimates and self-reported times:
  - Indicative time requirements estimated by course developers for a “representative” learner are positively related to average self-reported learning times.
  - Numerical self-reported times are imputed using the top of the time brackets in the survey, using 8 hours for the uncapped 6+ hours bracket.
- Variability in completion time:
  - Actual time varies significantly across participants depending on educational background, perceived usefulness, and language barriers.
- Evidence on language constraints:
  - Participants who spend more time completing a course tend to flag language issues more frequently in their written comments; pattern more pronounced for courses delivered in English.
  - Participants citing language constraints are more likely to come from Sub-Saharan Africa and the Western Hemisphere.
- Table 10 — Critical comments about language issues (percent of respondents)
  - All languages: 1.92 3.07 3.71 4.34 4.38 1.88 1.66 2.18 3.76 3.18
  - English: 2.85 4.26 4.68 6.03 5.38 3.00 2.27 3.74 5.96 4.30
  - Note: Table shows share of answers indicating ‘language issues’ as a major weakness among respondents that identified any weakness; data broken down by self-reported study time and region. Regions follow IMF Area Departments: Sub-Saharan Africa (AFR), Asia and Pacific (APD), Europe (EUR), Middle East and Central Asia (MCD), and Western Hemisphere (WHD).
- Regional differences in intent to apply learning:
  - Shares of respondents reporting they intend to apply new knowledge and skills in their jobs:
    - Africa: 51 percent
    - Latin America: 56 percent
    - Europe: 35 percent
  - Implication: Importance of offering IMF economics and finance training in languages other than English to reach more learners.
- Course design implications:
  - Learner feedback indicates preferences for shorter learning content overall, with disutility associated with lengthy courses varying significantly by gender.
  - Language barriers are a critical consideration for accessibility and applicability.
- Methodological context:
  - The IMF OL Program collects feedback from every active MOOC participant; over 7,000 responses every trimester.
  - After fine-tuning BERT on labelled data, classification of qualitative feedback was reduced to minutes with model accuracy on par with human annotators.
  - Classifier trained mostly on English delivered largely consistent performance on other languages, indicating effectiveness even when task-specific training data for certain languages is sparse.

*Source: wpiea2024166-print-pdf — IMF Working Paper — selected chapters and sections.*

### 1. Introduction ........................................................................................................

### 1. Introduction

### Overview of IMF training and online expansion
- Since 1981, IMF economists have taught over 7,000 courses worldwide and trained over 200,000 country officials in face-to-face and live virtual classrooms.
- The IMF Online Learning (OL) Program began in 2013 and has expanded exponentially.
- As of May 2024:
  - the IMF offered over 100 Massive Open Online Courses (MOOCs) in six languages on the edX platform;
  - the program has registered more than 210,000 active participants;
  - more than one-third of participating government officials are from Sub-Saharan Africa.

### Motivation for analyzing qualitative learner feedback
- The IMF learning team collects participant feedback (quantitative and open-ended qualitative responses) to understand reception and areas for improvement.
- Open-ended responses present scaling challenges for manual review as course offerings and participant numbers grow.
- This paper describes combining a pre-trained Large Language Model (LLM) with human-labelled text responses in a supervised classification task to process open-ended survey responses efficiently.

### Manual labelling and supervised learning workflow
- Learning experts created a framework to classify text responses and manually labelled thousands of data points.
- A pre-trained encoder-only LLM (BERT) was appended with a simple classification head; the classifier uses sequence embeddings to predict expert-defined categories.
- Training procedure highlights:
  - For each survey question a separate model was trained with a random 90 percent training / 10 percent testing split.
  - Training ran for a maximum of 30 epochs with early stopping based on the best F1 score within the first 5 epochs.
  - Mini-batch size was 16 and the Adam optimizer was used with a decreasing learning rate schedule.

### Key performance and calibration findings
- The model outperforms several simple benchmarks (e.g., carefully selected keyword searches).
- The model’s agreement with human coders closely matches human-to-human agreement when coding the same responses under the same guidelines.
- Pre-training on a multi-lingual corpus enables similar performance across languages despite considerably less non-English labelled training data.
- Neural network classifiers are not well-suited for direct probabilistic interpretation of raw outputs; the classifier tended to be underconfident in its final predicted class.
- A post-processing calibration step was applied to map raw confidence scores to the probability that the prediction is correct, enabling informed decision rules for human review.

### Illustrative benefits and insights
- Two examples of value from automated qualitative analysis:
  - Shorter and modular learning content better caters to IMF online learners, with flexibility particularly appreciated by female participants.
  - Qualitative responses provide evidence that adapting courses to non-English languages increases efficiency and reach of training.
- Automation enables large-scale analysis of learner feedback that would be impractical via manual annotation alone.

### Efficiency gains and implementation details
- Manual coding example and workload:
  - Six learning experts reviewed comments from six courses (totaling 185 surveys and over 700 individual comments) to build themes.
  - A group of learning experts spent an estimated 90 hours to classify over 15,000 comments from 89 courses between June 2022 and Feb 2023.
  - About 4 percent of comments went through a group audit for consensus; less than 5 percent of comments received more than one label in practice.
- Preprocessing steps prior to fine-tuning:
  - Drop entries indicating empty responses and nonsensical inputs containing only one unique character (0.4 percent of non-empty responses).
  - For responses with more than one code (<5 percent of observations), only the first code was retained.
  - Remove characters that are not words, white spaces, or digits; convert all responses to lowercase.
- Model implementation specifics:
  - The BertForSequenceClassification implementation from the Hugging Face Transformers library with “bert-large-uncased” was used.
- Training data sizes after preprocessing (Table 3):
  - Strengths: 3,517
  - Weaknesses: 3,794
  - Missing Topics: 1,509
  - Application: 3,661
  - Other Comments: 1,370
  - Total: 13,851
- Operational gains:
  - The process that previously required about 30 hours of manual effort every trimester is executed in 5 minutes using the BERT classifier and a simple front end (user interface outputs classified responses and associated confidence scores).

### Survey and qualitative data context
- Post-course (Level 1) survey:
  - Administered using Cvent’s survey functionality and includes closed-ended and open-ended questions (Appendix I).
  - Survey response rate is over 20 percent, yielding responses from over 7,000 participants per trimester in six languages.
  - Language composition of text responses: most in English; French 10 percent; Spanish 6 percent; Portuguese, Arabic, and Russian about 1 percent each.
  - Most open-ended responses are short with an average length of 13 words.

*IMF Working Paper: Enhancing IMF Economics Training: AI Powered Analysis of Qualitative Learner Feedback*

### 5. Model Performance

### 5. Model Performance

### Predictive Accuracy
- Purpose: Evaluate the text classifier’s final prediction accuracy and compare it to benchmarks (Mode baseline, Keyword baseline, Human reliability).
- Metrics used: accuracy (share of correct matches), precision (Type I error), recall (Type II error), and F1 score (harmonic mean of precision and recall). All metrics are weighted averages according to class size in the training data.
- Benchmarks:
  - Mode baseline: predicts the most frequent class from the training data for all test observations (example: for the Strengths question, 78 percent of responses in the training data were labelled Material).
  - Keyword baseline: matches expert-provided keywords after stem processing; two evaluation approaches described (share of responses containing any relevant keyword; label assigned by highest number of matching keywords).
  - Human reliability: inter-coder agreement measured by average F1 score between pairs of human experts who independently coded 100 responses for Strengths and 100 responses for Weaknesses.
- Key model performance results (test sample sizes shown in parentheses):
  - Strengths (n = 389)
    - BERT: Accuracy 0.905; F1 0.823; Precision 0.827; Recall 0.844
    - Mode: Accuracy 0.781; F1 0.686; Precision 0.611; Recall 0.781
    - Keyword: F1 0.189
    - Human reliability: F1 0.822
  - Weaknesses (n = 422)
    - BERT: Accuracy 0.903; F1 0.775; Precision 0.774; Recall 0.795
    - Mode: Accuracy 0.359; F1 0.189; Precision 0.129; Recall 0.359
    - Keyword: F1 0.223
    - Human reliability: F1 0.762
  - Missing Topics (n = 168)
    - BERT: Accuracy 0.929; F1 0.811; Precision 0.820; Recall 0.805
    - Mode: Accuracy 0.604; F1 0.455; Precision 0.365; Recall 0.604
    - Keyword: F1 0.215
    - Human reliability: -
  - Application (n = 407)
    - BERT: Accuracy 0.811; F1 0.652; Precision 0.658; Recall 0.657
    - Mode: Accuracy 0.442; F1 0.271; Precision 0.195; Recall 0.442
    - Keyword: F1 0.095
    - Human reliability: -
  - Other Comments (n = 153)
    - BERT: Accuracy 0.948; F1 0.824; Precision 0.827; Recall 0.846
    - Mode: Accuracy 0.537; F1 0.375; Precision 0.288; Recall 0.537
    - Keyword: F1 0.328
    - Human reliability: -
- Keyword match share (share of responses that contain at least one relevant keyword):
  - Strengths 32.7 (percent)
  - Weaknesses 28.8 (percent)
  - Missing Topics 14.4 (percent)
  - Application 16.6 (percent)
  - Other Comments 29.7 (percent)
- Main findings:
  - The finetuned BERT model substantially outperforms Mode and Keyword baselines on accuracy and F1 across all question types.
  - The model’s agreement with collective human coding is indistinguishable from inter-coder agreement between two independent human coders (example: Strengths — BERT F1 0.823 vs. Human reliability F1 0.822).
  - Keyword-based methods produce substantially lower match rates and F1 scores, indicating the BERT model learns more complex patterns than simple word matching.

### Model Accuracy Across Languages
- Training data language composition: English majority, Spanish 7 percent, French 5 percent (Portuguese responses dropped below 0.2 percent).
- Rationale: Multilingual pre-training of the underlying LLM produces embeddings that make mappings learned from English potentially transferable to other languages.
- Per-question accuracy by language (Training / Test sample counts shown):
  - Strengths
    - Accuracy (training): En 0.91; Fr 0.92; Sp 0.81; All 0.90
    - Accuracy (test): En 0.92; Fr 0.75; Sp 0.79; All 0.91
    - # of responses (training): En 3,056; Fr 187; Sp 264; All 3,507
    - # of responses (test): En 342; Fr 16; Sp 29; All 387
    - Pearson’s chi-squared p-value: 0.01 (significant difference across languages for Strengths)
  - Weaknesses
    - Accuracy (training): En 0.88; Fr 0.81; Sp 0.83; All 0.87
    - Accuracy (test): En 0.91; Fr 0.90; Sp 0.78; All 0.90
    - # of responses (training): En 3,344; Fr 180; Sp 259; All 3,783
    - # of responses (test): En 379; Fr 20; Sp 23; All 422
    - Pearson’s chi-squared p-value: 0.13
  - Missing Topics
    - Accuracy (training): En 0.90; Fr 0.92; Sp 0.93; All 0.91
    - Accuracy (test): En 0.93; Fr 0.92; Sp 1.00; All 0.93
    - # of responses (training): En 1,279; Fr 107; Sp 121; All 1,507
    - # of responses (test): En 147; Fr 12; Sp 9; All 168
    - Pearson’s chi-squared p-value: 0.69
  - Application
    - Accuracy (training): En 0.81; Fr 0.84; Sp 0.80; All 0.81
    - Accuracy (test): En 0.81; Fr 0.83; Sp 0.86; All 0.81
    - # of responses (training): En 3,215; Fr 180; Sp 256; All 3,651
    - # of responses (test): En 360; Fr 18; Sp 29; All 407
    - Pearson’s chi-squared p-value: 0.73
  - Other Comments
    - Accuracy (training): En 0.94; Fr 0.89; Sp 0.96; All 0.94
    - Accuracy (test): En 0.93; Fr 1.00; Sp 1.00; All 0.95
    - # of responses (training): En 1,161; Fr 85; Sp 121; All 1,367
    - # of responses (test): En 117; Fr 18; Sp 17; All 152
    - Pearson’s chi-squared p-value: 0.28
  - All questions pooled
    - Accuracy (training): En 0.88; Fr 0.87; Sp 0.85; All 0.87
    - Accuracy (test): En 0.89; Fr 0.88; Sp 0.86; All 0.89
    - # of responses (training): En 12,055; Fr 739; Sp 1,021; All 13,815
    - # of responses (test): En 1,345; Fr 84; Sp 107; All 1,536
    - Pearson’s chi-squared p-value: 0.64 (no significant difference pooled)
- Main findings:
  - For most survey questions, variation in model performance across languages is not statistically significant.
  - The notable exception is the Strengths category where French and Spanish classification in the test sample is less successful than English (p=0.01).
  - Pooling all survey questions yields nearly identical predictive accuracy across languages (p=0.64).

### Uncertainty Around Model Predictions
- Motivation: Use model confidence to guide selective human review and to assess uncertainty for downstream empirical analysis or decision-making.
- Calibration concept: SoftMax output is not necessarily interpretable as true probability; calibration seeks to align predicted confidence with empirical correctness frequency.
- Empirical finding: Trained classifiers tend to be “underconfident” for the predicted (top) class — estimated probability curves (logit and linear probability models) lie above the 45-degree line for most question types, indicating actual correctness probabilities exceed reported SoftMax scores.
- Margin analysis (distance between top two classes) provides additional signal on prediction distinctiveness:
  - Logistic regression results (dependent variable = Prob(Correct Class)):
    - Confidence (SoftMax score) coefficients (standard errors):
      - Strengths: 6.61*** (0.32)
      - Weaknesses: 6.64*** (0.27)
      - Missing Topics: 7.32*** (0.64)
      - Application: 7.05*** (0.34)
      - Other Comments: 8.85*** (0.92)
    - Margin (normalized difference between highest and second highest SoftMax scores) coefficients (standard errors):
      - Strengths: 0.94*** (0.29)
      - Weaknesses: 2.45*** (0.32)
      - Missing Topics: 0.64 (0.42)
      - Application: 0.56*** (0.20)
      - Other Comments: 0.06 (0.73)
    - Constants:
      - Strengths: -3.65*** (0.25)
      - Weaknesses: -4.67*** (0.29)
      - Missing Topics: -3.33*** (0.38)
      - Application: -3.54*** (0.23)
      - Other Comments: -5.34*** (0.71)
    - Note: Standard errors in parentheses. ***p < 0.01, **p < 0.05, *p < 0.1.
  - Interpretation:
    - The estimated coefficient on the margin is consistently positive and highly significant in three out of five question types (Strengths, Weaknesses, Application), indicating a more distinctive prediction (larger margin) is more likely to be correct.
    - In a perfectly calibrated model, the margin would not carry extra information beyond the model’s confidence; the observed significance indicates imperfect calibration.
- Reliability diagrams and calibration:
  - Reliability diagrams (decile bins of predicted SoftMax scores) show dots above the 45-degree line (under-confidence).
  - Platt-scaling (logistic mapping of raw scores to calibrated probabilities) aligns reliability diagrams more closely to the 45-degree line, improving calibration.

### Calibration Results and Use of Calibrated Probabilities
- Post-processing: Applied Platt-scaling to transform SoftMax scores into calibrated probabilities.
- Average calibrated probabilities (after Platt-scaling) for most likely class:
  - Correct Predictions:
    - Strengths 0.92
    - Weaknesses 0.90
    - Missing Topics 0.93
    - Application 0.84
    - Other Comments 0.95
    - All 0.90
  - Incorrect Predictions:
    - Strengths 0.72
    - Weaknesses 0.68
    - Missing Topics 0.75
    - Application 0.71
    - Other Comments 0.81
    - All 0.71
  - Differences (Correct − Incorrect):
    - Strengths 0.20
    - Weaknesses 0.22
    - Missing Topics 0.18
    - Application 0.13
    - Other Comments 0.14
    - All 0.19
- Main findings:
  - Calibrated probabilities are, on average, substantially higher for correct predictions (average 0.90) than for incorrect predictions (average 0.71) across question types.
  - This provides reassurance that the models do not systematically misclassify responses with high calibrated confidence.
- Practical implication:
  - Use calibrated probabilities and margin measures to prioritize human review for low-confidence or low-margin predictions, enabling efficient selective human intervention and better handling of edge cases.

*Source: IMF Working Paper — 5. Model Performance (chapter content).*

### 6.2 Self-Reported Learning Time and Language Barriers

### 6.2 Self-Reported Learning Time and Language Barriers

### Relationship between indicative (developer) time estimates and self-reported learning time
- The indicative time requirements estimated by course developers for a “representative” learner are positively related to the average self-reported learning times across survey respondents (Figure 5).
- Numerical self-reported times are imputed using the top of the time brackets in the survey, using 8 hours for the uncapped 6+ hours bracket.

### Variability in individual completion time
- Actual time each participant spends completing a course can vary significantly based on individual factors such as:
  - educational background
  - perceived usefulness of the course
  - language barriers

### Evidence on language as a constraining factor
- Participants who spend more time completing a course tend to flag language issues more frequently in their written comments; this pattern is more pronounced for courses delivered in English.
- Participants citing language constraints are more likely to come from Sub-Saharan Africa and the Western Hemisphere, where dominant regional languages facilitate communication.

### Table 10 — Critical comments about language issues (percent of respondents)
- All languages: 1.92 3.07 3.71 4.34 4.38 1.88 1.66 2.18 3.76 3.18
- English: 2.85 4.26 4.68 6.03 5.38 3.00 2.27 3.74 5.96 4.30
- Note: The table shows the share of answers indicating ‘language issues’ as a major weakness of the course, among respondents that identified any weakness. The data are broken down by self-reported study time and region. Regions follow the IMF’s organizational structure of Area Departments: Sub-Saharan Africa (AFR), Asia and Pacific (APD), Europe (EUR), Middle East and Central Asia (MCD), and Western Hemisphere (WHD).

### Regional differences in intent to apply learning and language implications
- Shares of respondents reporting they intend to apply new knowledge and skills in their jobs (and expecting it to contribute to institutional mandates and career prospects):
  - Africa: 51 percent
  - Latin America: 56 percent
  - Europe: 35 percent
- These differences underscore the importance of offering IMF economics and finance training in languages other than English to reach more learners across IMF membership.

### Implications for course design and tailoring
- Qualitative learner feedback indicates:
  - Preferences for shorter learning content overall, with disutility associated with lengthy courses varying significantly by gender.
  - Language barriers are an important consideration for accessibility and applicability of course content.
- The evaluation framework and AI-assisted approach facilitate detailed analysis of learner perceptions and their relation to learning outcomes across dimensions such as age, educational and employment background, and work experience.

### Methodological and operational notes from the wider paper (relevant context)
- The IMF Online Learning Program collects feedback from every active MOOC participant; a large portion of the survey comprises open-ended questions, resulting in over 7,000 responses every trimester.
- After fine-tuning a pre-trained open-source LLM (BERT) on labelled data, the classification process of qualitative feedback was reduced to minutes, with model accuracy on par with human annotators.
- The classifier trained mostly on English responses delivered largely consistent performance on other languages, suggesting fine-tuned pre-trained language models can be effective in multilingual use cases even when task-specific training data for certain languages is sparse.

*Source: wpiea2024166-print-pdf — 6.2 Self-Reported Learning Time and Language Barriers.*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2024/english/wpiea2024166-print-pdf.pdf_