Enhancing IMF Economics Training: AI-Powered Analysis of Qualitative Learner Feedback

IMF Working Papers, August 2, 2024

Source details

Canonical URL: Enhancing IMF Economics Training: AI-Powered Analysis of Qualitative Learner Feedback

Authors: Andras Komaromi, Xiaomin Wu, Ran Pan, Yang Liu, Pablo Cisneros, Anchal Manocha, Hiba El Oirghi
Published: August 2, 2024
Series: IMF Working Papers
DOI: https://doi.org/10.5089/9798400286124.001

The International Monetary Fund (IMF) has expanded its online learning program, offering over 100 Massive Open Online Courses (MOOCs) to support economic and financial policymaking worldwide.
The paper explores the application of Artificial Intelligence (AI), specifically Large Language Models (LLMs), to analyze qualitative feedback from participants in these courses.
Aim: Use AI to classify open-ended survey responses efficiently and inform continuous improvement of IMF online courses to align with capacity development goals.

Fine-tuned a pre-trained LLM on expert-annotated text data to develop classification models for open-ended learner feedback.
Evaluation compares model classification accuracy against human coders.

Models achieved classification accuracy comparable to human coders.
Demonstrated robust performance across multiple languages, including English, French, and Spanish.

Preference for shorter, modular content.
Variations in content preferences across genders.
Significant impact of language barriers on learning outcomes.
Unstructured learner feedback yields insights that support continuous course improvement and alignment with IMF capacity development objectives to enhance economic and financial expertise globally.

Adopt AI-assisted classification of qualitative feedback to scale analysis of learner responses efficiently.
Use model outputs to:
- Inform course design toward shorter, modular formats.
- Address gender-based variations in learner preferences.
- Mitigate language barriers to improve learning outcomes across multilingual cohorts.
Integrate AI-derived insights into ongoing monitoring and improvement cycles for online training programs to better support global capacity development.

Authors: Andras Komaromi, Xiaomin Wu, Ran Pan, Yang Liu, Pablo Cisneros, Anchal Manocha, Hiba El Oirghi
Date: August 2, 2024
Pages: 37
Series: Working Paper No. 2024/166
Volume: 2024
Issue: 166
DOI: https://doi.org/10.5089/9798400286124.001
ISBN: 9798400286124
ISSN: 1018-5941
Subjects: Artificial intelligence, Automation, Information systems and platforms, Machine learning, Technology
Keywords: application of artificial intelligence, Artificial intelligence, Asia and Pacific, Automation, Europe, IMF Economics training, Information systems and platforms, learner feedback, learner satisfaction, Machine learning, measuring training impact, Middle East and Central Asia, Sub-Saharan Africa, Western Hemisphere

Enhancing IMF Economics Training: AI-Powered Analysis of Qualitative Learner Feedback, IMF Working Paper No. 2024/166 (August 2, 2024).