{ "title": "Enhancing IMF Economics Training: AI-Powered Analysis of Qualitative Learner Feedback", "publication": "IMF Working Papers, August 2, 2024", "sourceUrl": "https://www.imf.org/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926", "canonical": "https://www.imf.org/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926", "overlayPath": "/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926/index.md", "summary": "The International Monetary Fund (IMF) has expanded its online learning program, offering over 100 Massive Open Online Courses (MOOCs) to support economic and financial policymaking worldwide.", "sections": [ { "heading": "Overview", "content": "- The International Monetary Fund (IMF) has expanded its online learning program, offering over 100 Massive Open Online Courses (MOOCs) to support economic and financial policymaking worldwide.\n- The paper explores the application of Artificial Intelligence (AI), specifically Large Language Models (LLMs), to analyze qualitative feedback from participants in these courses.\n- Aim: Use AI to classify open-ended survey responses efficiently and inform continuous improvement of IMF online courses to align with capacity development goals." }, { "heading": "Methods and Model Development", "content": "- Fine-tuned a pre-trained LLM on expert-annotated text data to develop classification models for open-ended learner feedback.\n- Evaluation compares model classification accuracy against human coders." }, { "heading": "Performance and Multilingual Robustness", "content": "- Models achieved classification accuracy comparable to human coders.\n- Demonstrated robust performance across multiple languages, including English, French, and Spanish." }, { "heading": "Key Findings from Learner Feedback", "content": "- Preference for shorter, modular content.\n- Variations in content preferences across genders.\n- Significant impact of language barriers on learning outcomes.\n- Unstructured learner feedback yields insights that support continuous course improvement and alignment with IMF capacity development objectives to enhance economic and financial expertise globally." }, { "heading": "Policy Implications and Recommendations", "content": "- Adopt AI-assisted classification of qualitative feedback to scale analysis of learner responses efficiently.\n- Use model outputs to:\n - Inform course design toward shorter, modular formats.\n - Address gender-based variations in learner preferences.\n - Mitigate language barriers to improve learning outcomes across multilingual cohorts.\n- Integrate AI-derived insights into ongoing monitoring and improvement cycles for online training programs to better support global capacity development." }, { "heading": "Publication and Identifiers", "content": "- Authors: Andras Komaromi, Xiaomin Wu, Ran Pan, Yang Liu, Pablo Cisneros, Anchal Manocha, Hiba El Oirghi\n- Date: August 2, 2024\n- Pages: 37\n- Series: Working Paper No. 2024/166\n- Volume: 2024\n- Issue: 166\n- DOI: https://doi.org/10.5089/9798400286124.001\n- ISBN: 9798400286124\n- ISSN: 1018-5941\n- Subjects: Artificial intelligence, Automation, Information systems and platforms, Machine learning, Technology\n- Keywords: application of artificial intelligence, Artificial intelligence, Asia and Pacific, Automation, Europe, IMF Economics training, Information systems and platforms, learner feedback, learner satisfaction, Machine learning, measuring training impact, Middle East and Central Asia, Sub-Saharan Africa, Western Hemisphere\n\nEnhancing IMF Economics Training: AI-Powered Analysis of Qualitative Learner Feedback, IMF Working Paper No. 2024/166 (August 2, 2024).\n\n---\n\n Content in this bundle\n\n- Working Paper\n - Working Paper (Markdown version){rel=\"alternate\" type=\"text/markdown\"}\n - Working Paper (PDF){rel=\"external\" type=\"application/pdf\"}\n\n---\n\nSource: https://www.imf.org/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926" } ], "bullets": [ "[Markdown version](/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926/index.md)", "[Structured JSON version](/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926/index.json)", "[Bundle manifest](/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926/bundle-manifest.json)", "Authors: Andras Komaromi, Xiaomin Wu, Ran Pan, Yang Liu, Pablo Cisneros, Anchal Manocha, Hiba El Oirghi", "Published: August 2, 2024", "Series: IMF Working Papers", "DOI: https://doi.org/10.5089/9798400286124.001", "The International Monetary Fund (IMF) has expanded its online learning program, offering over 100 Massive Open Online Courses (MOOCs) to support economic and financial policymaking worldwide.", "The paper explores the application of Artificial Intelligence (AI), specifically Large Language Models (LLMs), to analyze qualitative feedback from participants in these courses.", "Aim: Use AI to classify open-ended survey responses efficiently and inform continuous improvement of IMF online courses to align with capacity development goals.", "Fine-tuned a pre-trained LLM on expert-annotated text data to develop classification models for open-ended learner feedback.", "Evaluation compares model classification accuracy against human coders.", "Models achieved classification accuracy comparable to human coders.", "Demonstrated robust performance across multiple languages, including English, French, and Spanish.", "Preference for shorter, modular content.", "Variations in content preferences across genders.", "Significant impact of language barriers on learning outcomes.", "Unstructured learner feedback yields insights that support continuous course improvement and alignment with IMF capacity development objectives to enhance economic and financial expertise globally.", "Adopt AI-assisted classification of qualitative feedback to scale analysis of learner responses efficiently.", "Use model outputs to:", "Integrate AI-derived insights into ongoing monitoring and improvement cycles for online training programs to better support global capacity development.", "Authors: Andras Komaromi, Xiaomin Wu, Ran Pan, Yang Liu, Pablo Cisneros, Anchal Manocha, Hiba El Oirghi", "Date: August 2, 2024", "Pages: 37", "Series: Working Paper No. 2024/166", "Volume: 2024", "Issue: 166", "DOI: https://doi.org/10.5089/9798400286124.001", "ISBN: 9798400286124", "ISSN: 1018-5941", "Subjects: Artificial intelligence, Automation, Information systems and platforms, Machine learning, Technology", "Keywords: application of artificial intelligence, Artificial intelligence, Asia and Pacific, Automation, Europe, IMF Economics training, Information systems and platforms, learner feedback, learner satisfaction, Machine learning, measuring training impact, Middle East and Central Asia, Sub-Saharan Africa, Western Hemisphere", "**Working Paper**" ], "related": [ { "title": "Working Paper", "role": "paper", "sourceUrl": "https://www.imf.org/-/media/files/publications/wp/2024/english/wpiea2024166-print-pdf.pdf", "summary": { "path": "/-/media/files/publications/wp/2024/english/wpiea2024166-print-pdf.pdf.md", "mime": "text/markdown" }, "binary": { "path": "/-/media/files/publications/wp/2024/english/wpiea2024166-print-pdf.pdf", "mime": "application/pdf" } } ], "alternates": { "markdown": "/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926/index.md", "json": "/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926/index.json", "bundleManifest": "/en/publications/wp/issues/2024/08/02/enhancing-imf-economics-training-ai-powered-analysis-of-qualitative-learner-feedback-552926/bundle-manifest.json" }, "generatedAtUtc": "2026-09-17T00:27:32.790Z" }