## 1. ICD Training Program: End-of-Course Surveys by Department, FY 2007–13

## Source details

**Canonical URL:** [1. ICD Training Program: End-of-Course Surveys by Department, FY 2007–13](https://www.imf.org/-/media/files/publications/wp/2018/wp18151.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2018/wp18151.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2018/wp18151.pdf.json)

---

### Introduction and Purpose
- The IMF provides training as part of its mandate on capacity building to strengthen analytical and technical capabilities of member country officials to formulate and manage macroeconomic and financial sector policies.
- The ICD-coordinated program of courses is called the ICD Training Program.
- This paper examines evaluation methods and resulting data collected over the period 2006–13.
- Assessment tools used during 2006–13 were subsequently revised over 2015–18 to increase results focus and introduce course-specific content.
- Starting in 2017, the IMF implemented results-based management (RBM) to training.
- ICD has undertaken a curriculum review and course revamp since 2015.
- Objective: distill evaluation information over a seven-year period and combine it with participant demographic information.

### ICD Training Program (as of 2014) — characteristics, volume, and delivery
- Objective: share IMF staff expertise to strengthen officials’ skills to analyze economic developments and formulate/manage macroeconomic and financial sector policies.
- Course characteristics:
  - Practically oriented; integration of theoretical frameworks and real-world applications using case studies.
  - General approach: “hand-on” workshops and real-world policy applications rather than lecturing.
  - Most courses are two weeks long and are taught by three IMF economists (typically two full-time IMF staff and one consultant).
  - Participants are government officials; selection varies by course—application or nomination.
- Provider shares and delivery:
  - ICD courses account for about 55 percent of training.
  - The Statistics Department (STA) provides over 20 percent of training.
  - Other IMF departments and regional technical assistance centers (RTACs) provide the remainder.
- Volume and geographic distribution:
  - Since 2006 the volume of training increased almost 25 percent to over 12,500 participant weeks in FY 2013.
  - Training offered at multiple locations including HQ, RTCs, RTACs, and other venues.
  - As of 2014, there were 4 regional training centers (RTCs): Africa Training Institute; IMF-Middle East Center for Economics and Finance; the Joint Vienna Institute; and the IMF Singapore Regional Training Institute.
  - Three regional training programs: Joint Regional Training Center for Latin America in Brazil; Joint China-IMF Training Program; Joint Partnership for Africa.
  - Growth in training delivered to the Middle East and Central Asia region (MCD) with opening of a center in Kuwait.
  - In FY 2013, levels of training to AFR, MCD, and APD were roughly comparable; adjusting for number of countries, training to AFR stood at just three-quarters of training to MCD and APD.
  - India Training Program in Pune delivered training from FY 2007 to FY 2013.

### Results-Based Management (RBM) for Training — outcome levels and indicators
- RBM assesses two sequential outcome levels:
  - (i) Reaction: participant perceptions measured by a composite satisfaction rating from end-of-course surveys.
  - (ii) Learning: knowledge acquisition measured by increase in average post-test score vs. pre-test and share of participants achieving 60 percent and above on post-course test.
- On-the-job behavior change assessed through follow-up surveys to participants and their direct supervisors on use of course knowledge/skills and course-specific forecasting techniques/tools/models on the job (within three to six months).

### End-of-Course Surveys: Methodology and data coverage
- End-of-course surveys: first tier of evaluation, conducted after each training course based on participants’ perceptions.
- Results routinely presented in back-to-office reports and, since 2014, shared in the IMF knowledge exchange.
- Questionnaire structure (Sections A–D):
  - Section A. Content: topics appropriate; contents met expectations; workshops useful.
  - Section B. Presentational Methods: structure/mix appropriate; lecturers effective; lecturers encouraged discussion; counselors contributed effectively.
  - Section C. Value (recorded in database): Skill use; Recommend to others; Satisfaction.
  - Section D. Additional comments (sometimes course-specific).
- Response scale: 1–5 where 1 = strong disagreement; 2 = disagreement; 3 = neutral; 4 = agreement; 5 = strong agreement.
- Administration and response:
  - Historically paper-based; some electronic collection introduced.
  - Response rate to surveys is close to 100 percent.
- ICD database coverage:
  - Information on 863 courses during FY 2007–13.
  - About two-thirds (542) of the 863 courses have reported end-of-course survey results.
  - Focus of analysis: end-of-course surveys for ICD-delivered courses: 514 courses.
  - Each record contains course description, identification number, location, dates, delivering department, and average rating for each of the three Section C questions.

### Aggregate end-of-course findings (pooled, FY 2007–13; 514 courses)
- Overall average scores:
  - Recommend to others: 4.8
  - Overall satisfaction: 4.7
  - Knowledge/skills learned (Skill use): 4.5
- Ordering and statistical significance:
  - Recommend > Skill: t (512) = 36.07, p <0.05
  - Recommend > Overall satisfaction: t (512) = 22.08, p <0.05
  - Overall satisfaction > Skill: t (512) = 15.51, p <0.05
- Distributional and time variation:
  - Notable variation in distribution; some courses receive low ratings (example: FY 2007 one course rated ~3.4 across questions).
  - Skill use exhibits largest dispersion; recommend to others has lowest dispersion and highest ratings.
  - Little variation over time; no discernible time trend across FY 2007–13.
  - Aggregated yearly averages reported (exact yearly values in source).

### Location and course-type disaggregation (selected averages, FY 2007–13; courses delivered by INS/ICD only)
- Location averages (Number of courses — Skill use; Recommend to others; Overall Satisfaction):
  - BT Brazil: 27 — 4.5; 4.8; 4.7
  - CE Kuwait: 14 — 4.4; 4.8; 4.7
  - CT China: 16 — 4.3; 4.5; 4.6
  - DL Distance Learning: 234 — 4.7; 4.8; 4.7
  - HQ Washington, DC: 51 — 4.6; 4.8; 4.6
  - IP India: 20 — 4.5; 4.8; 4.7
  - JA Africa: 12 — 4.6; 4.8; 4.6
  - JP Africa: 124 — 4.5; 4.8; 4.6
  - JV Austria: 108 — 4.5; 4.8; 4.7
  - OT Other: 136 — 4.5; 4.8; 4.6
  - RT UAE and other: 20 — 4.4; 4.8; 4.6
  - ST Singapore: 75 — 4.6; 4.8; 4.6
  - Total 514 — 4.5; 4.8; 4.7
- Observed regional differences:
  - Courses in China, Kuwait, and the United Arab Emirates tend to have slightly lower average ratings and together accounted for six of the ten lowest rated courses by skill use.
  - Possible explanations: course types, language, or other factors.
- Course-type variation:
  - Little widespread variation across basic and intermediate courses.
  - All seven course types score between 4.71 and 4.79 in the recommend to others category.
  - FPP (basic) follows aggregate ratings; MDS skill use ratings slightly higher; finance courses align with aggregate.
  - Paired t-test: differences between basic and intermediate courses not statistically significant at 5 percent; at 10 percent basic satisfaction statistically lower than intermediate, t(356) = 1.45, p <0.1.

### End-of-course survey regression results — main coefficients and interpretation
- Regression dependent variables: average rating (skill usefulness, recommend, overall satisfaction).
- Independent variables include: percentage male, percentage central bank (CB), percentage with at least a master’s degree, mean age, regional dummies (APD, AFR, WHD, EUR; MCD omitted), national course dummy.
- Data source: ICD’s Participant Tracking Systems (PATS).
- Key statistically significant findings (Skill rating regressions; coefficients with t-statistics in parentheses):
  - Percentage of Male Participants:
    - (All) 0.18 * (2.4)
    - (Basic) 0.33 * (2.6)
    - (Intermediate) 0.07 (0.6)
    - (Finance) 0.07 (0.3)
  - AFR:
    - (All) 0.10 ** (2.7)
    - (Finance) 0.38 * (2.6)
  - APD:
    - (All) 0.09 ** (2.9)
    - (Intermediate) 0.13 ** (2.7)
    - (Finance) 0.48 ** (4.8)
  - EUR:
    - (All) 0.12 ** (3.1)
    - (Finance) 0.53 *** (4.4)
  - WHD:
    - (All) 0.12 ** (3.8)
    - (Finance) 0.47 ** (4.2)
  - National Course:
    - (All) -0.12 ** (-4.3)
    - (Basic) -0.18 ** (-3.6)
    - (Intermediate) -0.17 ** (-3.8)
- Goodness of fit and samples:
  - Observations: (All) 458; (Basic) 123; (Intermediate) 158; (Finance) 69
  - R-squared: (All) 0.10; (Basic) 0.28; (Intermediate) 0.15; (Finance) 0.34
  - Adjusted R-squared and RMSE reported in source.
- Interpretation:
  - Overall explanatory power limited (low R-squared).
  - Higher share of male participants associated with higher reported skills rating in some samples.
  - National courses associated with lower reported skills ratings.
  - Regional dummies indicate differences with MCD as omitted category.

### Follow-up surveys — scope, response rates, main pooled results
- Follow-up surveys administered 12–18 months after course completion to assess sustained benefits and application on the job.
- Total follow-up surveys conducted: 26 (courses taught between 2006 and 2011 in sample; first course in database taught in 2006).
- In the 26 courses there were 797 participants from 581 agencies.
- Surveys sent and responses:
  - Surveys sent to participants: 260; responses from participants: 183.
  - Surveys sent to sponsors: 163; responses from sponsors: 65.
  - Example 2010 response rates: participants ~70 percent; sponsors ~40 percent.
- Main pooled follow-up findings (participants and sponsors):
  - Questions 1–3 (Performance, Analysis, Advice):
    - Participants: more than 95 percent strongly agree/somewhat agree that training improved job performance, analysis and forecasting, and policy advice.
    - Sponsors: overall less positive than participants, relatively less sure that training improved policy advice.
  - Question 4 (Responsibility):
    - In each surveyed year, less than 30 percent of participants strongly agreed training enhanced job responsibility.
    - 25–40 percent of participants disagreed with Responsibility statement.
    - Sponsors rated additional responsibilities markedly lower than other items but gave a more positive rating on Responsibility than participants did.
  - Question 5 (Career opportunities):
    - Rated quite low by participants; sponsors more positive than participants.
  - Question 6 (Share):
    - Both participants and sponsors agree participants regularly share learned material; participants strongly agreed more than sponsors.
  - Robustness: slight downward trend over time for first four questions but little variation across venue or region.

### Follow-up survey regressions — sample constraints and key coefficients
- Regression sample: 24 observations (24 courses) — dummy variables excluded due to small sample.
- Specification: average rating = β0 + β1 percentage_male + β2 percentage_CB + β3 percentage_with_Masters/PhD + β4 mean_age.
- Selected coefficient findings (t-statistics in parentheses; significance indicated in source):
  - Percentage of Male Participants:
    - Analysis and Forecasting: 0.83** (2.9)
    - Career Opportunities: 1.1* (2.7)
    - Shared Learning: -0.32 (-1.3)
  - Percentage of CB Participants:
    - Additional Responsibilities: 0.74 (1.9)
    - Shared Learning: 0.55** (3.0)
  - Mean Age of Participants:
    - Analysis and Forecasting: -0.03 (-1.8)
  - Constants (selected):
    - Job Performance: 3.62** (5.6)
    - Analysis and Forecasting: 4.09** (7.5)
    - Shared Learning: 3.68** (8.0)
  - Observations: 24 for each regression.
  - R2 values:
    - Job Performance: 0.09
    - Analysis and Forecasting: 0.32
    - Shared Learning: 0.43
- Interpretation:
  - Results described as not very promising overall given small sample.
  - Gender positively correlated with some average course ratings (significant for Analysis and Forecasting and Career Opportunities).
  - Participant’s agency (CB) not broadly influential except for Shared Learning.
  - Education and mean age not statistically significant for most outcomes.

### Linking end-of-course and follow-up survey results
- Approach: compare follow-up responses on improved job performance (questions 1–3) with end-of-course skill use rating (rescaled by factor 0.8 to align scales).
- Pooled yearly and course-type comparisons:
  - Participants continue to perceive IMF training helped improve job performance and technical skills 12–18 months after training.
  - Improvement perception less pervasive for policy advice.
  - Course-type (18 courses across FPP, MDS, MF, MFP, MMF): FPP shows slightly stronger perceived improvement than intermediate MMF; lowest responses for improved policy advice across types.
- Regression linking end-of-course skill rating to follow-up outcomes (sample: 24 observations) — selected coefficients (T-statistics in parentheses):
  - Skill (end-of-course skill rating) coefficients:
    - Improved Job Performance: 0.79** (3.71)
    - Improved Analysis and Forecasting: 0.47* (2.23)
    - Improved Policy Advice: 0.64* (2.50)
  - Percentage of Male Participants:
    - Improved Analysis and Forecasting: 0.79** (3.06)
  - Observations: 24 for each regression.
  - R2, Adjusted R2, RMSE reported in source.
- Interpretation:
  - End-of-course skill rating is a strong predictor of follow-up survey ratings.
  - A unit increase in end-of-course skill rating associated with a 0.79 increase in job performance rating, 0.47 increase in analysis and forecasting rating, and 0.64 increase in policy advice rating in the sample regressions.
  - Gender effect significant for analysis and forecasting in linked regressions.

### Triennial surveys — coverage and main results
- Since 1995, six triennial surveys conducted; earlier surveys not fully comparable due to design changes.
- For this paper, surveys pooled to consider changes over time.
- Response rates have ranged between 40 percent and 60 percent; 2012 response rate: 42 percent.
- Questionnaires sent: 2003 (432), 2006 (516), 2009 (532), 2012 (576).
- General findings:
  - Sponsors generally quite satisfied (strongly or somewhat agree) with training.
  - Nearly 75 percent of respondents strongly agreed on overall satisfaction question.
  - Majority strongly agree training helped improve ability to perform job better.
  - Majority only somewhat agreed training improved policy advice or resulted in added responsibilities/promotions.
  - Lowest average rating across surveys: whether training contributed to additional responsibilities or promotions.
  - Triennial survey findings similar to follow-up survey.
- Subgroup robustness:
  - Across agencies: central banks more favorable than ministries of finance for Q1–Q3; other agencies relatively more favorable for Q2–Q4.
  - Across regions: EUR most favorable for Q1; AFR larger share of strongly agreed responses for Q2 and Q3, strengthened over time.
  - Across income groups (World Bank classification): high income respondents more favorable on Q1; low-income and low-middle income respondents tend to strongly agree training improves job performance and policy formulation (Q2, Q3).
- Summary statistic:
  - Triennial survey reported a 97 percent overall satisfaction rating in pooled results (source text states “The results of this survey are very positive, including a 97 percent overall satisfaction rating.”).

### Conclusions — evaluation methods, additional results, and recommendations
- Evaluation methods described: end-of-course surveys, follow-up surveys, triennial surveys (2006–13 period).
- Consistent message: ICD training is well-liked by participants and sponsoring agencies; overall satisfaction questions rated highest.
- Additional observed results:
  - Geographical differences: courses taught in China and Middle East tend to have lower participant ratings.
  - Follow-up surveys suggest managers/sponsors typically had less favorable views than participants.
    - Possible explanations: insufficient outreach/participation by managers; sponsors not direct supervisors and may not observe on-the-job application.
  - Revision proposed to follow-up surveys: request participant’s direct supervisor in application form and ask supervisor to fill out survey after a much shorter lag than previous 12–18 months.
- Areas for further evaluation work:
  - Document existing evaluation information: organize and analyze individual response data for ICD and gather evaluation data on training conducted outside ICD; establish an ongoing process for storage.
  - Revise tools: implement a more systematic, robust evaluation framework; ICD has introduced a new common evaluation framework that implements RBM for training to routinely monitor effectiveness.
  - Conduct case-study evaluations: design evaluations of a sample of countries that have received extensive training to assess impact on institutional capacity.

*Source: wp18151 - 1. ICD Training Program: End-of-Course Surveys by Department, FY 2007–13 (content excerpt).*

### 1. ICD Training P rogra m: End-of-C ourse Sur veys by Department , FY 2007–13..................13

### 1. ICD Training Program: End-of-Course Surveys by Department, FY 2007–13

### Introduction
- The IMF provides training to its membership in its core areas of expertise as part of its mandate on capacity building.
- Main objective: strengthen the analytical and technical capabilities of member country officials to formulate and manage effectively macroeconomic and financial sector policies.
- The ICD-coordinated program of courses is called the ICD Training Program.
- This paper examines evaluation methods and resulting data collected over the period 2006–13. The assessment tools used during the sample period (2006–13) were subsequently revised over 2015–18 to increase their results focus and introduce course-specific content in the surveys.
- Starting in 2017, the IMF implemented results-based management (RBM) to training.
- Since 2015, ICD has undertaken a review of its curriculum and revamped its courses.
- One objective of the paper: distill information from all evaluations over a seven-year period and combine evaluation information with participant demographic information.

### ICD Training Program (as of 2014)
- Objective: share the expertise of IMF staff in its core areas to help strengthen the skills of officials in member countries to analyze economic developments and formulate and manage macroeconomic and financial sector policies.
- Course characteristics:
  - Practically oriented, focusing on knowledge essential to effective macroeconomic and financial analysis, including statistical and legal frameworks.
  - Integration of theoretical frameworks and real-world applications, using case studies drawn from Fund experience.
  - General approach: “hand-on” workshops and real-world policy applications rather than lecturing.
  - Most courses are two weeks long and are taught by three IMF economists, often referred to as counselors.
    - Typically, the course has three instructors, two full-time IMF staff and one consultant, who is either a former IMF staff member or an academic.
  - Participants are government officials; selection varies by course—application or nomination by organization.
- Provider shares and delivery:
  - ICD courses account for about 55 percent of training.
  - The Statistics Department (STA) provides another sizable share (over 20 percent) of the training.
  - Other IMF departments and regional technical assistance centers (RTACs) provide the rest.
- Volume and geographic distribution:
  - Since 2006 the volume of training has increased almost 25 percent to over 12,500 participant weeks in FY 2013.
  - Training is offered at multiple locations including HQ, RTCs, RTACs, and other venues.
  - As of 2014, there were 4 regional training centers (RTCs): Africa Training Institute; IMF-Middle East Center for Economics and Finance; the Joint Vienna Institute; and the IMF Singapore Regional Training Institute.
  - There were three regional training programs: Joint Regional Training Center for Latin America in Brazil; Joint China-IMF Training Program; and the Joint Partnership for Africa.
  - Expansion of RTCs has facilitated tailoring courses to regional needs; recent growth in training delivered to the Middle East and Central Asia region (MCD) with the opening of a center in Kuwait.
  - In FY 2013, the levels of training delivered to Africa (AFR), MCD, and the Asia and Pacific region (APD) were roughly comparable to each other; adjusting for the number of countries in the region, training to AFR stood at just three-quarters of training to MCD and APD.
  - The India Training Program in Pune delivered training from FY 2007 to FY 2013.

### Results-Based Management (RBM) for Training
- RBM assesses two sequential outcome levels:
  - (i) Reaction, which measures how participants perceived the training based on self-reported satisfaction. The outcome indicator is a composite satisfaction rating derived from the end-of-course surveys regarding participants’ assessments of the value of the course.
  - (ii) Learning, which measures knowledge acquisition using two indicators: the increase in the average post-test score compared to the average pre-test score and the share of participants achieving 60 percent and above on the post-course test.
- On-the-job behavior change is assessed through follow-up surveys sent to participants and their direct supervisors on participants’ use of the knowledge/skills taught in the course and participants’ use of course-specific forecasting techniques/tools/models on the job after training (within three to six months).

### End-of-Course Surveys: Methodology
- End-of-course surveys are the first tier of the evaluation system and are conducted after each training course based on participants’ perceptions.
- Results are routinely presented in a memorandum to the management of the department providing the training (a back-to-office report) following the completion of the training course.
- Historically, there has been no further dissemination of the results or an annual summary of all survey results, partly reflecting the relatively positive responses. Since 2014, these back-to-office reports are shared in the IMF knowledge exchange.
- Questionnaire structure (Box 1 sample referenced):
  - The questionnaire asks participants to evaluate the training course in three specific areas: content or appropriateness of topics covered (section A); presentational methods or the effectiveness of the lectures and workshops (section B); and the overall value of training (section C). There is usually a fourth section for additional comments (section D).
  - Each section consists of two to four questions. The first three sections contain closed-ended questions. Responses are reported on a 1–5 scale with ratings: 1 = strong disagreement; 2 = disagreement; 3 = neutral; 4 = agreement; and 5 = strong agreement.
  - The survey distinguishes between the role of lecturer and counselor, with the latter guiding workshop sessions.
  - Section D sometimes contains course-specific additional questions (for example, after a Financial Programming and Policies (FPP) course, participants are asked whether it has improved their understanding of the IMF macro-framework).
- Administration and response:
  - Historically questionnaires were filled out on paper; some courses and training centers are now gathering the information electronically.
  - The response rate to the surveys is close to 100 percent.

### Evaluation System and Complementary Surveys
- The IMF employs a three-tiered evaluation system linked to the Kirkpatrick (1994) evaluation model:
  - End-of-course surveys (reaction/satisfaction).
  - Follow-up surveys (on-the-job application and behavioral change).
  - Triennial surveys (broader assessments of effectiveness; 2012 triennial survey found 98 percent satisfaction with training programs).
- Other evaluation methods used by IMF training providers include RTC periodic meetings with heads of training and occasional tracer studies or learning quizzes; African Regional training division piloted beginning- and end-course quizzes showing learning in sampled courses.

### Key Messages and Observations from 2006–13 Data (as presented)
- Overall participant and sponsor perceptions of IMF training are very positive.
- The 2012 triennial survey finds an extraordinarily high level of satisfaction: 98 percent with the training programs.
- ICD has systematically gathered information about training effectiveness, but prior to this paper there was limited effort to combine and distill lessons across data sources or to disseminate findings across the Fund and to member country authorities.
- The paper addresses:
  - What methods were used to assess IMF training before 2015?
  - What key messages emerge from the evaluation data?
  - What additional information can be gleaned using filters or participant demographic information?

### Data and Structure of the Paper
- Organization:
  - Section II: brief description of ICD training program as of 2014.
  - Section III: end-of-course survey methods and results.
  - Section IV: methods and results of follow-up surveys.
  - Section V: summary of the triennial survey focusing on training effectiveness.
  - Section VI: brief conclusion and recommendations to fine tune assessment methods.
  - Appendix 1: Data Description.
  - Appendix 2: End-of-Course Survey Regression Results.
- Figures and appendices listed in the paper (figures, boxes, appendices) document training volumes, locations, response rates, follow-up survey results, and regression analyses.

*Source: wp18151 - 1. ICD Training Program: End-of-Course Surveys by Department, FY 2007–13 (content excerpt).*

### Box 1. End-of-Cours e  Surve y Que stionnaire

### Box 1. End-of-Course Survey Questionnaire

### Survey instrument (Sections A–D)
- Section A. Content
  - The topics covered were appropriate.
  - The contents of the lectures met expectations.
  - The workshops were useful.
- Section B. Presentational Methods
  - The structure of the course and the mix of presentational methods (e.g., balance of lectures and workshops) were broadly appropriate.
  - Lecturers were effective in delivering their presentations.
  - Lecturers encouraged discussion and participation.
  - Counselors contributed effectively to workshop sessions.
- Section C. Value (Recorded in database)
  - The knowledge/skills learned during the course will be helpful for your job. (Skill use)
  - You would recommend the course to others. (Recommend to others)
  - Overall you were satisfied with the course. (Satisfaction)
- Section D. Additional Comments

### The data (coverage and composition)
- The ICD database contains information on 863 courses taught within the ICD training program during FY 2007–13.
- About two-thirds (542) of the 863 courses have reported end-of-course survey results.
- Focus is limited to end-of-course surveys for ICD-delivered courses: 514 courses.
- Each record contains:
  - course description, identification number, location, dates of delivery, department teaching the course, and the average rating for each of the three “value” questions in section C.
- Other parts of the survey are used for back-to-office reports and mission chiefs but are not stored in the database.
- Data prior to 2007 is available in paper format only and has not been included in the current analysis.

### What can we learn from the 514 end-of-course surveys — general findings
- Overall average scores (pooled, FY 2007–13):
  - Recommend to others: 4.8
  - Overall satisfaction: 4.7
  - Knowledge/skills learned (Skill use): 4.5
- The lower Skill use rating may reflect the question targeting knowledge directly useful for one’s current job.
- The differences in ratings across survey questions are statistically significant (paired t-tests).
  - Paired t-test results cited: recommend > skill, t (512) = 36.07, p <0.05; recommend > overall satisfaction, t (512) = 22.08, p <0.05; overall satisfaction > skill, t (512) = 15.51, p <0.05.

### Robustness and disaggregation findings
- Distributional variation
  - Notable variation in the distribution of ratings; a few courses receive low ratings.
  - Example: in FY 2007 one course rated around 3.4 for each of the three questions.
  - Skill use has the largest dispersion; recommend to others has the lowest dispersion and highest ratings.
- Time variation
  - Little variation over time; no discernible time trend.
  - Table (FY 2007–13) aggregated averages:
    - Total 514 courses: Skill use 4.5, Recommend to others 4.8, Overall satisfaction 4.7
    - 2007 (16 courses): 4.5, 4.7, 4.6
    - 2008 (82 courses): 4.5, 4.8, 4.6
    - 2009 (814?) [note: table shows 814 but number likely a typesetting error]: 4.5, 4.8, 4.7
    - 2010 (80 courses): 4.5, 4.8, 4.7
    - 2011 (91 courses): 4.5, 4.8, 4.6
    - 2012 (95 courses): 4.5, 4.8, 4.7
    - 2013 (69 courses): 4.6, 4.8, 4.6
- Location (Table 3) — average scores across locations, FY 2007–13 (Courses delivered by INS/ICD only)
  - BT Brazil: Number of courses 27 — Skill use 4.5, Recommend to others 4.8, Overall Satisfaction 4.7
  - CE Kuwait: Number of courses 14 — Skill use 4.4, Recommend to others 4.8, Overall Satisfaction 4.7
  - CT China: Number of courses 16 — Skill use 4.3, Recommend to others 4.5, Overall Satisfaction 4.6
  - DL Distance Learning: Number of courses 234 — Skill use 4.7, Recommend to others 4.8, Overall Satisfaction 4.7
  - HQ Washington, DC: Number of courses 51 — Skill use 4.6, Recommend to others 4.8, Overall Satisfaction 4.6
  - IP India: Number of courses 20 — Skill use 4.5, Recommend to others 4.8, Overall Satisfaction 4.7
  - JA Africa: Number of courses 12 — Skill use 4.6, Recommend to others 4.8, Overall Satisfaction 4.6
  - JP Africa: Number of courses 124 — Skill use 4.5, Recommend to others 4.8, Overall Satisfaction 4.6
  - JV Austria: Number of courses 108 — Skill use 4.5, Recommend to others 4.8, Overall Satisfaction 4.7
  - OT Other: Number of courses 136 — Skill use 4.5, Recommend to others 4.8, Overall Satisfaction 4.6
  - RT UAE and other: Number of courses 20 — Skill use 4.4, Recommend to others 4.8, Overall Satisfaction 4.6
  - ST Singapore: Number of courses 75 — Skill use 4.6, Recommend to others 4.8, Overall Satisfaction 4.6
  - Total 514 — Skill use 4.5, Recommend to others 4.8, Overall Satisfaction 4.7
- Observed regional differences
  - Courses in China, Kuwait, and the United Arab Emirates tend to have slightly lower average ratings and together accounted for six of the ten lowest rated courses (by skill use).
  - These regional differences may reflect course types, language, or other factors.
- Course-type variation
  - Little widespread variation across basic and intermediate courses.
  - Specific course group findings:
    - FPP (Financial Programming and Policies, basic) follows aggregate ratings.
    - Intermediate courses MFP, MMF show broadly similar ratings.
    - MDS (Macroeconomic Diagnostics) — skill use ratings tend to be slightly higher.
    - Finance courses align closely with aggregate.
    - All seven course types score between 4.71 and 4.79 in the recommend to others category.
  - Paired t-test: differences between basic and intermediate courses not statistically significant at 5 percent; at 10 percent, basic satisfaction statistically lower than intermediate, t(356) = 1.45, p <0.1.

### Explaining the ratings — regression approach and main results
- Regression specification (dependent variable): average rating (skill usefulness, recommend, overall satisfaction).
- Independent variables include:
  - Percentage of male participants (male percent)
  - Percentage of central bank participants (agency)
  - Percentage with at least a master’s degree (education)
  - Mean age of participants (age)
  - Regional dummies: APD, AFR, WHD, EUR (MCD omitted)
  - National course dummy (majority participants from same country)
- Data source for this section: ICD’s Participant Tracking Systems (PATS).
- Main interpretation notes:
  - Overall explanatory power of demographic variables is limited (low statistical significance and low R-squared).
  - Key statistically significant findings:
    - Higher share of male participants associated with higher reported skills rating.
    - Participants in national courses tend to report lower skills ratings.
    - Regional dummies show courses delivered in AFR, APD, EUR, WHD received higher skills ratings compared to MCD.
- Table 4. End-of-Course Survey Regression Results — Skill rating (coefficients with t-statistics in parentheses)
  - Columns: (1) All, (2) Basic, (3) Intermediate, (4) Finance
  - Percentage of Male Participants:
    - (1) 0.18 * (2.4)
    - (2) 0.33 * (2.6)
    - (3) 0.07 (0.6)
    - (4) 0.07 (0.3)
  - Percentage of CB Participants:
    - (1) 0.001 (0.03)
    - (2) -0.11 (-1.1)
    - (3) 0.12 (1.4)
    - (4) 0.074 (0.4)
  - Percentage with Masters/Ph.D.:
    - (1) 0.02 (0.3)
    - (2) -0.003 (-0.03)
    - (3) 0.1 (1.1)
    - (4) -0.13 (-0.8)
  - Mean Age of Participants:
    - (1) 0.005 (1.6)
    - (2) -0.002 (-0.3)
    - (3) 0.004 (0.8)
    - (4) 0.02 (1.9)
  - AFR:
    - (1) 0.10 ** (2.7)
    - (2) 0.04 (0.6)
    - (3) 0.09 (1.6)
    - (4) 0.38 * (2.6)
  - APD:
    - (1) 0.09 ** (2.9)
    - (2) -0.06 (-1.1)
    - (3) 0.13 ** (2.7)
    - (4) 0.48 ** (4.8)
  - EUR:
    - (1) 0.12 ** (3.1)
    - (2) 0.11 (1.5)
    - (3) 0.10 (1.8)
    - (4) 0.53 *** (4.4)
  - WHD:
    - (1) 0.12 ** (3.8)
    - (2) 0.02 (0.4)
    - (3) 0.09 (2.0)
    - (4) 0.47 ** (4.2)
  - National Course:
    - (1) -0.12 ** (-4.3)
    - (2) -0.18 ** (-3.6)
    - (3) -0.17 ** (-3.8)
    - (4) -0.10 (-0.5)
  - Constant:
    - (1) 4.15 ** (32.3)
    - (2) 4.49 ** (18.9)
    - (3) 4.1 ** (18.1)
    - (4) 3.3 ** (7.4)
  - Observations:
    - (1) 458
    - (2) 123
    - (3) 158
    - (4) 69
  - R-squared:
    - (1) 0.10
    - (2) 0.28
    - (3) 0.15
    - (4) 0.34
  - Adjusted R-squared and RMSE reported in table (adjusted R2: 0.09, 0.19, 0.22, 0.17; RMSE: 0.10, 0.17, 0.24, 0.19).
  - Note: * and ** indicate coefficients significant at the 95 and 99 percent level (table also reports *** in one cell).

### Summary — key takeaways
- End-of-course surveys capture immediate participant reactions on:
  - skills acquired;
  - likelihood to recommend the course; and
  - overall satisfaction.
- Key messages:
  - Participants are quite satisfied with ICD training.
  - Participants respond most positively to “would recommend the training to others.” Ranking consistent across courses: recommend > satisfied > skill use.
  - Distribution shows notable variation: a few courses receive low ratings, often explained by external factors (e.g., language deficiencies, insufficient technical preparation, mismatch with expectations).
  - Slight variation across course offerings (basic vs. intermediate), regions (small differences, notable for national courses), and years (no notable time trend).
  - Regression insights (limited explanatory power):
    - Higher percentage of males in a course leads to higher reported skills rating.
    - Participants in national courses tend to be more critical (lower skills ratings).
    - Regional variations matter for skills rating.

### Follow-up surveys (preview)
- Follow-up surveys are administered 12–18 months after course completion to participants and sponsors to assess sustained benefits and application of knowledge/skills.
- Follow-up surveys were revised and administered in 2017 with expanded coverage and additional questions on application and barriers/enablers to application.

*Institute for Capacity Development — End-of-Course Survey results and analysis (FY 2007–13).*

### Box 2. Follow-Up Surve ys : Que s tions

### Box 2. Follow-Up Surveys: Questions

### A. Follow-up survey questionnaire
- 1. The course has helped the participant to do his job better (Performance).
- 2. The course has strengthened the participant’s analysis of economic developments and preparation of forecasts (Analysis).
- 3. The course has improved the participant’s ability to provide policy advice (Advice).
- 4. The participant has been given added responsibility as a result of this training (Responsibility).
- 5. I believe this course training will give the participant opportunities to advance in his/her career that he/she might not have had without the training (Career).
- 6. The participant has shared with colleagues what he/she has learned in the course (Share).

### B. Data and response rates
- Total number of follow-up surveys conducted: 26.
- First course in database taught in 2006.
- Number of courses surveyed by year (number in parentheses reported in Figure notes): 2006 (1), 2007 (1), 2008 (1), 2009 (7), 2010 (8), 2011 (8).
- In the 26 courses there were 797 participants coming from 581 agencies.
- Total number of surveys sent to participants: 260; responses from participants: 183.
- Total number of surveys sent to sponsors: 163; responses from sponsors: 65.
- Response-rate example (2010): average response rate for participants was about 70 percent while the average response rate for sponsors was 40 percent.

### C. Main follow-up survey results (pooled)
- General finding: Both participants and sponsors perceive ICD training to be valuable; both view that the training helped participants do their job better.
- Questions 1–3 (Performance, Analysis, Advice):
  - Participants: more than 95 percent strongly agree/somewhat agree that the training resulted in improved job performance, improved analysis and forecasting, and improved policy advice.
  - Sponsors: ratings for the same items are overall less positive than participants’, with sponsors relatively less sure that training resulted in improved policy advice.
- Question 4 (Responsibility):
  - In each year with surveyed courses, less than 30 percent of participants strongly agreed that training enhanced job responsibility.
  - 25–40 percent of participants disagreed with the Responsibility statement.
  - Sponsors rated additional responsibilities markedly lower than for other questions, yet sponsors gave a more positive rating on Responsibility than participants did.
- Question 5 (Career opportunities):
  - Rated quite low by participants.
  - Sponsors are broadly more positive about career opportunities as a result of the training than participants.
- Question 6 (Share):
  - Both participants and sponsors agree participants regularly share learned material with colleagues.
  - Participants strongly agreed more than sponsors.
- Robustness:
  - Focusing on the first four questions, Figure 7 indicates a slight downward trend over time but little variation of ratings across venue of delivery or regions.

### D. Explaining the ratings — econometric analysis
- Sample size constraint:
  - Regression analysis using follow-up survey data covers 24 observations (24 courses).
  - Owing to the small sample size, dummy variables were excluded in the follow-up regression specification.
- Specification used (symbolic form in text): average rating = β0 + β1 percentage_male + β2 percentage_CB + β3 percentage_with_Masters/PhD + β4 mean_age.
- Key findings from Table 5 (Follow-up Survey Regression Results; T-statistics in parentheses; * and ** indicate significance at the 95 and 99 percent level):
  - Percentage of Male Participants coefficients (with t-statistics):
    - Job Performance: 0.25 (0.8)
    - Analysis and Forecasting: 0.83** (2.9)
    - Policy Advice: 0.37 (1.1)
    - Additional Responsibilities: 0.30 (0.6)
    - Career Opportunities: 1.1* (2.7)
    - Shared Learning: -0.32 (-1.3)
  - Percentage of CB Participants coefficients (with t-statistics):
    - Job Performance: 0.27 (1.0)
    - Analysis and Forecasting: 0.09 (0.4)
    - Policy Advice: -0.02 (-0.1)
    - Additional Responsibilities: 0.74 (1.9)
    - Career Opportunities: 0.19 (0.6)
    - Shared Learning: 0.55** (3.0)
  - Percentage with Masters/Ph.D. coefficients (with t-statistics):
    - Job Performance: 0.14 (0.7)
    - Analysis and Forecasting: 0.03 (0.2)
    - Policy Advice: 0.23 (1.1)
    - Additional Responsibilities: 0.15 (0.5)
    - Career Opportunities: -0.06 (-0.2)
    - Shared Learning: 0.17 (1.1)
  - Mean Age of Participants coefficients (with t-statistics):
    - Job Performance: -0.01 (-0.6)
    - Analysis and Forecasting: -0.03 (-1.8)
    - Policy Advice: -0.01 (-0.3)
    - Additional Responsibilities: -0.03 (-0.9)
    - Career Opportunities: -0.02 (-0.9)
    - Shared Learning: -0.01 (-0.7)
  - Constants (with t-statistics):
    - Job Performance: 3.62** (5.6)
    - Analysis and Forecasting: 4.09** (7.5)
    - Policy Advice: 3.39** (5.0)
    - Additional Responsibilities: 3.13** (3.2)
    - Career Opportunities: 3.37** (4.4)
    - Shared Learning: 3.68** (8.0)
  - Observations: 24 (for each column)
  - R2 values:
    - Job Performance: 0.09
    - Analysis and Forecasting: 0.32
    - Policy Advice: 0.10
    - Additional Responsibilities: 0.18
    - Career Opportunities: 0.28
    - Shared Learning: 0.43
  - Adjusted R2, RMSE values are reported in the table (exact values reported in source).
- Interpretation:
  - Results are described as not very promising overall.
  - Positive correlation between gender (percentage male) and average course rating; gender is significant for 2 of the 6 questions (Forecasting and Responsibility per text).
  - Participant’s work agency (central bank, ministry of finance) does not seem to influence course rating except for Shared Learning.
  - Education (Masters/Ph.D.) and mean age do not have statistically significant influence on follow-up survey course ratings.

### E. Linking follow-up surveys with end-of-course evaluations
- Approach:
  - Follow-up surveys (12–18 months after training) are linked to respective end-of-course evaluations by comparing participants’ responses on improved job performance (follow-up questions 1–3) with participants’ responses on skill use in the end-of-course survey.
  - Because the two surveys use different rating scales, end-of-course responses were rescaled (end-of-course survey responses have been rescaled by a factor of 0.8).
- Pooled yearly follow-up results versus end-of-course skill-use (Table 6; specific year-by-year numbers reported in source):
  - Participants continue to perceive that IMF training helped improve job performance and technical skills such as forecasting and policy analysis 12–18 months after training.
  - The improvement perception is less pervasive for policy advice.
- Course-type variation (Table 7; 18 courses across five topics):
  - Course types reported: FPP, MDS, MF, MFP, MMF.
  - Participants in both end-of-course and follow-up surveys perceived IMF training helped improve skills; this result appears slightly stronger for FPP than for intermediate MMF course.
  - Response uniformly lowest for improved policy advice across course types.
- Do end-of-course surveys explain follow-up results? (regression adding end-of-course skill rating — specification shown in text)
  - Sample: 24 observations (same 24 courses).
  - Table 8 key coefficients (T-statistics in parentheses; * and ** indicate significance at the 95 and 99 percent level):
    - Skill (end-of-course skill rating) coefficients:
      - Improved Job Performance: 0.79** (3.71)
      - Improved Analysis and Forecasting: 0.47* (2.23)
      - Improved Policy Advice: 0.64* (2.50)
    - Percentage of Male Participants:
      - Improved Job Performance: 0.19 (0.73)
      - Improved Analysis and Forecasting: 0.79** (3.06)
      - Improved Policy Advice: 0.32 (1.02)
    - Percentage of CB Participants:
      - Improved Job Performance: -0.06 (-0.29)
      - Improved Analysis and Forecasting: -0.11 (-0.51)
      - Improved Policy Advice: -0.28 (-1.07)
    - Percentage with Masters/Ph.D.:
      - Improved Job Performance: 0.13 (0.81)
      - Improved Analysis and Forecasting: 0.03 (0.17)
      - Improved Policy Advice: 0.23 (1.14)
    - Mean Age of Participants:
      - Improved Job Performance: -0.002 (-0.16)
      - Improved Analysis and Forecasting: -0.02 (-1.61)
      - Improved Policy Advice: -0.0003 (-0.02)
    - Constant terms:
      - Improved Job Performance: -0.09 (-0.08)
      - Improved Analysis and Forecasting: 1.88 (1.70)
      - Improved Policy Advice: 0.39 (0.29)
    - Observations: 24 for each regression.
    - R2, Adjusted R2, RMSE reported in table (exact values reported in source).
  - Interpretation:
    - End-of-course ratings on skill are a strong predictor of follow-up survey ratings.
    - A unit increase in end-of-course skill rating leads to a 0.79 increase in job performance rating, a 0.47 increase in analysis and forecasting rating, and a 0.64 increase in policy advice rating (as reported).
    - Gender has a statistically significant effect in column 2 (analysis and forecasting), with higher proportion of males leading to higher course ratings for that category.
    - Remaining variables are not statistically significant.

### F. Summary (Section E of source)
- ICD has conducted 26 follow-up surveys on a sample of courses since 2006 addressing sustainability of training benefits as viewed by participants and sponsoring government agencies.
- Overwhelming majority of respondents perceive ICD training to be valuable in contributing to developing human capital.
- Pooled results highlights:
  - Participants value the training, perceiving improvements in job performance, analysis, and policy advice.
  - Sponsors also value the training but responses are somewhat more muted, particularly on whether training improved policy advice.
  - Neither participants nor sponsors perceive that the training has led to additional responsibilities; sponsors provided a somewhat more positive response on this question relative to participants.
  - No discernible time trend, regional differences, or variation across venue of delivery was found in pooled results.
  - Linking follow-up with end-of-course results:
    - Participants continue to perceive benefits a year and a half after training.
    - End-of-course rating on skill use is a good predictor of follow-up survey ratings.

*Source: Institute for Capacity Development, follow-up surveys and regression tables as presented in the Box 2 content.*

### 1.5 IMF Institute t rain in g h a s  improved the way  our staff formulates and implements policy

### 1.5 IMF Institute training has improved the way our staff formulates and implements policy

### B. The data
- Since 1995, six triennial surveys have been conducted; earlier surveys are not fully comparable with more recent surveys because of sample and questionnaire design and are excluded from some analysis.
- For this paper, results of the surveys have been pooled to consider changes over time.
- Response rates have varied over time, ranging between 40 percent and 60 percent.
- The response rate for the 2012 survey was 42 percent.
- Questionnaires Sent: 2003 (432), 2006 (516), 2009 (532), 2012 (576).

### C. The effectiveness of training: Results — General findings
- Sponsors are generally quite satisfied (strongly or somewhat agree) with their training experience.
- Nearly 75 percent of respondents strongly agreed to the question related to overall satisfaction.
- A majority of respondents strongly agree that the training helped improve participants’ ability to perform their job better.
- The majority of respondents only somewhat agreed that the training improved policy advice or resulted in added responsibilities or promotions of the participants.
- The lowest average rating across surveys is for the question on whether training contributed to additional responsibilities or promotions of the participant.
- Triennial survey findings are similar to those captured in the follow-up survey.

### C. Robustness of results — subgroup breakdowns
- Analysis focuses on strongly agreed responses to explore differences across agencies, regions, and country income classifications.

- Across agencies (Central Bank (CB), Ministry of Finance (MoF), Other):
  - Central banks tend to give more favorable responses relative to ministries of finance, especially for the first three questions (Q1–Q3).
  - Other agencies tend to be relatively more favorable for questions Q2–Q4.
  - Broad agreement across agency types that training has not resulted in participants being given added responsibilities.

- Across regions:
  - For Q1 (overall satisfaction), most favorable responses are from the European region (EUR) across all three years.
  - For Q2 (improved policy analysis and formulation) and Q3 (improved job performance), a larger percentage of strongly agreed responses come from agencies in Africa (AFR); this result has become stronger over time.
  - Little regional variation on Q4 (whether training helps provide additional responsibilities).

- Across income groups (World Bank classification: High, upper-middle, low-middle, low):
  - Respondents from high income countries are relatively more favorable on overall satisfaction (Q1).
  - Respondents from low-income and low-middle income countries tend to strongly agree that training improves job performance and participants’ policy formulation (Q2, Q3).
  - Little variation across income groups on whether training helps add additional responsibilities (Q4).

### D. Summary — key statistics and findings
- A triennial survey of government agencies that have received IMF training has been carried out by ICD since 1995.
- The results of this survey are very positive, including a 97 percent overall satisfaction rating.
- Notable findings:
  - Sponsors broadly agreed they were satisfied with the training experience and that training helped participants do their job better by improving the way staff formulates and implements policy.
  - Few participants were given promotions or added responsibility as a direct result of the training.
  - Differences by agency and region: central banks tend to rate the training higher than ministries of finance; African sponsoring agencies tend to rate effectiveness on improving job performance more strongly.

### VI. Conclusions — evaluation methods, additional results, and recommendations
- The paper describes ICD evaluation methods used from 2006–13: end-of-course surveys, follow-up surveys, and triennial surveys.
- Consistent message: the ICD training program is well-liked across both participants and sponsoring agencies; overall satisfaction questions were rated highest.
- Additional results:
  - Geographical differences: courses taught in China and Middle East tend to have lower ratings by participants relative to similar courses taught across other regions.
  - Follow-up survey suggests different views between participants and sponsoring managers: managers typically had a less favorable view of ICD program impact than participants. Possible explanations:
    - Insufficient outreach or participation by managers (higher ranking officials) in the training program.
    - Sponsors are not the direct supervisors of participants and therefore may not be in a position to observe on-the-job application; they may give more moderate ratings.
  - One revision to follow-up surveys: request information on the participant's direct supervisor in the application form and ask the supervisor to fill out the survey after a much shorter time (rather than previous 12–18 months lag).

- Looking ahead — areas for further evaluation work:
  - Document existing evaluation information: organize and analyze individual response data for ICD courses; include evaluation data on training conducted outside ICD; gather historical information and establish an ongoing process for storing it.
  - Revise the tools: implement a more systematic, robust evaluation framework; ICD has introduced a new common evaluation framework that implements RBM for training to routinely monitor effectiveness.
  - Conduct case-study evaluations: design evaluations of a sample of countries that have received a lot of training to assess the extent to which IMF training has had an impact on the capacity of target institutions and organizations.

*Source: Institute for Capacity Development, pooled triennial and follow-up survey results as presented in the provided content.*

### References

### References

### Listed References
- Edison, Hali J. and Ranae Jabri. 2015. “IMF Training: Does it Build Capacity?” forthcoming (Washington: International Monetary Fund).
- International Monetary Fund. 2008. “Training as Part of Capacity Building-Recent Initiatives and Strategic Considerations,” SM/08/114, (Washington: International Monetary Fund).
- ______________________. 2013. “The Fund’s Capacity Development Strategy—Better Policies through Stronger Institutions—Background Paper” SM/13/129, Supplement 1 (Washington: International Monetary Fund).
- ______________________. 2014. “IMF Policies and Practices on Capacity Development,” (Washington: International Monetary Fund).
- ______________________. 2017. “New Common Evaluation Framework for IMF Capacity Development,” (Washington: International Monetary Fund).
- ______________________. (forthcoming). “Technical Note on Assessment Tools for Training Evaluations,” (Washington: International Monetary Fund).
- Kirkpatrick, Donald L. 1994. “Evaluating Training Programs: The Four Levels,” (San Francisco: Berrett-Koehler).
- Lamdany, Ruben and Louellen Stedman, 2015, Self-evaluation at the IMF: An IEO Assessment, (Washington: International Monetary Fund).
- World Bank Independent Evaluation Group. 2008. “Using Training to Build Capacity for Development: An Evaluation of the World Bank’s Project-Based and WBI Training,” (Washington: The World Bank).

### Appendix 1 — Data Description: Overview
- Focus: data sources used in the paper related to ICD training activities and evaluations.
- Training defined to include all training events within the ICD program; excludes single-country events delivered as part of a TA engagement.
- ICD central repository: Participant and Tracking System (PATS).
  - PATS contains course information starting in 1965 and participant information starting in 1981.
  - Data fields include course dates, delivering departments, names of participants, educational background, and agency affiliation.
- Participant selection: participants either submit an application or are nominated; analysis treats a participant as any official who attended the course, independent of selection method.

### Appendix 1 — End-of-course survey
- Dataset: aggregated end-of-course survey information on about 863 courses taught within the ICD training program during FY 2007–13.
- Each record contains:
  - course description, identification number, location, dates of delivery, department teaching the course.
  - average rating on three “value” questions in section C of the questionnaire:
    - (i) Skill use: “the knowledge/skills learned during the course will be helpful for your job (Skill use)”
    - (ii) Recommend to others
    - (iii) Satisfaction: overall satisfaction with the course (Satisfaction)
  - Responses on a 1–5 rating scale: 1 = strong disagreement; 2 = disagreement; 3 = neutral; 4 = agreement; 5 = strong agreement.
- Note: Data on end-of-course evaluation prior to 2007 is available in paper format only and have not been included in the current analysis.

### Appendix 1 — Follow-up survey
- Timing: 12 to 18 months after course completion.
- Implemented for a select sample of training courses starting in 2006; ICD partnered with an independent research organization to conduct the survey.
- Confidentiality: responses reported to independent research organization which transmits dataset to ICD removing traces of identification.
- Survey questions include:
  - whether the training helped the participant do his/her job better;
  - whether it strengthened analysis of economic developments and preparation of forecasts;
  - whether it improved ability to provide policy advice;
  - whether the participant received added responsibility and/or opportunity for career advancement;
  - whether the participant’s learning had been shared with others.
- Table 1A lists surveyed courses with numbers of participants and sponsors and responses (table contents excluded per overlay rules).

### Appendix 1 — Triennial survey
- Conducted every three years; includes IMF Institute and other IMF departments training.
- Questionnaire collects information from all organizations that sponsored participants in training during the three years prior to the survey on:
  - effectiveness of past training;
  - demand for future training in terms of delivery mode and topic.
- Response scale: 1–4 (1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree).
- Organizations categorized by IMF regional departments: Africa, Asia/Pacific, Middle East and Central Asia, Western Hemisphere.
- Sample classification by agency type: central banks, ministry of finance, other agencies.

### Table A1 — Donor Course Survey Courses, CY 2006–11 (notes)
- Source: Institute for Capacity Development.
- Course mnemonics listed: BS, EFS, FMA, FME, FPP, MDS, MERP, MF, MFP, MFS, MMF, MMNR.
- (Detailed table entries present in source; course mnemonics preserved here.)

### Appendix 2 — End-of-Course Survey Regression Results: Key Findings
- Focus: regressions for two ratings beyond Skill use — Recommend to others and Overall satisfaction.
- Four course samples analyzed: All courses, Basic (FPP), Intermediate (MDS, MFP, MMF), Finance (FMA, EFS, FME).

Recommend to Others (Table A2) — selected findings and exact reported values:
- Percentage of Male Participants coefficient (All): -0.05 (t-stat -0.8); (Finance): -0.53* (t-stat -2.3).
- Percentage CB Participants coefficient (All): -0.07 (t-stat -1.9); (Basic): -0.26** (t-stat -3.6).
- Percentage with Masters/Ph.D. coefficient (All): -0.06 (t-stat -1.3); (Basic): -0.23** (t-stat -3.5).
- Mean Age of Participants coefficient (All): 0.004 (t-stat 1.6).
- Regional coefficients (All sample):
  - AFR: 0.05 (t-stat 1.8)
  - APD: -0.03 (t-stat -1.3)
  - EUR: 0.05 (t-stat 1.6)
  - WHD: 0.03 (t-stat 1.3)
  - MCD omitted.
- National Course coefficient (All): -0.06* (t-stat -2.5); (Basic): -0.13** (t-stat -3.6).
- Constant (All): 4.74** (t-stat 46.0).
- Observations: 458 (All); 123 (Basic); 158 (Intermediate); 69 (Finance).
- R2: 0.09 (All); 0.42 (Basic); 0.07 (Intermediate); 0.36 (Finance).
- RMSE reported by column: 0.07, 0.15, 0.37, 0.12.
- Note: * and ** indicate coefficients significant at the 95 and 99 percent level. Table reports regression results estimated using end-of-course survey data from courses delivered from January 1, 2007 through December 31, 2012.

Overall Satisfaction (Table A3) — selected findings and exact reported values:
- Percentage of Male Participants coefficient (All): -0.11 (t-stat -1.6); (Finance): -0.65** (t-stat -2.8).
- Percentage of CB Participants coefficient (All): 0.03 (t-stat 0.6).
- Percentage with Masters/Ph.D. coefficient (All): -0.12* (t-stat -2.4); (Basic): -0.2* (t-stat -2.4).
- Mean Age of Participants coefficient (All): 0.006* (t-stat 2.1).
- Regional coefficients (All sample):
  - AFR: 0.05 (t-stat 1.4)
  - APD: 0.06* (t-stat 2.0)
  - EUR: 0.18** (t-stat 5.1)
  - WHD: 0.08** (t-stat 2.7)
  - MCD omitted.
- National Course coefficient (All): 0.01 (t-stat 0.4).
- Constant (All): 4.52** (t-stat 39.4).
- Observations: 458 (All); 123 (Basic); 158 (Intermediate); 69 (Finance).
- R2: 0.1 (All); 0.18 (Basic); 0.17 (Intermediate); 0.38 (Finance).
- RMSE reported by column: 0.08, 0.17, 0.12, 0.15.
- Note: * and ** indicate coefficients are significant at the 95 and 99 percent level. Table reports regression results estimated using end-of-course survey data from courses delivered from January 1, 2007 through December 31, 2012.
- Additional note in text: “Neither of the coefficients suggests that these are economically significant. A 10 percent increase in the number of participants with Masters/Ph.Ds. leads to a decline in overall satisfaction by 0.01 points while an increase in mean age by five years results in a 0.03-point increase in course satisfaction.”

*Source: wp18151 - References (wp18151 - References)*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2018/wp18151.pdf_