## Elastic Net

## Source details

**Canonical URL:** [Elastic Net](https://www.imf.org/-/media/files/publications/wp/2018/wp18230.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2018/wp18230.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2018/wp18230.pdf.json)

---

### Overview
- The Elastic Net algorithm provides a regression-based approach to forecasting that:
  - Retains advantages of classic OLS in terms of interpreting the contribution of each variable.
  - Offers a flexible way to conduct variable selection and prevent forecast over-fitting.
- Elastic Net is one of three supervised machine learning algorithms employed in the study alongside the Recurrent Neural Network (RNN) and Super Learner.

### Methodology and estimation details
- Elastic Net includes a penalty strength parameter denoted by λ that is tuned as part of model calibration.
- Model tuning and validation use k-fold cross-validation; the paper notes that it is common to apply 10 folds for learner parameter calibration.
- Testing/validation protocol:
  - Training set used to train the learner algorithm.
  - Test set held out to measure out-of-sample predictive performance and avoid misleading comparisons due to over-fitting.
  - Cross-validation splits the training data into k folds and iteratively re-estimates to calibrate parameters.

### Methodology and formulation
- Elastic Net combines ridge regression and the least absolute shrinkage and selection operator (LASSO) into a single penalized regression framework to improve OLS by performing dimension reduction and/or variable selection when dealing with large datasets with multiple, possibly correlated regressors.
- Ridge regression: minimizes residual sum of squares (RSS) plus a ridge penalty scaled by λ that shrinks coefficients toward zero but does not set them exactly to zero. The shrinkage intensity is controlled by the parameter λ; a higher λ implies stronger shrinkage, and λ = 0 yields standard OLS results.
- LASSO: minimizes RSS plus an L1 penalty scaled by λ that can set coefficients exactly to zero when λ is large enough, enabling variable selection.
- Elastic Net objective (combining ridge and LASSO penalties) uses both λ and an additional mixing parameter α:
  - α determines the relative weights of the ridge and LASSO penalties.
  - α = 0 yields the ridge regression specification.
  - α = 1 yields the LASSO specification.
- Both λ and α are determined in practice via iterative cross-validation.

### Penalty parameters and special cases
- The Elastic Net generalizes LASSO and ridge and is especially advantageous in the following situations noted in the source:
  - p > n (the number of regressors exceeds the number of observations, i.e., "fat data").
  - A group of variables show high pairwise correlations.
  - n > p (the number of observations largely exceeds the number of regressors, i.e., "tall data").
- Specific behaviors:
  - For p > n, a conventional LASSO penalty would select at most n variables in the resulting regression due to the convex nature of the optimization problem.
  - For high pairwise correlations among regressors, LASSO tends to select only one of the correlated regressors with little emphasis on which one is kept.
  - For tall data with strong correlations between regressors, LASSO can be outperformed by ridge regression.
- The Elastic Net’s combination via α aims to ensure predictive performance at least as powerful as either standalone approach.
- Implementation note:
  - Optimal λ and α are selected via cross-validation in practice.

### Comparative advantages and practical considerations
- Advantages highlighted:
  - Intuitiveness and high computational efficiency.
  - Resilience against potential multicollinearity among regressors.
  - Incorporation of both dimension reduction and variable selection in a single model.
  - Ability to produce an output indicating selected variables and their respective weights in the final model specification, enabling interpretability relative to many machine learning "black-box" models.
- Empirical and simulation evidence cited:
  - Smeekes and Wijler (2018) confirm penalized regression methods are more robust to mis-specification than the dynamic factor approach.
  - Zou and Hastie (2005) present the Elastic Net as a generalization of LASSO with superior performance in the situations listed above.

### Application context and dataset
- Forecast target and horizon:
  - One-step ahead GDP growth forecasts were computed. For quarterly data, "one-step ahead" designates the forecast for next quarter; for annual data the next year.
- Countries analyzed using the Elastic Net (and other ML methods):
  - Germany, Mexico, Philippines, Spain, United Kingdom, United States, and Vietnam.
  - Country coverage includes three advanced/G-7 economies (United States, United Kingdom and Germany) and emerging and developing economies (Mexico, Philippines and Vietnam). Spain is included as an example of a Euro-area, recession hit country.
- Data sources and scope:
  - Primary macroeconomic source: International Monetary Fund’s World Economic Outlook (WEO) database as of April 2017.
  - Additional series: PMI, business and consumer confidence indexes, stock market indexes, world energy prices (from Bloomberg), and ICRG political/economic/financial risk data (for countries with series available on or after 1984).
  - Forecasts are based on widely available economic data and do not add granular or novel data sources; some annual specifications supplement annual data with current quarterly/monthly data for now-casting.
- Data availability constraints:
  - Exact data series vary by country but consistently represent indicators or proxies for overall business sentiment.
  - All data for each country (besides world energy prices) were restricted to what was potentially available to WEO forecasters in April of each respective year.
  - Mixed frequency lead-time indicators and granular proxies (e.g., satellite data) were excluded to preserve comparability.

### Temporal alignment and forecasting setup
- Forecast target: one-year ahead annual real GDP growth (dependent variable Y).
  - Annual Y at time t is associated with explanatory variables X in time t−1 for training.
  - Last actual data from 2016 (X_t+n) used to produce a real GDP growth forecast representing the 2017 annual growth figure (Y_t+n).
- Quarterly forecasting: quarterly y-o-y growth figures Y_t associated with explanatory variables from a quarter ago X_t−1 (e.g., X data of 2016 Q4 to predict growth in 2017 Q1).
- Training data vintage: WEO benchmarks taken from the April edition of each year’s World Economic Outlook; data used was available before April in each year.
- Variable selection: All three machine learning models were fed all possible variables; no prior variable selection by the authors.
- Modeling scope: For each country, only that country’s data was used (except world energy prices).

### Estimation procedure and evaluation methodology
- Out-of-sample evaluation:
  - "Recursive out-of-sample" RMSE calculation in the style of Clark and McCracken (2001).
  - On any date t, include all data from the beginning of the series up to t in the training set; enlarge training data stepwise as new observations arrive.
  - RMSEs calculated for the single point forecast in t+h, representing a practical "online" ML context.
  - This design mitigates over-fitting because each forecast uses a virtually new training set.
- Forecast comparison benchmark:
  - Benchmarked against IMF’s World Economic Outlook (WEO) forecasts.
  - WEO annual forecast errors: compare annual next-year forecasts reported in fall WEO vintages (October) to actual outruns.
  - WEO quarterly forecast errors: compare next-quarter forecasts reported in spring (April) and fall (October) vintages to actual outruns.
- Test period for RMSE aggregation: squared the individual forecast errors for each year, averaged and square-rooted to derive an overall RMSE for a test period from 2010 to 2016.
- Test-sample selection rationale:
  - Fixed-period test set used to ensure inclusion of extreme events (e.g., global financial crisis 2008-2009) in the training set rather than the test set.
  - Robustness checks include contemplating two different test periods (see text).

### Machine learning models used
- Models mentioned: Recurrent Neural Network (RNN), Elastic Net, Super Learner.
- Estimation note: Elastic Net and Super Learner produce outputs specifying selected variables and ensemble models; however, a new model specification is estimated for each year (or quarter). Individual model specifications are not reported due to large number.

### Key empirical findings (pertaining to Elastic Net within the ML suite)
- Overall accuracy improvements of ML forecasts (the suite that includes Elastic Net) over the WEO benchmarks:
  - Quarterly forecasts: range between 49%-82% (depending on the country).
  - Annual forecasts: range between 4%-38%.
- Relative performance across algorithms:
  - The ensemble-based approach (Super Learner) consistently outperforms other approaches when dealing with quarterly data.
  - The RNN offers some advantages for a select set of advanced countries when forecasting on an annual horizon.
- Practical considerations:
  - Elastic Net combines interpretability (OLS-like contribution assessment) with variable selection and over-fitting control via penalization.
  - Calibration of λ and other tuning parameters is performed through cross-validation to avoid over-fitting.

### Forecast performance — quarterly 1-step ahead RMSEs (test period 2010–2016)
- Reported RMSEs and related statistics (table rows preserve exact values):
  - United States: obs. 118, WEO 0.72, RNN 0.77, Elastic Net 0.46, Super Learner 0.33, Accuracy Increase 55%
  - United Kingdom: obs. 98, WEO 0.63, RNN 0.71, Elastic Net 0.43, Super Learner 0.25, Accuracy Increase 60%
  - Germany: obs. 91, WEO 1.18, RNN 1.55, Elastic Net 0.73, Super Learner 0.37, Accuracy Increase 69%
  - Spain: obs. 79, WEO 1.05, RNN 3.21, Elastic Net 0.27, Super Learner 0.17, Accuracy Increase 83%
  - Mexico: obs. 123, WEO 0.95, RNN 1.46, Elastic Net 0.87, Super Learner 0.62, Accuracy Increase 36%
  - Philippines: obs. 67, WEO 2.33, RNN 1.83, Elastic Net 1.09, Super Learner 0.54, Accuracy Increase 77%
  - Vietnam: obs. 54, WEO 1.08, RNN 1.15, Elastic Net 1.08, Super Learner 0.55, Accuracy Increase 49%
- Note: Gray-shaded cells in the original table indicate forecasts outperforming (lower RMSE) the benchmark WEO RMSE in each row.

### Main findings on forecast accuracy (out-of-sample 2010–2016)
- Machine learning algorithms outperform the WEO forecasts in the test period between 2010 and 2016 across all cases examined.
- The Elastic Net and Super Learner models consistently outperform the WEO benchmark.
- The Recurrent Neural Network (RNN) outperforms WEO forecasts only once (Philippines) in the quarterly tests reported.
- The Super Learner is the best-performing ML technique; accuracy improvements of the Super Learner over the WEO benchmark range from 49% (Vietnam) - 83% (Spain), with the average increase being approximately 61% across all countries.
- Examination of Super Learner ensembles shows no discernible pattern of one algorithm receiving disproportionate weight; better-performing model specifications may exist within algorithm families that were not exhaustively explored.

### RMSE benchmarks for 1-step ahead real GDP growth (annual) — out-of-sample 2010–2016 (Table 4)
- Reported RMSEs and Accuracy Increase:
  - United States: obs. 320; WEO 0.73; RNN 0.65; Elastic Net 1.49; Super Learner 1.01; Accuracy Increase 11%
  - United Kingdom: obs. 250; WEO 0.74; RNN 0.55; Elastic Net 1.55; Super Learner 0.76; Accuracy Increase 26%
  - Germany: obs. 441; WEO 1.53; RNN 1.29; Elastic Net 1.26; Super Learner 4.49; Accuracy Increase 18%
  - Spain: obs. 311; WEO 1.87; RNN 2.73; Elastic Net 2.51; Super Learner 2.03; Accuracy Increase –
  - Mexico: obs. 281; WEO 1.17; RNN 3.34; Elastic Net 1.28; Super Learner 1.80; Accuracy Increase –
  - Philippines: obs. 312; WEO 0.41; RNN 1.96; Elastic Net 2.34; Super Learner 3.35; Accuracy Increase 4%
  - Vietnam: obs. 460; WEO 0.79; RNN 1.17; Elastic Net 0.49; Super Learner 0.91; Accuracy Increase 38%
- Commentary:
  - Forecast performance declines in absolute and relative terms when using annual data versus quarterly models.
  - For Spain and Mexico none of the ML models outperform the WEO RMSE at the annual horizon.
  - Accuracy improvements at the annual horizon range from 4% (Philippines) – 38% (Vietnam) with the average increase being approximately 19% across all countries registering an improvement.
  - The low number of observations at the annual horizon greatly hampers ML performance; ML methods tend to perform poorly in small samples with volatile data-series.

### Crisis-period forecasting and additional benchmarks (forecast period 2007–2016) — next-quarter growth (Table 5)
- Reported RMSEs and Accuracy Increase:
  - United States: obs. 118; WEO 1.11; AR 0.94; BMA 0.51; VAR 1.65; ML 0.35; Accuracy Increase 68%
  - United Kingdom: obs. 98; WEO 1.39; AR 1.03; BMA 0.62; VAR 1.35; ML 0.55; Accuracy Increase 60%
  - Germany: obs. 91; WEO 1.56; AR 1.49; BMA 1.17; VAR 1.54; ML 0.41; Accuracy Increase 74%
  - Spain: obs. 79; WEO 1.12; AR 0.80; BMA 0.36; VAR 1.09; ML 0.20; Accuracy Increase 82%
  - Mexico: obs. 123; WEO 1.60; AR 1.52; BMA 2.09; VAR 1.15; ML 0.81; Accuracy Increase 49%
  - Philippines: obs. 67; WEO 2.20; AR 1.37; BMA 1.51; VAR 2.20; ML 0.56; Accuracy Increase 74%
  - Vietnam: obs. 54; WEO 1.27; AR 1.18; BMA 0.88; VAR 1.34; ML 0.49; Accuracy Increase 61%
- Findings:
  - Despite deterioration in RMSEs when including crisis years, ML models still consistently outperform WEO forecasts.
  - Accuracy improvements of the Super Learner over WEO range from 49% (Mexico) – 82% (Spain) with the average increase being approximately 67% across all countries.
  - Traditional econometric models (AR, BMA, VAR) sometimes outperform WEO but are outperformed by the best-performing ML algorithm in each case.

### Sensitivity to data revisions and robustness checks
- Dropping the year 2016 (subject to considerable ex-post revisions) leaves results largely intact:
  - Accuracy improvements range from 34-84 percent for the quarterly horizon and 8-38 percent for the annual horizon.
  - ML forecasts continue to outperform WEO for all countries over the quarterly horizon and for 5 out of 7 countries for the annual horizon (no improvements for Mexico and Spain).
- Caveat:
  - Historical data series may be revised over longer horizons (periodic national accounts revisions), which could affect ex-post forecast quality and bias; direction of bias is unknown ex-ante.

### Limitations, interpretation, and research avenues
- Limitations:
  - Causal inference is limited for many ML algorithms because standard statistical significance tests for individual variables are not generally applicable.
  - Presented forecast errors and underlying forecasts are stand-alone point predictions; explanatory power is not assessed in the traditional econometric sense in this paper.
  - ML algorithm performance is sensitive to training/test splits; performance depends strongly on underlying data properties.
  - Computationally intensive hyperparameter tuning (especially for RNNs) may yield better-performing specifications that were not exhaustively explored.
  - Low sample sizes and volatile series hamper ML performance; caution advised for small-sample, high-volatility contexts.
- Research directions and potential extensions:
  - Explore high-frequency data and auxiliary granular “big data” sources as proxies for traditional macroeconomic indicators.
  - Use recent advances in handling missing data to extend coverage to low-income and developing economies.
  - Add policy variables (monetary and fiscal measures during crises) to forecast policy scenarios beyond baseline forecasts.
  - Investigate alternative specifications for learner algorithms: computational complexity, layer depth and types, and regularization parameters.
  - Develop and apply methods that “unbox” and interpret ML models to provide explanations for outputs and to understand differences across model and expert forecasts.

### Appendix — Methodology notes and Super Learner library
- Practical note on neural networks:
  - Real-world decision-making modeled by networks of perceptrons (neural networks) with multiple layers, weights, and thresholds; single perceptron insufficient for complex relationships.
- Super Learner algorithm library (selected learners listed as in source):
  - bartMachine: Bayesian additive regression trees
  - bayesglm: Bayesian generalized linear model
  - biglasso: extending Lasso model fitting to Big Data
  - caret: classification and regression training
  - cforest: conditional inference trees
  - dbarts: discrete Bayesian additive regression trees sampler
  - earth: multivariate adaptive regression splines
  - extraTrees: extremely randomized trees
  - gam: generalized additive models
  - gbm: gradient boosting
  - glm: generalized linear model
  - glmnet: Elastic Net
  - ipredbagg: improved bagging predictor
  - interaction: polynomial linear model
  - kernelKnn: kernelized k-nearest neighbor
  - knn: k-nearest neighbor
  - ksvm: kernel support vector machine
  - ldalatent: Dirichlet allocation
  - leekasso: Lasso modification using only the top 10 predictors
  - lm: linear model
  - loess: local polynomial regression
  - logreg: logistic regression
  - mean: arithmetic mean
  - nnet: neural network
  - nnls: non-negative least squares
  - polymarsp: polynomial spline regression
  - qda: quadratic discriminant analysis
  - randomForest: random Forest
  - ranger: fast implementation of random Forests
  - ridge: ridge regression
  - rpart: recursive partitioning
  - speedglm: linear and generalized linear models for large data sets
  - speedlm: linear models for large data sets
  - step: stepwise regression
  - stepAIC: stepwise model selection by AIC
  - svm: support vector machine
  - xgboost: eXtreme gradient boost

*Source: wp18230 — IMF Working Paper content provided.*

### 1.    Elastic Net  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . 

### 1.    Elastic Net

### Overview
- The Elastic Net algorithm provides a regression-based approach to forecasting that:
  - Retains advantages of classic OLS in terms of interpreting the contribution of each variable.
  - Offers a flexible way to conduct variable selection and prevent forecast over-fitting.
- Elastic Net is one of three supervised machine learning algorithms employed in the study alongside the Recurrent Neural Network (RNN) and Super Learner.

### Methodology and estimation details
- Elastic Net includes a penalty strength parameter denoted by λ that is tuned as part of model calibration.
- Model tuning and validation use k-fold cross-validation; the paper notes that it is common to apply 10 folds for learner parameter calibration.
- Testing/validation protocol:
  - Training set used to train the learner algorithm.
  - Test set held out to measure out-of-sample predictive performance and avoid misleading comparisons due to over-fitting.
  - Cross-validation splits the training data into k folds and iteratively re-estimates to calibrate parameters.

### Application context
- Forecast target and horizon:
  - One-step ahead GDP growth forecasts were computed. For quarterly data, "one-step ahead" designates the forecast for next quarter; for annual data the next year.
- Countries analyzed using the Elastic Net (and other ML methods):
  - Germany, Mexico, Philippines, Spain, United Kingdom, United States, and Viet-nam.
  - Country coverage includes three advanced/G-7 economies (United States, United Kingdom and Germany) and emerging and developing economies (Mexico, Philippines and Vietnam). Spain is included as an example of a Euro-area, recession hit country.
- Data sources and scope:
  - Forecasts are based on widely available economic data (IMF-WEO, for example) and do not add granular or novel data sources; some annual specifications supplement annual data with current quarterly/monthly data for now-casting.

### Key empirical findings (pertaining to Elastic Net within the ML suite)
- Overall accuracy improvements of ML forecasts (the suite that includes Elastic Net) over the WEO benchmarks:
  - Quarterly forecasts: range between 49%-82% (depending on the country).
  - Annual forecasts: range between 4%-38%.
- Relative performance across algorithms:
  - The ensemble-based approach (Super Learner) consistently outperforms other approaches when dealing with quarterly data.
  - The RNN offers some advantages for a select set of advanced countries when forecasting on an annual horizon.
- Practical considerations:
  - Elastic Net combines interpretability (OLS-like contribution assessment) with variable selection and over-fitting control via penalization.
  - Calibration of λ and other tuning parameters is performed through cross-validation to avoid over-fitting.

*Source: wp18230 - 1.    Elastic Net (IMF working paper content provided)*

### 1.  Elastic Net

### 1. Elastic Net

### Methodology and formulation
- Elastic Net combines ridge regression and the least absolute shrinkage and selection operator (LASSO) into a single penalized regression framework to improve OLS by performing dimension reduction and/or variable selection when dealing with large datasets with multiple, possibly correlated regressors.
- Ridge regression: minimizes residual sum of squares (RSS) plus a ridge penalty scaled by λ that shrinks coefficients toward zero but does not set them exactly to zero. The shrinkage intensity is controlled by the parameter λ; a higher λ implies stronger shrinkage, and λ = 0 yields standard OLS results.
- LASSO: minimizes RSS plus an L1 penalty scaled by λ that can set coefficients exactly to zero when λ is large enough, enabling variable selection.
- Elastic Net objective (combining ridge and LASSO penalties) uses both λ and an additional mixing parameter α:
  - α determines the relative weights of the ridge and LASSO penalties.
  - α = 0 yields the ridge regression specification.
  - α = 1 yields the LASSO specification.
- Both λ and α are determined in practice via iterative cross-validation.

### Penalty parameters and special cases
- The Elastic Net generalizes LASSO and ridge and is especially advantageous in the following situations noted in the source:
  - p > n (the number of regressors exceeds the number of observations, i.e., "fat data").
  - A group of variables show high pairwise correlations.
  - n > p (the number of observations largely exceeds the number of regressors, i.e., "tall data").
- Specific behaviors:
  - For p > n, a conventional LASSO penalty would select at most n variables in the resulting regression due to the convex nature of the optimization problem.
  - For high pairwise correlations among regressors, LASSO tends to select only one of the correlated regressors with little emphasis on which one is kept.
  - For tall data with strong correlations between regressors, LASSO can be outperformed by ridge regression.
- The Elastic Net’s combination via α aims to ensure predictive performance at least as powerful as either standalone approach.

### Comparative advantages and practical considerations
- Advantages highlighted:
  - Intuitiveness and high computational efficiency.
  - Resilience against potential multicollinearity among regressors.
  - Incorporation of both dimension reduction and variable selection in a single model.
  - Ability to produce an output indicating selected variables and their respective weights in the final model specification, enabling interpretability relative to many machine learning "black-box" models.
- Empirical and simulation evidence cited:
  - Smeekes and Wijler (2018) confirm penalized regression methods are more robust to mis-specification than the dynamic factor approach.
  - Zou and Hastie (2005) present the Elastic Net as a generalization of LASSO with superior performance in the situations listed above.
- Implementation note:
  - Optimal λ and α are selected via cross-validation in practice.

### Application to the paper’s forecasting problem
- The dataset used to predict GDP growth rates in the study resembles "fat" data, making the Elastic Net a particularly suitable approach for the forecasting exercise described in the paper.
- The Elastic Net’s ability to perform both variable selection and shrinkage, while handling correlated predictors and p > n configurations, motivates its selection for the GDP growth prediction task.

*Source: wp18230 - 1.  Elastic Net*

### 3.  Preferably achieve balanced set of advanced, emerging and developing economies

### 3.  Preferably achieve balanced set of advanced, emerging and developing economies

### Country selection and data coverage
- Countries included: Germany, Mexico, Philippines, Spain, United Kingdom, United States and Vietnam.
- Rationale for selection:
  - Represent advanced and emerging economies.
  - Reasonable data availability across at least 29 (quarterly) and 11 (annual) macroeconomic variables.
  - Include three advanced/G-7 economies: United States, United Kingdom and Germany.
  - Include diverse emerging economies: Mexico, Philippines and Vietnam.
  - Include Spain as an example of a Euro-area, recession hit country with a faster-than-expected recovery to test ML on crisis-induced volatility episodes.
- Exclusion note: Other developing and low-income economies were excluded due to missing data or availability only for shorter time periods, resulting in prohibitively low numbers of observations.

### Data sources and variables
- Primary macroeconomic source: International Monetary Fund’s World Economic Outlook (WEO) database as of April 2017.
  - WEO provides national accounts, monetary, trade and labor statistics, fiscal data, and balance of payments accounts.
- Additional series added:
  - Survey data: purchasing managers indexes (PMI), business and consumer confidence indexes.
  - Financial market data: stock market indexes, world energy prices (from Bloomberg).
  - Political/economic/financial risk data: International Country Risk Guide (ICRG) for countries with other macroeconomic data available on or after 1984.
- Data availability constraints:
  - Exact data series vary by country but consistently represent indicators or proxies for overall business sentiment.
  - All data for each country (besides world energy prices) were restricted to what was potentially available to WEO forecasters in April of each respective year.
  - Mixed frequency lead-time indicators and granular proxies (e.g., satellite data) were excluded to preserve comparability.

### Temporal alignment and forecasting setup
- Forecast target: one-year ahead annual real GDP growth (dependent variable Y).
  - Annual Y at time t is associated with explanatory variables X in time t−1 for training.
  - Last actual data from 2016 (X_t+n) used to produce a real GDP growth forecast representing the 2017 annual growth figure (Y_t+n).
- Quarterly forecasting: quarterly y-o-y growth figures Y_t associated with explanatory variables from a quarter ago X_t−1 (e.g., X data of 2016 Q4 to predict growth in 2017 Q1).
- Training data vintage: WEO benchmarks taken from the April edition of each year’s World Economic Outlook; data used was available before April in each year.
- Variable selection: All three machine learning models were fed all possible variables; no prior variable selection by the authors.
- Modeling scope: For each country, only that country’s data was used (except world energy prices).

### Estimation procedure and evaluation methodology
- Out-of-sample evaluation:
  - "Recursive out-of-sample" RMSE calculation in the style of Clark and McCracken (2001).
  - On any date t, include all data from the beginning of the series up to t in the training set; enlarge training data stepwise as new observations arrive.
  - RMSEs calculated for the single point forecast in t+h, representing a practical "online" ML context.
  - This design mitigates over-fitting because each forecast uses a virtually new training set.
- Forecast comparison benchmark:
  - Benchmarked against IMF’s World Economic Outlook (WEO) forecasts.
  - WEO annual forecast errors: compare annual next-year forecasts reported in fall WEO vintages (October) to actual outruns.
  - WEO quarterly forecast errors: compare next-quarter forecasts reported in spring (April) and fall (October) vintages to actual outruns.
- Test period for RMSE aggregation: squared the individual forecast errors for each year, averaged and square-rooted to derive an overall RMSE for a test period from 2010 to 2016.
- Test-sample selection rationale:
  - Fixed-period test set used to ensure inclusion of extreme events (e.g., global financial crisis 2008-2009) in the training set rather than the test set.
  - Robustness checks include contemplating two different test periods (see text).

### Machine learning models used
- Models mentioned: Random Neural Network (RNN), Elastic Net, Super Learner.
- Estimation note: Elastic Net and Super Learner produce outputs specifying selected variables and ensemble models; however, a new model specification is estimated for each year (or quarter). Individual model specifications are not reported due to large number.

### Forecast performance (quarterly 1-step ahead RMSEs, test period 2010–2016)
- Benchmark and model RMSEs (table rows preserve formatting and exact values):
  - United States: obs. 118, WEO 0.72, RNN 0.77, Elastic Net 0.46, Super Learner 0.33, Accuracy Increase 55%
  - United Kingdom: obs. 98, WEO 0.63, RNN 0.71, Elastic Net 0.43, Super Learner 0.25, Accuracy Increase 60%
  - Germany: obs. 91, WEO 1.18, RNN 1.55, Elastic Net 0.73, Super Learner 0.37, Accuracy Increase 69%
  - Spain: obs. 79, WEO 1.05, RNN 3.21, Elastic Net 0.27, Super Learner 0.17, Accuracy Increase 83%
  - Mexico: obs. 123, WEO 0.95, RNN 1.46, Elastic Net 0.87, Super Learner 0.62, Accuracy Increase 36%
  - Philippines: obs. 67, WEO 2.33, RNN 1.83, Elastic Net 1.09, Super Learner 0.54, Accuracy Increase 77%
  - Vietnam: obs. 54, WEO 1.08, RNN 1.15, Elastic Net 1.08, Super Learner 0.55, Accuracy Increase 49%
- Note: Gray-shaded cells in the original table indicate forecasts outperforming (lower RMSE) the benchmark WEO RMSE in each row.

*Source: wp18230 - 3.  Preferably achieve balanced set of advanced, emerging and developing economies (IMF working paper content).*

### 2016. The columnWEOreports RMSEs from IMF forecasts. Gray-shaded cells indicate better fit than the

### wp18230 - 2016

### Main findings on forecast accuracy (out-of-sample 2010–2016)
- Machine learning algorithms outperform the WEO forecasts in the test period between 2010 and 2016 across all cases examined.
- The Elastic Net and Super Learner models consistently outperform the WEO benchmark.
- The Recurrent Neural Network (RNN) outperforms WEO forecasts only once (Philippines) in the quarterly tests reported.
- The Super Learner is the best-performing ML technique; accuracy improvements of the Super Learner over the WEO benchmark range from 49% (Vietnam) - 83% (Spain), with the average increase being approximately 61% across all countries.
- Examination of Super Learner ensembles shows no discernible pattern of one algorithm receiving disproportionate weight; better-performing model specifications may exist within algorithm families that were not exhaustively explored.

### RMSE benchmarks for 1-step ahead real GDP growth (annual) — out-of-sample 2010–2016 (Table 4)
- Note: RMSEs calculated for annual real GDP growth data on an out-of-sample period from 2010-2016. The column WEO reports RMSEs from IMF forecasts. Gray-shaded cells indicate better fit than the respective benchmark WEO RMSE. The column Accuracy Increase indicates the percentage difference in the RMSE between the ML (best-model) and the WEO forecasts.
- Reported RMSEs and Accuracy Increase:
  - United States: obs. 320; WEO 0.73; RNN 0.65; Elastic Net 1.49; Super Learner 1.01; Accuracy Increase 11%
  - United Kingdom: obs. 250; WEO 0.74; RNN 0.55; Elastic Net 1.55; Super Learner 0.76; Accuracy Increase 26%
  - Germany: obs. 441; WEO 1.53; RNN 1.29; Elastic Net 1.26; Super Learner 4.49; Accuracy Increase 18%
  - Spain: obs. 311; WEO 1.87; RNN 2.73; Elastic Net 2.51; Super Learner 2.03; Accuracy Increase –
  - Mexico: obs. 281; WEO 1.17; RNN 3.34; Elastic Net 1.28; Super Learner 1.80; Accuracy Increase –
  - Philippines: obs. 312; WEO 0.41; RNN 1.96; Elastic Net 2.34; Super Learner 3.35; Accuracy Increase 4%
  - Vietnam: obs. 460; WEO 0.79; RNN 1.17; Elastic Net 0.49; Super Learner 0.91; Accuracy Increase 38%
- Commentary:
  - Forecast performance declines in absolute and relative terms when using annual data versus quarterly models.
  - For Spain and Mexico none of the ML models outperform the WEO RMSE at the annual horizon.
  - Accuracy improvements at the annual horizon range from 4% (Philippines) – 38% (Vietnam) with the average increase being approximately 19% across all countries registering an improvement.
  - The low number of observations at the annual horizon greatly hampers ML performance; ML methods tend to perform poorly in small samples with volatile data-series.

### Crisis-period forecasting and additional benchmarks (forecast period 2007–2016) — next-quarter growth (Table 5)
- Note: Table 5 reports RMSEs for next-quarter real GDP growth forecasts for quarterly data from 2007-2016. Columns: WEO (IMF forecasts), AR (Autoregressive Model), BMA (Bayesian Model Averaging), VAR (Vector Autoregressive Model), ML (best-performing ML algorithm). Gray-shaded cells indicate the best RMSE for each country. Accuracy Increase indicates the percentage difference in the RMSE between the ML (best-model) and the WEO forecasts.
- Reported RMSEs and Accuracy Increase:
  - United States: obs. 118; WEO 1.11; AR 0.94; BMA 0.51; VAR 1.65; ML 0.35; Accuracy Increase 68%
  - United Kingdom: obs. 98; WEO 1.39; AR 1.03; BMA 0.62; VAR 1.35; ML 0.55; Accuracy Increase 60%
  - Germany: obs. 91; WEO 1.56; AR 1.49; BMA 1.17; VAR 1.54; ML 0.41; Accuracy Increase 74%
  - Spain: obs. 79; WEO 1.12; AR 0.80; BMA 0.36; VAR 1.09; ML 0.20; Accuracy Increase 82%
  - Mexico: obs. 123; WEO 1.60; AR 1.52; BMA 2.09; VAR 1.15; ML 0.81; Accuracy Increase 49%
  - Philippines: obs. 67; WEO 2.20; AR 1.37; BMA 1.51; VAR 2.20; ML 0.56; Accuracy Increase 74%
  - Vietnam: obs. 54; WEO 1.27; AR 1.18; BMA 0.88; VAR 1.34; ML 0.49; Accuracy Increase 61%
- Findings:
  - Despite deterioration in RMSEs when including crisis years, ML models still consistently outperform WEO forecasts.
  - Accuracy improvements of the Super Learner over WEO range from 49% (Mexico) – 82% (Spain) with the average increase being approximately 67% across all countries.
  - Traditional econometric models (AR, BMA, VAR) sometimes outperform WEO but are outperformed by the best-performing ML algorithm in each case.

### Sensitivity to data revisions and robustness checks
- Dropping the year 2016 (subject to considerable ex-post revisions) leaves results largely intact:
  - Accuracy improvements range from 34-84 percent for the quarterly horizon and 8-38 percent for the annual horizon.
  - ML forecasts continue to outperform WEO for all countries over the quarterly horizon and for 5 out of 7 countries for the annual horizon (no improvements for Mexico and Spain).
- Caveat:
  - Historical data series may be revised over longer horizons (periodic national accounts revisions), which could affect ex-post forecast quality and bias; direction of bias is unknown ex-ante.

### Limitations, interpretation, and research avenues
- Limitations:
  - Causal inference is limited for many ML algorithms because standard statistical significance tests for individual variables are not generally applicable.
  - Presented forecast errors and underlying forecasts are stand-alone point predictions; explanatory power is not assessed in the traditional econometric sense in this paper.
  - ML algorithm performance is sensitive to training/test splits; performance depends strongly on underlying data properties.
  - Computationally intensive hyperparameter tuning (especially for RNNs) may yield better-performing specifications that were not exhaustively explored.
  - Low sample sizes and volatile series hamper ML performance; caution advised for small-sample, high-volatility contexts.
- Research directions and potential extensions:
  - Explore high-frequency data and auxiliary granular “big data” sources as proxies for traditional macroeconomic indicators.
  - Use recent advances in handling missing data to extend coverage to low-income and developing economies.
  - Add policy variables (monetary and fiscal measures during crises) to forecast policy scenarios beyond baseline forecasts.
  - Investigate alternative specifications for learner algorithms: computational complexity, layer depth and types, and regularization parameters.
  - Develop and apply methods that “unbox” and interpret ML models to provide explanations for outputs and to understand differences across model and expert forecasts.

### Appendix — Methodology notes and Super Learner library
- Practical note on neural networks:
  - Real-world decision-making modeled by networks of perceptrons (neural networks) with multiple layers, weights, and thresholds; single perceptron insufficient for complex relationships.
- Super Learner algorithm library (selected learners listed as in source):
  - bartMachine: Bayesian additive regression trees
  - bayesglm: Bayesian generalized linear model
  - biglasso: extending Lasso model fitting to Big Data
  - caret: classification and regression training
  - cforest: conditional inference trees
  - dbarts: discrete Bayesian additive regression trees sampler
  - earth: multivariate adaptive regression splines
  - extraTrees: extremely randomized trees
  - gam: generalized additive models
  - gbm: gradient boosting
  - glm: generalized linear model
  - glmnet: Elastic Net
  - ipredbagg: improved bagging predictor
  - interaction: polynomial linear model
  - kernelKnn: kernelized k-nearest neighbor
  - knn: k-nearest neighbor
  - ksvm: kernel support vector machine
  - ldalatent: Dirichlet allocation
  - leekasso: Lasso modification using only the top 10 predictors
  - lm: linear model
  - loess: local polynomial regression
  - logreg: logistic regression
  - mean: arithmetic mean
  - nnet: neural network
  - nnls: non-negative least squares
  - polymarsp: polynomial spline regression
  - qda: quadratic discriminant analysis
  - randomForest: random Forest
  - ranger: fast implementation of random Forests
  - ridge: ridge regression
  - rpart: recursive partitioning
  - speedglm: linear and generalized linear models for large data sets
  - speedlm: linear models for large data sets
  - step: stepwise regression
  - stepAIC: stepwise model selection by AIC
  - svm: support vector machine
  - xgboost: eXtreme gradient boost

*Source: IMF Working Paper wp18230 (2016) — content as provided.*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2018/wp18230.pdf_