## Deus ex Machina? A Framework for Macro Forecasting with Machine Learning

## Source details

**Canonical URL:** [Deus ex Machina? A Framework for Macro Forecasting with Machine Learning](https://www.imf.org/-/media/files/publications/wp/2020/english/wpiea2020045-print-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2020/english/wpiea2020045-print-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2020/english/wpiea2020045-print-pdf.pdf.json)

---

### Abstract and key findings
- Framework developed to nowcast (and forecast) economic variables with machine learning techniques.
- ML models predict real output growth with lower forecast errors than traditional models; ensembles of multiple ML models lower forecast errors further.
- Measures of variable importance are identified to improve transparency of ML-based forecasts.
- Applying the framework to Turkey reduces forecast errors by at least 30 percent relative to traditional models.
- The framework better predicts economic volatility and large swings in growth rates.

### I. Motivation and scope
- Traditional forecasting (OLS-based) struggles with collinearity, dimensionality, predictor relevance, and nonlinearity.
- Dynamic factor models address collinearity and dimensionality but not predictor relevance or nonlinearity; they perform poorly when the target variable is volatile.
- ML methods emphasize out-of-sample performance and better handle nonlinear interactions among many predictors.
- Evidence summarized: penalized ML methods and supervised factor models often outperform traditional factor models in forecast accuracy and robustness.

### II. Forecasting basics — bias-variance tradeoff
- Forecast objective: pick function f(·) mapping predictor vector X_t (K×1) to h-step ahead forecast y_t+h to minimize average loss L(y_t+h − f(X_t)).
- Under quadratic loss, expected squared forecast error decomposes exactly as:
  - squared bias + variance + irreducible error (σ^2).
- Tradeoff: more complex models reduce bias but increase variance through overfitting; impossible in general to reduce both simultaneously.
- Sources of model complexity: number of variables, number of parameters, number of relationships (e.g., neural networks), number of observations used per prediction (e.g., nearest neighbors).

### III. Shortcomings of OLS-based forecasting methods
- With mean-zero predictors and error term i.i.d. N(0,σ^2) independent of X_t, OLS expected loss under quadratic loss becomes:
  - E[(f(X_t) − y_t+h)^2] = [E(y_t+h) − f(X_t)]^2 + (X_t′X_t/T +1)σ^2.
- Identified issues:
  - Collinearity: variance of OLS forecast increases with correlation among predictors.
  - Dimensionality: variance of OLS forecast increases in number of predictors K.
  - Predictor relevance: irrelevant predictors increase forecast error via higher variance without bias reduction.
  - Nonlinearity: OLS is biased when the data generating process is nonlinear.
- Dynamic factor models:
  - Summarize variation into a small set of orthogonal factors (address collinearity and dimensionality).
  - Do not address predictor relevance or nonlinearity; factor summaries may not predict the forecast variable.

### IV. ML advantages and algorithms
- Design goals: optimize bias-variance tradeoff, select predictors for out-of-sample performance, handle nonlinear interactions among many predictors.
- Focused ML methods:
  - Random Forest (RF):
    - Aggregates many regression trees via bootstrap aggregation (‘bagging’) and random variable subsampling.
    - Produces many uncorrelated trees whose overfitting errors average out; requires little tuning.
  - Gradient Boosted Trees (GBT):
    - Sequentially fits shallow trees to previous residuals; final forecast is sum of tree predictions.
    - Targets bias sequentially to achieve low bias and low variance.
  - Support Vector Machine (SVM):
    - Constructs hyperplanes; optimization formalism preserved:
      - find f(X_t)=X_t′β+b and observation-specific slack constants ζ_i and ζ_i* that minimize β′β + C ∑(ζ_i + ζ_i*) subject to y_i − f(X_i) ≤ ε + ζ_i and f(X_i) − y_i ≤ ε + ζ_i*.
    - Complexity controlled by ε and C; cross-validation used to choose C.

### V. Framework for macro forecasting with ML — data and preprocessing
- Application: Turkey (motivated by unsatisfactory traditional forecasts).
- Database: 234 series (country-specific and global indicators) from Haver Analytics.
- Data features and transformations:
  - Mixed-frequency: monthly and quarterly indicators.
  - Deflate nominal indicators where appropriate.
  - Include 12 lags for indicators.
  - Stationary variables: include level and quarter-on-quarter difference.
  - Non-stationary variables: include first- and second-order log differences.
  - Constructed indicators: sovereign term spread, sovereign yield spread, US sovereign term spread, US high yield spread.
- Dimensionality management:
  - Hard thresholding applied; more data can increase forecast errors even with dimensionality reduction.

### VI. Methodological principles emphasized
- Limit preselection: collect broad indicator set and apply systematic transformations and lags rather than aggressive hand preselection.
- Use complementary algorithms (RF, GBT, SVM) and ensembles to exploit complementary strengths and reduce forecast errors.
- Emphasize out-of-sample performance via cross-validation and related techniques.
- Improve interpretability using variable importance measures and Shapley value decompositions for individual forecasts.

### VII. Methods and model selection (procedures)
- Hard thresholding (Bai & Ng, 2007): regress forecast variable on its lags and each indicator; select indicators with absolute t-statistic above threshold chosen by out-of-sample performance.
- Missing data imputation:
  - Initial median imputation.
  - Run Random Forest and replace missing values with weighted average of non-missing observations using random forest proximities.
- Model choices and rationale:
  - Chosen: RF, GBT, SVM (simple, accessible, little tuning, less prone to overfit, computationally inexpensive).
  - Avoided: neural networks (prone to overfitting with limited data), ridge/LASSO/elastic net (sensitive to large unexpected predictor changes).
- Ensembles constructed:
  - Ensemble 1: equal weights.
  - Ensemble 2: inverse root mean squared error (RMSE) weights.
  - Ensemble 3: inverse-RMSE rank weights.

### VIII. Evaluation approach and interpretability
- Rolling out-of-sample forecasts: split historical data into training and test sets; use cross-validation to tune; once calibrated, run model using all historical data available to obtain each nowcast.
- Variable importance:
  - Construct variable importance measures per ML model.
  - Use Shapley Values to decompose individual forecasts into predictor contributions relative to historical mean.
  - Implementation note: R package caret used for variable importance calculations.

### IX. Results — accuracy and forecast performance (Turkey application)
- Benchmark:
  - Static dynamic factor model (DFM) benchmark with three factors.
  - Benchmark RMSE: 1.66.
  - Corresponding mean absolute deviation: about 1.2 percentage points per nowcast.
- Individual ML model performance (RMSE reductions relative to benchmark, 2012–2019):
  - RF reduces RMSE by 24 percent.
  - GBT reduces RMSE by 22 percent.
  - SVM reduces RMSE by 18 percent.
- Alternate evaluation (forecast models):
  - RF and GBT outperform the benchmark by 18 and 22 percent, respectively.
- Performance in volatile quarters (volatile quarter defined as >3 percentage points change versus previous quarter):
  - SVM outperforms any of the models for volatile quarters, improving upon the factor model by 39 percent.
- Time-varying gains (2012 to 2019, relative to benchmark):
  - RF and GBT gain roughly 60 percent in accuracy relative to the benchmark.
  - SVM lowers errors by almost 80 percent.
- Temporal behavior: ML methods tend to move closer to actual quarterly growth as the end of the quarter approaches.
- Model-specific strengths:
  - RF: good overall predictive performance but less able to fully capture large swings.
  - GBT: more volatile predictions and better capture of large swings.
  - SVM: best at capturing large swings, at the expense of more volatility.
- Forecast RMSE observed series (displayed values):
  - 1.24
  - 1.24
  - 1.25
  - 1.62
  - 1.32
  - 1.39
  - 1.70
- Forecast RMSE, Volatile Quarters (RMSE, 2012-2019) observed series:
  - 2.02
  - 1.97
  - 1.95
  - 1.81
  - 2.09
  - 2.37
  - 2.68

### X. Ensemble performance
- All three ensembles outperform the benchmark by about 33 percent (2012–2019).
- Ensemble outperformance is an improvement of at least 9 percentage points compared to individual models.
- Ensembles deliver more stable outperformance over time by combining models with different strengths and partial independence.

### XI. Variable importance and Shapley decomposition (illustrative July 2019 nowcast)
- Top predictors (Ensemble 1, scaled 0 to 100) include:
  - Real GDP Growth, t=-3
  - Turkey Stock Market, t=-1, d1lq
  - Turkey Stock Market, t=-2, d1lq
  - Turkey Stock Market (real), t=-2, d1lq
  - Total Imports, t=0, d1lq
  - Final Demand (real), t=-1, d1lq
  - Turnover, N-Dom. Cap. (real), t=-9, d1lq
  - Vehicle Sales (real), t=0, d1lq
  - Truck Production, t=0, d1lq
  - Unemployment 2, t=-9, d1d1lq
  - Profits (real), t=-1, d1lq
  - US Term Spread, t=-4, levq
  - Consumer Confidence, t=-1, levq; Consumer Confidence, t=0, levq
  - Real Sector Conf., t=-1, d1q; Real Sector Conf., Current Orders, t=0, d1q; Real Sector Conf., CapEx, t=0, d1q
  - Bank Survey, Business, t=-12, d1q; Bank Survey, Business, t=-3, levq
  - Now-Casting Index, t=0, d1q
  - Turnover, N-Dom. Cap. (real), t=-10, d1lq
- Shapley Value decomposition (illustrative months: July, August, September):
  - Relative to the historic mean, lower production indicators and higher inflation contributed to lower forecasts in all months.
  - Over time, the nowcast mainly deteriorated due to worsening financial conditions and consumption indicators.
  - Category contributions shown for Production, Financial, Consumption, Confidence, Other, and Total (Shapley Value contributions measured as percentage point deviation from historical mean).

### XII. Variable inventories (selected)
- Stationary variables (level and first difference) include Consumer Confidence; Capacity Utilization: Consumer Goods; Real Sector Confidence Index; Now-Casting Index (NCI); World Uncertainty Index; Manufacturing PMI; Sovereign CDS Spread; US Federal Funds Effective Rate; CBOE VIX; Real short rate; Term spread (long rate—short rate); Real M2 growth; US credit spread (high yield – 10-year yield); among others.
- Non-stationary variables (first and second log difference) include Gross Domestic Product (Y); Industrial Production (and subcomponents); Truck Production; Registered Motor Vehicles; CPI: All Items and CPI components; M2; Housing Prices (Y); Retail Sales; Exports and Imports (volume and value); Total Bank Loans (Y); Non-Performing Loans; Residents FX deposits in banking sector, USD; World WTI Weekly Average Price; BIS Narrow NEER; and many detailed sectoral and banking series.

### XIII. Conclusions
- Machine learning techniques can improve forecasting performance relative to traditional models.
- Limitations of OLS-based techniques (collinearity, dimensionality, predictor relevance, nonlinearity) often produce large forecast errors when the target is volatile.
- ML advantages:
  - Emphasize out-of-sample performance.
  - Better handle nonlinear interactions among a large number of predictors.
  - Designed to learn complex relationships from past data while resisting over-extrapolation of historical relationships.

*Source: IMF Working Paper WP/20/45, “Deus ex Machina? A Framework for Macro Forecasting with Machine Learning,” Marijn A. Bolhuis and Brett Rayner, February 2020.*

### Section 1

### Deus ex Machina? A Framework for Macro Forecasting with Machine Learning

### Abstract and key findings
- Framework developed to nowcast (and forecast) economic variables with machine learning techniques.
- Machine learning (ML) models predict real output growth with lower forecast errors than traditional models; ensembles of multiple ML models lower forecast errors further.
- Measures of variable importance are identified to improve transparency of ML-based forecasts.
- Applying the framework to Turkey reduces forecast errors by at least 30 percent relative to traditional models.
- The framework better predicts economic volatility and large swings in growth rates, suggesting ML techniques could be an important part of macro forecasting toolkits for many countries.

### I. Introducing machine learning — motivation and scope
- Traditional forecasting (OLS-based) struggles with collinearity, dimensionality, predictor relevance, and nonlinearity.
- State-of-the-art forecasting models such as dynamic factor models address collinearity and dimensionality but not predictor relevance or nonlinearity; they perform particularly poorly when the variable to be predicted is volatile.
- ML methods emphasize out-of-sample performance and better handle nonlinear interactions among many predictors, designed to learn complex relationships while resisting over-extrapolation of historical relationships.
- Literature evidence cited: penalized ML methods and supervised factor models often outperform traditional factor models in forecast accuracy and robustness (references summarized in source).

### II. The basics of forecasting — a bias-variance tradeoff
- Forecasting objective: pick function f(·) mapping predictor vector X_t (K×1) to h-step ahead forecast y_t+h to minimize average loss L(y_t+h − f(X_t)).
- Under quadratic loss, expected squared forecast error decomposes exactly as:
  - squared bias + variance + irreducible error (σ^2).
- Tradeoff: more complex models reduce bias but increase variance through overfitting; impossible in general to reduce both simultaneously.
- Complexity sources: number of included variables, number of parameters, number of relationships (e.g., neural networks), number of observations used per prediction (e.g., nearest neighbors).

### III.A. Shortcomings of OLS-based forecasting methods
- With mean-zero predictors and error term i.i.d. N(0,σ^2) independent of X_t, OLS expected loss under quadratic loss becomes:
  - E[(f(X_t) − y_t+h)^2] = [E(y_t+h) − f(X_t)]^2 + (X_t′X_t/T +1)σ^2  (equation from source preserved).
- Identified issues:
  - Collinearity: variance of OLS forecast increases with correlation among predictors because expected X_t′X_t equals covariance of X_t.
  - Dimensionality: variance of OLS forecast increases in number of predictors K; with orthogonal predictors and T observations, forecast variance proportional to K (specific form preserved in source).
  - Predictor relevance: irrelevant predictors increase forecast error by increasing variance without reducing bias.
  - Nonlinearity: if DGP is nonlinear, OLS is biased as f(X_t)=E(y_t+h) only when underlying model is linear (f(X_t)=β′X_t).
- Dynamic factor models:
  - Help address collinearity and dimensionality by summarizing variation into a small set of orthogonal factors.
  - Do not address predictor relevance or nonlinearity; factor summaries may not predict the forecast variable even when predictors follow a factor structure.

### III.B. Advantages of machine learning methods (overview and algorithms)
- ML methods are designed to optimize the bias-variance tradeoff by selecting predictors to optimize out-of-sample performance and handling nonlinear interactions among many predictors.
- Study focuses on three ML methods: Random Forest; Gradient Boosted Trees; Support Vector Machines.

- Random Forest (RF):
  - Combines multiple decision trees (regression trees) to form aggregate forecasts.
  - Decision trees: repeatedly split data to minimize forecast error; regression trees predict continuous variables by group means; prone to overfitting and path dependence.
  - RF modifications:
    - Random sampling via bootstrap aggregation (‘bagging’) — each tree built on a random sample of observations.
    - At each split, use a random subsample of predictors.
  - Result: many uncorrelated trees whose individual overfitting errors average out; computationally easy and requires almost no tuning — suitable for time-series with relatively few observations.

- Gradient Boosted Trees (GBT):
  - Sequentially constructs decision trees where each tree is trained on previous trees’ residuals (errors).
  - Final forecast is the sum of individual tree predictions.
  - RF vs GBT: RF uses relatively deep trees (low bias, high variance); GBT uses relatively shallow trees (high bias, low variance) and sequentially targets bias so final prediction can have both low bias and low variance.

- Support Vector Machine (SVM):
  - Constructs hyperplanes to partition predictor space; akin to penalized kernel regression.
  - SVM optimization (preserved formalism): find f(X_t)=X_t′β+b and observation-specific slack constants ζ_i and ζ_i* that minimize β′β + C ∑(ζ_i + ζ_i*) subject to y_i − f(X_i) ≤ ε + ζ_i and f(X_i) − y_i ≤ ε + ζ_i*.
  - Complexity parameters ε and C govern margin and penalty; cost parameter C determines model complexity (C = 0 produces simplest hyperplane; large C fits training data perfectly).
  - Cross-validation used to find optimal C balancing bias-variance to maximize out-of-sample accuracy.

### IV. A framework for macro forecasting with machine learning — data and preprocessing
- Application to Turkey motivated by unsatisfactory traditional forecasts.
- Database assembled with 234 separate series (country-specific and global indicators) from Haver Analytics (Tables A5.1 and A5.2 cited in source).
- Data features:
  - Mixed-frequency: monthly and quarterly leading and coincident indicators.
  - Basic transformations applied:
    - Deflation of nominal indicators where appropriate.
    - Inclusion of 12 lags for indicators.
    - For stationary variables (e.g., capacity utilization, consumer confidence): include level and quarter-on-quarter difference.
    - For non-stationary variables (e.g., production, money): include first- and second-order log differences.
  - Constructed indicators: sovereign term spread, sovereign yield spread, US sovereign term spread, US high yield spread.
- Dimensionality management:
  - Hard thresholding applied to address large predictor set; more data can increase forecast errors even with dimensionality reduction.

### Methodological elements emphasized in the framework
- Limit preselection: collect broad set of indicators and apply systematic transformations and lag structure rather than aggressive hand preselection.
- Identify complementary algorithms: use multiple ML methods (RF, GBT, SVM) and ensembles to exploit complementary strengths and reduce forecast errors further.
- Evaluate performance and improve interpretability:
  - Emphasize out-of-sample performance via cross-validation and related techniques (Annex III referenced).
  - Use measures of variable importance and methods (e.g., Shapley values referenced in Annex IV) to uncover contribution of each predictor to individual forecasts and improve transparency.

*Source: IMF Working Paper WP/20/45, “Deus ex Machina? A Framework for Macro Forecasting with Machine Learning,” Marijn A. Bolhuis and Brett Rayner, February 2020.*

### Section 2

### wpiea2020045-print-pdf - Section 2

### Methods and Model Selection
- Hard thresholding (Bai & Ng, 2007): regress the forecast variable on its lags and each individual indicator and select all indicators with an absolute t-statistic above a threshold chosen by comparing out-of-sample forecast performance across a range of thresholds and selecting the threshold that delivers the lowest forecast errors.
- Imputation approach for missing data:
  - Initially imputes missing values with each indicator’s median.
  - Runs a Random Forest and replaces missing values with the weighted average of non-missing observations, where weights are the proximities (the fraction of final nodes shared by two observations) of the random forest.
- Chosen ML models and rationale:
  - Random Forest (RF), Gradient Boosted Trees (GBT), and Support Vector Machines (SVM).
  - Rationale: relatively simple, accessible, require little parameter tuning, less likely to overfit, and computationally relatively inexpensive.
  - Avoided methods: neural networks (prone to overfitting with limited training data), linear penalized regression methods such as ridge, LASSO and elastic net (sensitive to large, unexpected changes in predictor values).
- Ensembles constructed:
  - Ensemble 1: equal weights.
  - Ensemble 2: inverse root mean squared error (RMSE) weights.
  - Ensemble 3: inverse-RMSE rank weights.
- Complementarity of models:
  - RF and GBT: tend to outperform during stable periods of growth.
  - SVM: more likely to pick up effects of extreme shocks and act as a counterweight against GBT and RF; SVM tends to be more volatile in predictions.

### Evaluation Approach and Interpretability
- Performance evaluation:
  - Rolling out-of-sample forecasts: split historical data available at the time of the nowcast into a training set and a test set; use cross-validation techniques to tune model parameters; once calibrated, run the model using all historical data available at that time to obtain each individual nowcast.
- Variable importance and contribution:
  - Construct variable importance measures for each ML model to assess predictor importance.
  - Use Shapley Values to decompose individual forecasts into contributions of each predictor to the forecast’s deviation from its historical mean.
  - Implementation note: R package caret used for variable importance calculations.

### Results — Accuracy and Forecast Performance
- Benchmark:
  - Static dynamic factor model (DFM) benchmark with three factors.
  - Benchmark RMSE: 1.66.
  - Corresponding mean absolute deviation: about 1.2 percentage points per nowcast.
- Individual ML model performance (RMSE reductions relative to benchmark, 2012–2019):
  - RF reduces RMSE by 24 percent.
  - GBT reduces RMSE by 22 percent.
  - SVM reduces RMSE by 18 percent.
- Forecast models (alternate evaluation):
  - RF and GBT outperform the benchmark by 18 and 22 percent, respectively.
- Performance in volatile quarters:
  - Volatile quarter defined as one with a more than 3 percentage points higher or lower growth rate than the previous quarter.
  - SVM outperforms any of the models for volatile quarters, improving upon the factor model by 39 percent.
- Time-varying gains in accuracy (2012 to 2019, relative to benchmark):
  - RF and GBT gain roughly 60 percent in accuracy relative to the benchmark.
  - SVM lowers errors by almost 80 percent.
- Temporal behavior:
  - ML methods tend to move closer to the actual quarterly growth rate as we get closer to the end of the quarter.
- Model-specific strengths:
  - RF: good predictive performance overall but does not fully capture large swings in growth.
  - GBT: more volatile predictions and better capture of large swings.
  - SVM: best at capturing large swings, at the expense of more volatility.

### Ensemble Performance
- Ensemble improvements (2012–2019):
  - All three ensembles outperform the benchmark by about 33 percent.
  - This ensemble outperformance is an improvement of at least 9 percentage points compared to the individual models.
  - Ensembles deliver more stable outperformance over time.
- Ensemble rationale: combining models with different strengths (RF, GBT, SVM) reduces forecast errors relative to individual models by producing a single, weighted forecast; ensembles tend to outperform individual forecasts especially when models are relatively independent yet similar in forecast accuracy.

### Variable Importance and Shapley Decomposition (Illustrative: Turkey nowcasts)
- Variable importance (Ensemble 1, scaled 0 to 100) — top predictors for July 2019 (illustrative list includes):
  - Real GDP Growth, t=-3
  - Turkey Stock Market, t=-1, d1lq
  - Turkey Stock Market, t=-2, d1lq
  - Turkey Stock Market (real), t=-2, d1lq
  - Total Imports, t=0, d1lq
  - Final Demand (real), t=-1, d1lq
  - Turnover, N-Dom. Cap. (real), t=-9, d1lq
  - Turnover, N-Dom., Cap., t=-9, d1lq
  - Vehicle Sales (real), t=0, d1lq
  - Truck Production, t=0, d1lq
  - Unemployment 2, t=-9, d1d1lq
  - Profits (real), t=-1, d1lq
  - US Term Spread, t=-4, levq
  - Consumer Confidence, t=-1, levq; Consumer Confidence, t=0, levq
  - Real Sector Conf., t=-1, d1q; Real Sector Conf., Current Orders, t=0, d1q; Real Sector Conf., CapEx, t=0, d1q
  - Bank Survey, Business, t=-12, d1q; Bank Survey, Business, t=-3, levq
  - Now-Casting Index, t=0, d1q
  - Turnover, N-Dom. Cap. (real), t=-10, d1lq
- Shapley Value decomposition (illustrative months: July, August, September):
  - Relative to the historic mean, lower production indicators and higher inflation contributed to lower forecasts in all months.
  - Over time, the nowcast mainly deteriorated due to worsening financial conditions and consumption indicators.
  - Category contributions shown for Production, Financial, Consumption, Confidence, Other, and Total (Shapley Value contributions measured as percentage point deviation from historical mean).

### Conclusions
- Machine learning techniques can improve forecasting performance relative to traditional models.
- Limitations of OLS-based techniques include collinearity, dimensionality, predictor relevance, and nonlinearity; these limitations often produce large forecast errors when the target is volatile.
- ML advantages:
  - Emphasize out-of-sample performance.
  - Better handle nonlinear interactions among a large number of predictors.
  - Designed to learn complex relationships from past data while resisting over-extrapolation of historical relationships into the future.

*Source: wpiea2020045-print-pdf - Section 2*

### Section 3

### Section 3

### Model selection and cross validation (CV)
- Researcher selects parameters 훽 and 훼 via:
  - min_{훽,훼} L( y_i,t — f(X_i,t, 훽) ) subject to 훽 ∈ Θ(훼)
- 훽 determine the specific function within the model class; 훼 are ‘tuning parameters’ or ‘regularizers’ that determine potential model complexity by constraining 훽 to be in Θ(훼).
- Any ML algorithm consists of:
  - (a) For every degree of model complexity α, find the model configuration β that maximizes forecast accuracy on the training data.
  - (b) Forecast on the test data using this model configuration β.
  - (c) Across all possible α, pick the degree of model complexity α that maximizes forecast accuracy on the test data.
- Cross validation (CV) splits the entire data set into multiple subgroups (‘folds’) used as separate test sets.
  - In this paper, 10 folds are used to tune model complexity parameters.

### ML algorithm components (α and β)
- Class — Model — 훃 (alpha) — 훂 (beta) (most common)
  - Linear — OLS — Linear coefficients — # of variables (e.g., forward stepwise regression)
  - Ridge — L2 norm penalty
  - LASSO — L1 norm penalty
  - Elastic Net — Overall penalty, LASSO/ridge weight
  - Tree-based — Decision tree — Splits — Depth, # of leaves, observations per leaf
  - Random forest — Splits, aggregation rule (...), # of variables, observations per bootstrap
  - Boosted Trees — Splits (...), # of iterations
  - Prototype methods — KNN — Sets of nearest neighbors — K, weighting of neighbors
  - Support Vector Machines — Linear — Linear coefficients — Cost
  - Support Vector Machines — Polynomial — Coefficients — Cost, scale, degree
  - Support Vector Machines — Exponential — Coefficients — Cost, decay factor

### Decision tree and random forest examples (notes)
- Decision tree example:
  - Nowcasting real GDP growth at time t using lags of real GDP growth, stock market growth, and the US term premium.
  - Each leaf contains two training observations; the trained decision tree predicts the average observed GDP growth of these two observations.
- Random forest example:
  - Nowcasting real GDP growth at time t using lags of real GDP growth, stock market growth, and the US term premium.
  - Each tree uses different observations and considers different variables at each split.
  - In this example, each leaf contains only one training observation.
  - The trained RF predicts the average of the GDP growth rates of the leaves that the new observation belongs to.

### Interpreting forecasts: Shapley Values
- Shapley Values:
  - Concept from coalitional game theory measuring contribution of each player when payoff depends on interactions (Shapley, 1953).
  - Constructed as the mean of each player’s marginal contributions for every possible combination of other players’ actions.
  - In ML, Shapley Values measure each variable’s contribution to an individual prediction’s deviation from the historical mean.
  - For an OLS-based model, contributions equal the predictor’s coefficient multiplied by its specific value.
  - Particularly useful for decomposing predictions from methods with interactions among predictors (e.g., a Random Forest).
- Example decomposition (trained ML model predicting real GDP growth):
  - Model prediction: 5 percent for a period where:
    - (i) nominal credit growth is above 10 percent; and
    - (ii) the country’s major trading partner is expanding.
  - Variables act as complements.
  - Average marginal contribution of credit growth being above 10 percent: 4.5 percent.
  - Average marginal contribution of trading partner expanding: 1.5 percent.
  - Historical mean of the model forecast: 1 percent.
  - Shapley Values (reported in example):
    - Credit growth: 3 percent
    - Trading partner expansion: 1 percent
  - Intermediate computations shown in example:
    - 1.5% = (5 − 4 + 1 − (−1)) / 2
    - 4.5% = (5 − 1 + 4 − (−1)) / 2
    - 3.0 = (5 − 1) ⋅ 0.75 ; 1.0 = (5 − 1) ⋅ 0.25

### Forecast performance (RMSE and RSE figures)
- Forecast RMSE (displayed values):
  - 1.24
  - 1.24
  - 1.25
  - 1.62
  - 1.32
  - 1.39
  - 1.70
- Forecast RMSE, Volatile Quarters (RMSE, 2012-2019):
  - 2.02
  - 1.97
  - 1.95
  - 1.81
  - 2.09
  - 2.37
  - 2.68
- Chart legends and series referenced (no additional numeric values):
  - Ensemble 3 (MSE rank), Ensemble 2 (MSE), Ensemble 1 (equal), SVM, GBT, RF, Benchmark
- Time series charts plotted for rolling out-of-sample forecasts vs. actual real GDP growth (percent, quarter on quarter seasonally adjusted) for:
  - Benchmark1, Random forest, SVM, Ensemble 1, Ensemble 2, Ensemble 3, Gradient boosted trees, with dates ranging Jan-12 through Jan-19 and markers of actual vs. forecast.

### Variable inventories
- Table A5.1. Stationary Variables — Level and First Difference (selected entries)
  - Consumer Confidence
  - Capacity Utilization: Consumer Goods
  - Capacity Utilization: Intermediate Goods
  - Capacity Utilization: Investment Goods
  - Capacity Util: Manufacture of Other Non-metallic Mineral Prods
  - CCI: Buying Time of Durables [Present]
  - CCI: Assmt on Consumer Price Chg Rate [Last 12 Mo]
  - Real Sector Confidence Index
  - Real Sec Conf Index: Tot Amt of Orders [Curr Sit]
  - Real Sec Conf Index: Stocks of Fin Goods [Curr Sit]
  - Real Sec Conf Index: Vol of Output over Next 3 Months
  - Real Sector Conf Index: Employment over Next 3 Months
  - Real Sec Conf Index: Tot Amt of Orders Past 3 Months
  - Real Sector Conf Index: Exp Orders over Next 3 Months
  - Real Sector Conf Index: Fixed Investment Expend
  - Real Sector Conf Index: Gen Business Situation
  - CCI: Expect for Wage Chg Rate [12M vs Past 12M]
  - CCI: Number of Ppl Unemployed Exp. [Next 12M]
  - Reference Ask Rate: 1-Month
  - Reference Ask Rate: 3-Month
  - Reference Ask Rate: 9-Month
  - 1 Week Repo Rate [Policy Rate]
  - Late Liquidity Borrowing Rate
  - Late Liquidity Lending Rate
  - Overnight Borrowing Rate
  - Overnight Lending Rate
  - Reference Bid Rate: 3-Month
  - Labor Force Survey: Labor Force Participation Rate
  - Labor Force Survey: Employment Rate
  - Labor Force Survey: Unemployment Rate
  - Unemployment Rate
  - Official/policy interest rates
  - 10-year interest rates
  - Gross ED, percent of GDP
  - Current Account, percent of GDP
  - BoP, percent of GDP
  - Now-Casting Index (NCI)
  - World Uncertainty Index
  - Manufacturing PMI
  - Sovereign CDS Spread
  - Equity Fund Flows, monthly percent
  - Bond Fund Flows, monthly percent
  - Bank Loans Tendency Survey, Enterprises, Expected
  - Bank Loans Tendency Survey, Housing, Expected
  - Bank Loans Tendency Survey, Funding Conditions, Expected
  - JPM Global Composite PMI
  - JPM Global Manufacturing PMI
  - US Corporate High Yield
  - US Federal Funds Effective Rate
  - US 10-year Treasury Yield
  - World Uncertainty Index
  - Sentix Economic Expectations
  - Sentix Current Economic Situation
  - CBOE VIX
  - CBOE 10-year Treasury VIX
  - Real short rate
  - Real long rate
  - Term spread (long rate—short rate)
  - Long dollar spread (long rate – US 10-year yield)
  - Real M2 growth
  - Real US Federal Funds Rate
  - Real US 10 year
  - Real US term spread (10-year yield – FFR)
  - US credit spread (high yield – 10-year yield)

- Table A5.2. Non-Stationary Variables — First and Second Log Difference (selected entries)
  - Gross Domestic Product (Y)
  - Industrial Production
  - Industrial Production: Mining & Quarrying
  - Industrial Production: Manufacturing
  - IP: Intermediate Goods
  - IP: Durable Consumer Goods
  - IP: Nondurable Consumer Goods
  - IP: Energy
  - IP: Capital Goods
  - Automobile Production
  - Truck Production
  - Const Permits: Buildings
  - Const Permits: One Dwelling Residential Buildings
  - Const Permits: 2+ Dwelling Residential Buildings
  - Const Permits: Residences for Communities
  - Const Permits: Hotels & Similar Buildings
  - Const Permits: Office Buildings
  - Const Permits: Wholesale & Retail Trade Buildings
  - Const Permits: Traffic & Communication Buildings
  - Const Permits: Industrial Buildings & Warehouses
  - Const Permits: Public Entertainment
  - Const Permits: Other Nonresidential Buildings
  - Registered Motor Vehicles
  - Registered Motor Vehicles: Cars
  - Registered Motor Vehicles: Trucks
  - Spot Exchange Middle Rate, NY Close: U.S. CPI Based Real Effective Exchange Rate
  - JPMorgan Real Broad Effective Exchange Rate Index, PPI Based
  - PPI Based Real Effective Exchange Rate
  - Exchange Rate, Selling (TL/Euro)
  - Exchange Rate, Selling (TL/100 Yen)
  - Exchange Rate, Selling (TL/Pound)
  - Exchange Rate, Selling (TL/US$)
  - Foreign Trade: Total Merchandise Imports, c.i.f.
  - BOP: Current Account
  - BOP: Current Acct: Goods, Services & Primary Income
  - BOP: Current Account: Goods And Services
  - CB Balance Sheet: Assets
  - CB Balance Sheet: Liabilities
  - CBBS: Liabilities: CB Money: Reserve Money: Deps of Banking Sector
  - CB Bal Sheet: Liabilities: CB Money: Reserve Money: Currency Issued
  - CBBS: Liabilities: CB Money: Reserve Money: Deposits of Nonbank Sector
  - CB Balance Sheet: Foreign Liabilities
  - CB Balance Sheet: Liabilities: Central Bank Money
  - CB Bal Sheet: Domestic Assets : Treasury: Other
  - Consumer Loans and Credit Cards
  - Banking Sector Credit Vol: Deposit Money Banks' Loans
  - ISE National 100 Stock Price Index (Y)
  - Labor Force Survey: Total Labor Force
  - Labor Force Survey: Employment
  - Labor Force Survey: Unemployment
  - Total Labor Force
  - Employment
  - Unemployment
  - Labor Force Survey: Population, 15 Years & Over
  - Labor Force Survey: Nonagricultural Unemployment Rate
  - Labor Force Survey: Not in Labor Force
  - CPI: All Items
  - CPI: Food and Non-alcoholic Beverages
  - CPI: Alcoholic Beverages and Tobacco
  - CPI: Clothing and Footwear
  - CPI: Housing, Water, Electricity, Gas and Other Fuels
  - CPI: Furniture/Furnishings/Carpets/Other Floor Coverings
  - CPI: Health
  - CPI: Transport
  - CPI: Communication
  - CPI: Recreation and Culture
  - CPI: Education
  - CPI: Restaurants and Hotels
  - CPI: Miscellaneous Goods and Services
  - M2
  - Gross ED, nominal
  - PPI
  - EPI
  - IPI
  - ToT
  - Housing Prices (Y)
  - Manufacturing Shipments
  - Retail Sales, value
  - Exports, value
  - Gross Operating Surplus or Corporate Profits (Y)
  - Nominal Final Domestic Demand (Y)
  - Earnings, industry (Y)
  - Earnings, trade and services (Y)
  - Earnings, construction (Y)
  - Exports, capital goods, volume
  - Exports, intermediate goods, volume
  - Exports, consumption goods, volume
  - Imports, capital goods, volume
  - Imports, intermediate goods, volume
  - Imports, consumption goods, volume
  - House sales
  - Total Tourism Income (Y)
  - Total Number of Visitors
  - Retail sales, food, volume
  - Retail sales, non-food, volume
  - Retail sales, automotive fuel, volume
  - Exports: Motor Vehicles and Trailers (Y)
  - Domestic Taxes on Goods and Services (Y)
  - Stamp Duties (Y)
  - Income Taxes (Y)
  - Government: Compensation of Employees (Y)
  - Government: Social Security Contributions (Y)
  - Government: Goods and Services Purchases (Y)
  - Government: Capital Expenditures (Y)
  - Industrial Domestic Turnover: Intermediate (Y)
  - Industrial Domestic Turnover: Durable (Y)
  - Industrial Domestic Turnover: Nondurable (Y)
  - Industrial Domestic Turnover: Capital Goods (Y)
  - Industrial Domestic Turnover: Energy (Y)
  - Industrial Non-Domestic Turnover: Intermediate (Y)
  - Industrial Non-Domestic Turnover: Durable (Y)
  - Industrial Non-Domestic Turnover: Nondurable (Y)
  - Industrial Non-Domestic Turnover: Capital Goods (Y)
  - Industrial Non-Domestic Turnover: Energy (Y)
  - Transportation and Storage Turnover (Y)
  - Motor Vehicle Sales (Y)
  - Domestic Cement Sales (Y)
  - Gross Demand of Electricity (Y)
  - VAT on Imports (Y)
  - Total Tax Revenue (Y)
  - Banking Sector TRY Assets: Non-Performing Loans
  - Banking Sector FX Assets: Non-Performing Loans
  - Foreign Liabilities to Residents: FX Deposits of Banking Sector
  - LT Pvt Loans from Abroad
  - Banking Sector Credit Vol: Dom Loans: TRY Loans
  - Banking Sector Credit Vol: Inv & Dev Bks: TRY Dom Loans
  - Banking Sector Cred Vol: Participation Banks: TRY Dom Loans
  - Banking Sector Cred Vol: Past Due Loans in TRY
  - Banking Sec Cred Vol: TRY Past Due Loans for Inv & Dev Bks
  - Banking Sector Cred Vol: Part Banks Past Due TRY Loans
  - Banking Sector Credit Vol: Dom Loans: FX Loans
  - Banking Sector Credit Vol: Inv & Dev Bks: FX Dom Loans
  - Banking Sector Cred Vol: Participation Banks: FX Dom Loans
  - Banking Sector Credit Vol: Past Due Loans in FX
  - Banking Sec Cred Vol: FX Past Due Loans for Inv & Dev Bks
  - Banking Sector Cred Vol: Part Banks Past Due FX Loans
  - Banking Sector: Loans [FX]
  - Banking Sector: FX Indexed Loans [LC]
  - Banking Sector: Non-Performing Loans [FX]
  - Banking Sector: Loans [LC]
  - Banking Sector: FX Indexed Loans [LC]
  - Banking Sector: Non-Performing Loans [LC]
  - Consumer Loans and Credit Cards
  - Banking Sector Credit Vol: Deposit Money Banks' Loans
  - Consumer Housing Loans
  - Consumer Automobile Loans
  - Individual and Corporate Credit Cards
  - Loans Indexed to FX: Housing Loans
  - Loans Indexed to FX: Automobile Loans
  - Individual Credit Cards: YTL
  - Individual Credit Cards: FX
  - Deposit Banks TRY Loans: Corporate Credit Cards
  - Total Bank Loans (Y)
  - Non-Performing Loans, TRY
  - Non-Performing Loans, FX
  - Residents FX deposits in banking sector, USD
  - Baltic Exchange Dry Index
  - Composite CPI for Advanced Economies
  - HWWI Commodity Price Index
  - Dallas Fed House Price Index
  - World Industrial Production ex Construction
  - CPB World Trade Volume
  - Dow Jones Global Index
  - World WTI Weekly Average Price
  - BIS Narrow NEER
  - Dollar AUD/JPY Nominal ER
  - Deposit Banks FX Loans: Corporate Credit Cards

*Source: wpiea2020045-print-pdf - Section 3*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2020/english/wpiea2020045-print-pdf.pdf_