## 1. The Bias-Variance Tradeoff

## Source details

**Canonical URL:** [1. The Bias-Variance Tradeoff](https://www.imf.org/-/media/files/publications/wp/2020/english/wpiea2020044-print-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2020/english/wpiea2020044-print-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2020/english/wpiea2020044-print-pdf.pdf.json)

---

### I. Introducing optimal pooling with machine learning
- Pooling data from multiple countries is commonly used to forecast country-specific macroeconomic aggregates and has been shown to lower forecast errors relative to predictions based on country-specific data, especially with scarce data.
- Bias-variance tradeoff in pooling:
  - Adding countries with a similar data-generating process (DGP) may reduce variance and improve forecasts.
  - Adding countries with a dissimilar DGP may introduce bias to forecasts.
- Objective: use machine learning to select the optimal set of countries for macro panel forecasting by explicitly optimizing this bias-variance tradeoff.
- Advantages of the proposed method relative to conventional pooling:
  - Applicable to both linear and non-linear models.
  - Designed to maximize out-of-sample prediction accuracy via cross validation.
  - Computationally efficient and applicable to both small and large panels.
  - Comparable-similarity approach can be used to select comparator countries for general panel analyses.

### II. Two-step algorithm for optimal pooling
- Step 1 — Proximity:
  - Infer similarity of economic structures by estimating country-specific DGPs using Random Forest and measuring how well the estimated DGP for one country predicts real output growth in the target country using only the target country’s data.
  - Rank countries by the predictive power of their DGPs for the target country; more predictive countries are considered more proximate (less likely to introduce bias).
- Step 2 — Optimal pool:
  - Cross validate each candidate combination of pooled countries using Random Forest and select the set that minimizes out-of-sample mean squared error.
  - The bias-variance tradeoff implies the optimal pool generally lies between (a) country-specific data only and (b) a very large pooled panel.

### III. Application: dataset, sample, and targets
- Sample and scope:
  - Nowcast quarterly real output growth with a panel of (up to) 102 countries for the period 1987–2018.
  - Nine example target countries: Austria, Canada, Costa Rica, El Salvador, Germany, Lithuania, Mexico, Iceland, Turkey.
- Implementation details:
  - Proximity uses country-level Random Forest estimates of the DGP (see Annex II).
  - Optimal pool selection uses cross validation that minimizes out-of-sample RMSE (see Annex I and Annex III for data and transformations).

### IV. Key empirical findings
- Overall performance:
  - The algorithm produces more accurate forecasts for eight of the nine example countries.
  - For Iceland, the optimal choice indicated by the algorithm is to use country-specific data only.
- Magnitude of improvements:
  - The pooling method reduces forecast errors by up to 20 percent relative to the same forecast model using alternate pools.
  - Overall, the algorithm reduces forecast errors by 10 to 20 percent relative to benchmarks of one country or the full panel.
- Examples of optimal pool sizes and behavior:
  - Turkey:
    - Forecast errors initially fall substantially as more countries are added.
    - Minimum RMSE found using a pool of around 20 countries, where the algorithm lowers forecast errors by about 15 percent.
    - After that point, adding more countries increases forecast error (U-shaped pattern due to bias-variance tradeoff).
  - Austria, Canada, Costa Rica, El Salvador, Germany, Mexico:
    - Out-of-sample accuracy is maximized with a panel of about 20 to 60 countries.
    - For Austria and Germany, adding too many countries can increase forecast errors even beyond the RMSE from using only country-specific data.
  - Lithuania:
    - Algorithm selects only 5 to 10 similar countries.
  - Iceland:
    - Any panel of more than one country yields higher forecast errors than using Iceland alone, reflecting Iceland’s distinctive economic structure.
- Example of proximity outcomes (Turkey):
  - Most proximate countries identified include Mexico, Malaysia, Paraguay, Thailand, Brazil, Japan, Finland, and others (relative inverse RMSE rankings shown in Figure 3 of the source).
  - Least proximate countries include Cote d’Ivoire, Palestinian Territories, Iran, Azerbaijan, Kyrgyz Republic, Macao, Qatar, Belize, Albania, Brunei, North Macedonia, Cameroon, Panama, Bahrain, Sri Lanka, Belarus, Iceland, China, Russia, Lesotho, Mozambique, El Salvador, Mongolia, Portugal, Morocco (relative inverse RMSE shown in Figure 4 of the source).

### V. Interpretation and implications
- The U-shaped relationship between number of pooled countries (K) and forecast error (RMSE) illustrates the bias-variance tradeoff: additional countries reduce variance but may introduce bias once dissimilar countries are included.
- Machine learning-based proximity measures plus cross validation provide a practical, computationally efficient way to navigate the tradeoff and improve out-of-sample nowcasts.
- The approach is flexible and can be applied to linear or non-linear forecasting models and to panels of varying size.

### VI. Conclusions
- Pooling can improve economic model performance but requires careful selection of pooled countries to balance bias and variance.
- The two-step machine learning algorithm—(1) infer DGP similarity (proximity) and (2) select the pool that minimizes out-of-sample forecast error via cross validation—substantially reduces nowcast errors for real output growth in the majority of tested cases.

### ANNEX I. Machine learning and cross validation — key points
- Machine learning objective and overfitting:
  - Seeks the optimal degree of complexity of a model that maximizes out-of-sample forecast accuracy.
  - More complex models tend to exhibit lower bias but higher variance and risk overfitting.
- Cross validation (CV) approach used:
  - Holdout validation is used to assess how a model “would have done in the future.”
- Algorithm cast as steps:
  - (a) Given a degree of model complexity, find the model configuration that maximizes forecast accuracy on the training data.
  - (b) Forecast on the test data using this model configuration.
  - (c) Across all, pick the degree of model complexity that maximizes forecast accuracy on the test data.
  - In this paper, the degree of model complexity is the number of countries to pool (K) and the model configuration that minimizes forecast errors for the training data is a Random Forest run on the panel of K countries most proximate to the one under consideration.

### ANNEX II. Random Forest — configuration and rationale
- Random Forest (RF) summary:
  - Uses forecast combinations of multiple decision trees to construct an aggregate forecast.
  - Computationally efficient and requires almost no tuning of model parameters.
  - Uses bagging (bootstrap aggregation) and random predictor subsampling to increase information utilization and reduce variance.
- Illustrative behavior:
  - Individual trees have low forecast accuracy; RF aggregates to improve performance.

### ANNEX IV. Proximate countries and Random Forest implementation details
- Random Forest configuration for proximate-country analysis: Random Forests with 500 trees.
- Data coverage for proximate-country tables: data up to and including 2017.
- Tables present top and bottom 25 countries that are most predictive of growth for each focal country.
- Example top 5 proximities preserved from tables:
  - Austria top 5: [1] Austria; [2] Belgium; [3] France; [4] Switzerland; [5] Germany.
  - Canada top 5: [1] Canada; [2] United States; [3] France; [4] Denmark; [5] United Kingdom.
  - Costa Rica top 5: [1] Costa Rica; [2] Japan; [3] Nicaragua; [4] Guatemala; [5] Malaysia.
  - El Salvador top 5: [1] El Salvador; [2] Brazil; [3] Morocco; [4] Mexico; [5] Norway.
  - Germany top 5: [1] Germany; [2] Japan; [3] Netherlands; [4] France; [5] Denmark.
  - Iceland top 5: [1] Iceland; [2] Lithuania; [3] Croatia; [4] Finland; [5] Romania.
  - Lithuania top 5: [1] Lithuania; [2] Finland; [3] Estonia; [4] Bulgaria; [5] Japan.
  - Mexico top 5: [1] Mexico; [2] Germany; [3] Japan; [4] Canada; [5] Hungary.
- Notes: tables present full top and bottom 25 lists for each focal country as generated by Random Forests with 500 trees, based on data up to and including 2017.

*Source: wpiea2020044-print-pdf — "1. The Bias-Variance Tradeoff" and Annexes I–IV.*

### 1. The Bias-Variance Tradeoff ________________________________________________________ 5

### 1. The Bias-Variance Tradeoff

### I. Introducing optimal pooling with machine learning
- Pooling data from multiple countries is commonly used to forecast country-specific macroeconomic aggregates and has been shown to lower forecast errors relative to predictions based on country-specific data, especially with scarce data.
- There is a bias-variance tradeoff in pooling:
  - Adding countries with a similar data-generating process (DGP) may reduce variance and improve forecasts.
  - Adding countries with a dissimilar DGP may introduce bias to forecasts.
- Objective: use machine learning to select the optimal set of countries for macro panel forecasting by explicitly optimizing this bias-variance tradeoff.
- Advantages of the proposed method relative to conventional pooling:
  - Applicable to both linear and non-linear models.
  - Designed to maximize out-of-sample prediction accuracy via cross validation.
  - Computationally efficient and applicable to both small and large panels.
  - Comparable-similarity approach can be used to select comparator countries for general panel analyses.

### II. Two-step algorithm for optimal pooling
- Step 1 — Proximity:
  - Infer similarity of economic structures by estimating country-specific DGPs using Random Forest and measuring how well the estimated DGP for one country predicts real output growth in the target country using only the target country’s data.
  - Rank countries by the predictive power of their DGPs for the target country; more predictive countries are considered more proximate (less likely to introduce bias).
- Step 2 — Optimal pool:
  - Cross validate each candidate combination of pooled countries using Random Forest and select the set that minimizes out-of-sample mean squared error.
  - The bias-variance tradeoff implies the optimal pool generally lies between (a) country-specific data only and (b) a very large pooled panel.

### III. Application: dataset, sample, and targets
- Sample and scope:
  - Nowcast quarterly real output growth with a panel of (up to) 102 countries for the period 1987–2018.
  - Nine example target countries: Austria, Canada, Costa Rica, El Salvador, Germany, Lithuania, Mexico, Iceland, Turkey.
- Implementation details:
  - Proximity uses country-level Random Forest estimates of the DGP (see Annex II).
  - Optimal pool selection uses cross validation that minimizes out-of-sample RMSE (see Annex I and Annex III for data and transformations).

### IV. Key empirical findings
- Overall performance:
  - The algorithm produces more accurate forecasts for eight of the nine example countries.
  - For Iceland, the optimal choice indicated by the algorithm is to use country-specific data only.
- Magnitude of improvements:
  - The pooling method reduces forecast errors by up to 20 percent relative to the same forecast model using alternate pools.
  - Overall, the algorithm reduces forecast errors by 10 to 20 percent relative to benchmarks of one country or the full panel.
- Examples of optimal pool sizes and behavior:
  - Turkey:
    - Forecast errors initially fall substantially as more countries are added.
    - Minimum RMSE found using a pool of around 20 countries, where the algorithm lowers forecast errors by about 15 percent.
    - After that point, adding more countries increases forecast error (U-shaped pattern due to bias-variance tradeoff).
  - Austria, Canada, Costa Rica, El Salvador, Germany, Mexico:
    - Out-of-sample accuracy is maximized with a panel of about 20 to 60 countries.
    - For Austria and Germany, adding too many countries can increase forecast errors even beyond the RMSE from using only country-specific data.
  - Lithuania:
    - Algorithm selects only 5 to 10 similar countries.
  - Iceland:
    - Any panel of more than one country yields higher forecast errors than using Iceland alone, reflecting Iceland’s distinctive economic structure.
- Example of proximity outcomes (Turkey):
  - Most proximate countries identified include Mexico, Malaysia, Paraguay, Thailand, Brazil, Japan, Finland, and others (relative inverse RMSE rankings shown in Figure 3 of the source).
  - Least proximate countries include Cote d’Ivoire, Palestinian Territories, Iran, Azerbaijan, Kyrgyz Republic, Macao, Qatar, Belize, Albania, Brunei, North Macedonia, Cameroon, Panama, Bahrain, Sri Lanka, Belarus, Iceland, China, Russia, Lesotho, Mozambique, El Salvador, Mongolia, Portugal, Morocco (relative inverse RMSE shown in Figure 4 of the source).

### V. Interpretation and implications
- The U-shaped relationship between number of pooled countries (K) and forecast error (RMSE) illustrates the bias-variance tradeoff: additional countries reduce variance but may introduce bias once dissimilar countries are included.
- Machine learning-based proximity measures plus cross validation provide a practical, computationally efficient way to navigate the tradeoff and improve out-of-sample nowcasts.
- The approach is flexible and can be applied to linear or non-linear forecasting models and to panels of varying size.

### VI. Conclusions
- Pooling can improve economic model performance but requires careful selection of pooled countries to balance bias and variance.
- The two-step machine learning algorithm—(1) infer DGP similarity (proximity) and (2) select the pool that minimizes out-of-sample forecast error via cross validation—substantially reduces nowcast errors for real output growth in the majority of tested cases.

*Source: IMF Working Paper chapter "1. The Bias-Variance Tradeoff" (figures, annexes, and empirical results summarized from the provided PDF).*

### ANNEX I. MACHINE LEARNING AND CROSS VALIDATION

### ANNEX I. MACHINE LEARNING AND CROSS VALIDATION

### Machine learning objective and overfitting
- Machine learning seeks the optimal degree of complexity of a model that maximizes out-of-sample forecast accuracy.
- More complex models tend to exhibit lower bias by capturing nuances in how predictors affect the forecast variable, but are more likely to capture perturbations (“noise”) in historical data, increasing forecast variance and potentially raising forecast errors (overfitting).

### Cross validation (CV) approach used
- CV is used to find optimal model parameters and to avoid overfitting by assessing performance on a new (test) data set.
- The type of CV used in this paper is holdout validation, which uses only one subgroup and is often used in forecast settings to assess how a model “would have done in the future.”

### Algorithm cast as steps
- (a) Given a degree of model complexity, find the model configuration that maximizes forecast accuracy on the training data.
- (b) Forecast on the test data using this model configuration.
- (c) Across all, pick the degree of model complexity that maximizes forecast accuracy on the test data.
- In this paper, the degree of model complexity is the number of countries to pool (K) and the model configuration that minimizes forecast errors for the training data is a Random Forest run on the panel of K countries most proximate to the one under consideration.

*Source: ANNEX I of the content unit.*

### Random Forest (from ANNEX II)
- Random Forest (RF) uses forecast combinations of multiple decision trees to construct an aggregate forecast.
- RF is computationally efficient and requires almost no tuning of model parameters, making it suitable for time-series data with relatively few observations.
- Decision trees partition predictor combinations into regions by recurrent yes/no questions; each leaf predicts the average observed outcome of the training observations in that leaf.
- RF increases information utilization by:
  - Using bootstrap aggregation (“bagging”)—building each individual tree on only a random sample of observations from the training data.
  - At each split, using only a random subsample of the predictors.
- Individual trees tend to have low forecast accuracy, but aggregate RF forecasts perform well because each tree is a partial reflection of training-data information and aggregation reduces variance.
- Figures (illustrative): decision tree example and Random Forest example show trees using different observations and variables; RF predicts the average of the leaves that a new observation belongs to.

*Source: ANNEX II of the content unit.*

### Data and preprocessing (from ANNEX III)
- Data source: mixed frequency (monthly and quarterly) standardized (across countries) leading and coincident indicator data from Haver Analytics.
- Predictor selection strategy: collect as many country-specific and global indicators as possible because Random Forest sifts through many predictors and tends to select the most predictive ones.
- Transformations:
  - Each indicator is transformed twice, deflated where necessary, and 1- through 12-month lags are included.
  - Stationary variables (e.g., capacity utilization, consumer confidence): use the level and quarter-on-quarter difference.
  - Non-stationary variables (e.g., production, money): use first- and second-order log differences.
- Missing value imputation:
  - K-Nearest Neighbor proximity measures with K = 10 neighbors.
  - Uses a Euclidian metric to find the 10 most similar observations and imputes missing values using the median value from these neighbors.
- Pre-selection of indicators:
  - ‘Hard thresholding’ (Bai & Ng, 2008): for each indicator, regress the forecast variable on its lags and the indicator, and select all indicators with an absolute t-statistic above 2.5.
- Indicator sets (as presented in the source):
  - Table 1. Stationary Variables—Level and First Difference: includes Official/policy interest rate; Long term sovereign bond yield; Gross External Debt, percent of GDP; Capacity Utilization; LFS Unemployment Rate; Current Account, percent of GDP; BoP, percent of GDP; Now-Casting Index (NCI); World Uncertainty Index; Composite PMI; Manufacturing PMI; Composite PMI, flash; Composite PMI, manufacturing, flash; Sovereign CDS Spread; JPM Global Composite PMI; JPM Global Manufacturing PMI; US Corporate High Yield; US Federal Funds Effective Rate; US 10-year Treasury Yield; World Uncertainty Index; Sentix Economic Expectations; Sentix Current Economic Situation; CBOE VIX; CBOE 10-year Treasury VIX; Real short rate; Real long rate; Term spread (long rate—short rate); Long dollar spread (long rate—US 10 year yield); Real M2 growth; Real US Federal Funds Rate; Real US 10 year; Real US term spread (10 year yield—FFR); US credit spread (high yield—10 year yield).
  - Table 2. Non-Stationary Variables—First and Second Log Difference (Y: in real terms): includes Nominal ER against USD; Nominal ER against EUR; Nominal Effective ER (38 partners); Real Effective ER (38 partners); M2; Private sector debt (Y); Domestic debt (Y); MSCI Stock Market, total return, USD; Gross ED, nominal, USD; CPI; Core CPI; PPI; EPI; IPI; ToT; Housing Prices (Y); Housing Permits; Industrial Production, total (Y); Industrial Production, manufacturing (Y); Employment; Earnings (Y); Manufacturing Shipments (Y); Retail Sales, value; Retail Sales, volume; Vehicle Registrations; Exports, value (Y); Imports, value (Y); Consumer Confidence; Consumer Expectations; Business Confidence; Tourist Arrivals; Gross Operating Surplus or Corporate Profits (Y); Nominal Final Domestic Demand; Composite CPI for Advanced Economies; HWWI Commodity Price Index; Dallas Fed House Price Index World; World Industrial Production ex Construction CPB; World Trade Volume; Dow Jones Global Index; World WTI Weekly Average Price; BIS Narrow NEER; Dollar; AUD/JPY ER.
- Additional constructed series noted: sovereign term spread, sovereign yield spread, the US sovereign term spread, and the US high yield spread.

*Source: ANNEX III of the content unit.*

### Proximate countries and Random Forest implementation details (from ANNEX IV)
- Random Forest configuration used for proximate-country analysis: Random Forests with 500 trees.
- Data coverage for proximate-country tables: data up to and including 2017.
- Tables present top and bottom 25 countries that are most predictive of growth for each focal country. Examples included in the source:
  - Table 3. Most/Least Similar Countries—Austria, Canada, Costa Rica (each lists [1] through [25] as most similar and [78] through [102] as least similar for each focal country).
  - Table 4. Most/Least Similar Countries—El Salvador, Germany, Iceland (same top/bottom structure).
  - Table 5. Most/Least Similar Countries—Lithuania and Mexico (same top/bottom structure).
- Example specifics preserved from tables:
  - Austria top 5: [1] Austria; [2] Belgium; [3] France; [4] Switzerland; [5] Germany.
  - Canada top 5: [1] Canada; [2] United States; [3] France; [4] Denmark; [5] United Kingdom.
  - Costa Rica top 5: [1] Costa Rica; [2] Japan; [3] Nicaragua; [4] Guatemala; [5] Malaysia.
  - El Salvador top 5: [1] El Salvador; [2] Brazil; [3] Morocco; [4] Mexico; [5] Norway.
  - Germany top 5: [1] Germany; [2] Japan; [3] Netherlands; [4] France; [5] Denmark.
  - Iceland top 5: [1] Iceland; [2] Lithuania; [3] Croatia; [4] Finland; [5] Romania.
  - Lithuania top 5: [1] Lithuania; [2] Finland; [3] Estonia; [4] Bulgaria; [5] Japan.
  - Mexico top 5: [1] Mexico; [2] Germany; [3] Japan; [4] Canada; [5] Hungary.
- Notes: tables present full top and bottom 25 lists for each focal country as generated by Random Forests with 500 trees, based on data up to and including 2017.

*Source: ANNEX IV of the content unit.*

*Source: wpiea2020044-print-pdf - ANNEX I. MACHINE LEARNING AND CROSS VALIDATION (content unit provided).*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2020/english/wpiea2020044-print-pdf.pdf_