## 1. Introduction

## Source details

**Canonical URL:** [1. Introduction](https://www.imf.org/-/media/files/publications/wp/2025/english/wpiea2025172-source-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2025/english/wpiea2025172-source-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2025/english/wpiea2025172-source-pdf.pdf.json)

---

### Motivation and core problem
- Statistical forecasts of economic time series often need constraints imposed while preserving smoothness over time.
- Common issues:
  - Ad hoc constraint imposition (e.g., manually altering only the terminal value) can introduce discontinuities between penultimate and terminal values.
  - Forecasts of aggregate variables and subcomponents (e.g., fiscal balance, revenue, expenditure) may violate accounting identity constraints if reconciled ad hoc (e.g., treating one variable as a residual), causing the residual to absorb others’ forecast errors and reducing smoothness.
- Trade-off: Adjusting forecasts to satisfy constraints often breaks smoothness; enforcing smoothness can break constraints.

### Importance and challenges of systematic treatment
- Constraints originate from accounting identities and expert judgement and are essential for internal consistency.
- Smoothness is critical because optimal forecasts typically exhibit less volatility than historical data (example: random walk optimal forecast is constant and equal to last observed value).
- Manual enforcement is resource-intensive when many variables and constraints exist.
- Central question: How to systematically impose constraints and smoothness across large systems of time series?

### Existing literature and methodological foundation
- Ando (2024) defines a quadratic programming approach that imposes constraints and temporal smoothness in closed form, applicable to large systems; draws on forecast reconciliation literature (reviewed by Athanasopoulos et al. (2024)) and the Hodrick and Prescott (1997) smoothing method.
- Ando (2024) provides replication code and illustrates combining statistical models with the smooth reconciliation method a la Ando and Kim (2023).

### Gaps in existing software
- Reconciliation-only packages:
  - hts (Hyndman et al., 2021) and FoReco (Girolimetto and Di Fonzo, 2023) in R — support reconciliation but reconciled forecasts may not be temporally smooth.
  - hierarchicalforecast (Olivares et al., 2024) in Python — similar limitation.
- Smoothing-only packages:
  - smooth (Svetunkov, 2024) and forecast (Hyndman et al., 2024) for R; statsmodels (Seabold and Perktold, 2010) for Python — provide smooth forecasts but lack constraint-imposition functionality.
- Conclusion: To our knowledge, no package supports simultaneous reconciliation and smoothing.

### Contribution: macroframe-forecast Python package
- Introduces macroframe-forecast, a Python package that generates forecasts that are temporally smooth and meet user-defined constraints.
- Two-step approach a la Ando and Kim (2023) and Ando (2024):
  1. First step (model-agnostic): users generate unconstrained forecasts using any forecasting model (compatible with sktime (Loning et al., 2019)), including machine learning pipelines or econometric methods.
  2. Second step: forecasts undergo reconciliation a la Ando (2024) to enforce equality or inequality constraints and smooth the forecast trajectory over time.
- Contrasts with Chan et al. (2025), who jointly forecast and reconcile but require Gaussian distributions for all variables.
- Benefits of Python implementation: compatibility with rich machine learning packages.

### Package design and user experience
- Plug-and-play simplicity and flexibility: main class MFF lets users operate without explicitly providing mathematical inputs for the reconciliation problem.
- Constraints can be specified using strings rather than matrices.
- Forecasters can fine-tune both steps independently:
  - Use sophisticated machine learning models in first step or supply first-step forecasts exogenously.
  - Experiment with different constraints, inequality constraints, or smoothness parameters in the reconciliation step.
- Use cases: experiment with different target values, inequality constraints, or smoothness before finalizing forecasts.

### Demonstrations and paper structure
- Two demonstrative examples:
  - Single-variable example: U.S. GDP forecasted subject to constraint that growth rate at forecast horizon end matches a predefined value.
  - Multivariable example: Forecasting fiscal variables (revenue, expenditure, interest expense) subject to accounting identity, with the path of the primary balance given exogenously.
- Paper structure:
  - Section 2: conceptual framework of methods implemented in the package.
  - Section 3: instructions for installing and using the package, with illustrative examples.
  - Section 4: conclusion.

### Ordinary Least Squares (OLS) using the first feature — key options and parameters
- Default forecaster when Grid Search cannot run due to too few non-NaN observations:
  - DefaultForecaster() is NaiveForecaster() from sktime.
- equality_constraints: A list of equality constraints expressed as strings.
  - Can bind at a certain period or use wildcards to extend to all forecast horizons.
  - Specified as a list of strings using the column names of df and time periods.
  - If no input is provided, no constraints will be imposed and the second-step forecast will be the smoothed version of the first-step forecast.
  - If an island value is provided in df as in Table 1, users do not need to specify it in equality_constraints.
- inequality_constraints: A list of inequality constraints, analogous to equality_constraints.
- parallelize: A Boolean indicator of whether dask’s parallel computing is utilized for generating forecasts.
- n_forecast_error: The number of time series cross validation splits to generate the first-step forecast error covariance matrix Σ̂.
  - Default: 5.
- shrinkage_method: Method for shrinking the sample covariance matrix estimated from first-step forecasts.
  - Default: Oracle Approximating Shrinkage (OAS) Estimator (Chen et al., 2010).
  - Other options: Oracle Approximating Estimator with Diagonal target (OASD), identity matrix, monotone diagonal method.
- default_lam: The value of lambda 휆i used for smoothing forecasts in the second step.
  - Default: -1 (lambda selected by minimizing mean-squared error from time-series cross validation).
  - When 0, no smoothing is applied.
- max_lam: The maximum value that the lambda parameter 휆i can take if estimated optimally (default_lam = -1).
  - Default: 129600.

### Main outputs and stored attributes
- Main output:
  - df2: A dataframe with the same size as df, with NaN replaced by the forecasts that satisfy specified constraints.
- Additional attributes capturing intermediate steps:
  - df0: The original input dataframe, with all island values set to NaN.
  - df1: A dataframe with first-step forecasts filled in the NaN cells of dataframe df0.
  - df1_model: A dataframe where each cell corresponds to the model used for generating the corresponding forecasted value in df1.
  - pred: A dataframe with predicted values from in-sample predictions generated using pseudo-historical datasets (used to generate first-step forecast errors).
  - true: A dataframe with actual values of variables corresponding to pred. First-step forecast errors = pred − true.
  - model: A dataframe containing the models used for generating in-sample forecasts stored in pred.
  - C_eq: Left side of equality constraint equation C_eq y − d_eq = 0 in matrix form from equation (3).
  - d_eq: Right side of equality constraint equation C_eq y − d_eq = 0 in matrix form from equation (3).
  - C_ineq: Left side of inequality constraint equation C_ineq y − d_ineq ≤ 0 in matrix form from equation (3).
  - d_ineq: Right side of inequality constraint equation C_ineq y − d_ineq ≤ 0 in matrix form from equation (3).
  - W: The weight matrix implied by the first step forecast error.
  - Phi: Smoothing matrix used for generating smooth second-step forecasts.
  - shrinkage: The shrinkage parameter 휌 associated with the second-step adjustment, corresponding to 휌_OAS when OAS is used and 휌_OASD when OASD is used. np.nan when identity option is used.
  - smoothness: A series containing the smoothing variable 휆i* used for each variable i.

### Examples: Single-variable forecasting (US nominal GDP)
- Data setup:
  - US nominal GDP in trillion US dollars from World Economic Outlook database.
  - Historical data: 1950-2024.
  - Forecast horizon: 2025-2030 (rows 2025-2030 are NaN).
- Example df0 fragment:
  - year GDP  
    1950 0.30  
    1951 0.35  
    ... ...  
    2024 28.18  
    2025 NaN  
    2026 NaN  
    2027 NaN  
    2028 NaN  
    2029 NaN  
    2030 NaN
- Constraining GDP growth for 2030 to around 4 percent:
  - Constraint written as: GDP_2030 - 1.04 * GDP_2029
  - MFF call example:
    - m = MFF(df =df0, equality_constraints = ['GDP_2030 - 1.04 * GDP_2029'])
    - m.fit()
  - Outputs:
    - First-step forecasts in m.df1 (may be unsmoothed).
    - Second-step forecasts in m.df2 (satisfy growth-rate constraint and are smoothed by default).
  - If default_lam set to 0, reconciliation occurs without smoothing (growth rates may not be smooth).
- Accessing models used for first-step forecasts:
  - m.df1_model
  - m.df1_model.loc['2028', 'GDP']

### Customizing second-step reconciliation (example workflow)
- Replace first-step forecasts with external forecasts:
  - GDP_forecasts_external = pd.DataFrame({"GDP": [29.0, 31.5, 33, 30.2, 36.8, 39]}, index = [2025, 2026, 2027, 2028, 2029, 2030])
  - y1_exo = m.y1; y1_exo[:] = GDP_forecasts_external['GDP']
- Replace weight matrix with identity:
  - W_exo = m.W; W_exo.iloc[:,:] = np.eye(len(m.y1))
- Halve smoothness parameter 휆∗ and update smoothing matrix Φ:
  - from macroframe_forecast.utils import GenSmoothingMatrix
  - smoothness_exo = m.smoothness * .5
  - Phi_exo = GenSmoothingMatrix(m.W, smoothness_exo)
- Reconcile with user-specified inputs:
  - from macroframe_forecast.utils import Reconciliation
  - alternative_y2 = Reconciliation(y1= y1_exo, W = W_exo, Phi = Phi_exo, C = m.C, d = m.d, C_ineq = m.C_ineq, d_ineq = m.d_ineq)

### Examples: Multivariable forecasting (US fiscal data)
- Variables (expressed as ratios to nominal GDP): exp (expenditure), rev (revenue), int_payments (interest payments), pb (primary balance).
- Identity:
  - Primary balance = revenue – expenditure + interest payments.
- Data setup (April 2025 WEO historical to 2024 and projections for 2025-30):
  - Example rows:
    - year exp rev int_payments pb  
      2001 32.80 32.26 3.26 2.71  
      2002 33.70 29.88 2.89 -0.93  
      2003 34.04 29.27 2.66 -2.11  
      2004 33.72 29.48 2.56 -1.68  
      2005 33.93 30.85 2.70 -0.37  
      ... ... ... ... ...  
      2024 37.59 30.33 4.20 -3.06  
      2025 NaN NaN NaN -2.16  
      2026 NaN NaN NaN -1.06  
      2027 NaN NaN NaN -0.90  
      2028 NaN NaN NaN -1.25  
      2029 NaN NaN NaN -1.14  
      2030 NaN NaN NaN -1.32
- Constraints example:
  - Fiscal identity across all horizons plus an explicit target for expenditure in 2030:
    - fiscal_constraint = ['pb? - rev? + exp? - int_payments?', 'exp_2030 - 37']
    - The question mark “?” is a wildcard that expands the constraint across all forecast horizons.
  - Inequality constraint example for interest payments staying between 2.5 and 3.5 percent:
    - int_constraint = ['int_payments? - 3.5', '2.5 - int_payments?']
- Using a user-defined OLS-based forecaster within sktime:
  - Wrap LinearRegression in DirectReductionForecaster inside a ForecastingPipeline:
    - ols_model = ForecastingPipeline(steps=[('ols', DirectReductionForecaster(LinearRegression()))])
  - Call MFF with user forecaster:
    - m = MFF(df = fiscal_data, equality_constraints = fiscal_constraint, inequality_constraints = int_constraint, forecaster = ols_model)
    - m.fit()
- Observed outcomes in examples:
  - Second-step forecasts are smoother than first-step forecasts.
  - Expenditure forecast in second step equals the target in the equality constraint.
  - Interest payments remain within predefined bounds.
  - First-step forecasts do not satisfy the accounting identity; second-step forecasts do.

### Conclusion and further development directions
- The package separates model estimation from imposition of constraints and smoothness adjustments, allowing any statistical or machine learning models for initial forecasts while ensuring final forecasts are internally consistent and smooth.
- The macroframe-forecast package is a beta version; planned or suggested extensions include:
  - Extending support to mixed-frequency time series.
  - Speeding up parallel processing.
  - Developing an Excel add-in.

*A Python Package to Assist Macroframework Forecasting — Working Paper No. WP/2025/172*

### 1. Introduction

### 1. Introduction

### Motivation and core problem
- Statistical forecasts of economic time series often need constraints imposed while preserving smoothness over time.
- Common issues:
  - Ad hoc constraint imposition (e.g., manually altering only the terminal value) can introduce discontinuities between penultimate and terminal values.
  - Forecasts of aggregate variables and subcomponents (e.g., fiscal balance, revenue, expenditure) may violate accounting identity constraints if reconciled ad hoc (e.g., treating one variable as a residual), causing the residual to absorb others’ forecast errors and reducing smoothness.
- Trade-off: Adjusting forecasts to satisfy constraints often breaks smoothness; enforcing smoothness can break constraints.

### Importance and challenges of systematic treatment
- Constraints originate from accounting identities and expert judgement and are essential for internal consistency.
- Smoothness is critical because optimal forecasts typically exhibit less volatility than historical data (example: random walk optimal forecast is constant and equal to last observed value).
- Manual enforcement is resource-intensive when many variables and constraints exist.
- Question posed: How to systematically impose constraints and smoothness across large systems of time series?

### Existing literature and methodological foundation
- Ando (2024) defines a quadratic programming approach that imposes constraints and temporal smoothness in closed form, applicable to large systems; draws on forecast reconciliation literature (reviewed by Athanasopoulos et al. (2024)) and the Hodrick and Prescott (1997) smoothing method.
- Ando (2024) provides replication code and illustrates combining statistical models with the smooth reconciliation method a la Ando and Kim (2023).
- Gap identified: No user-friendly package implements Ando (2024)’s method.

### Gaps in existing software
- Reconciliation-only packages:
  - hts (Hyndman et al., 2021) and FoReco (Girolimetto and Di Fonzo, 2023) in R — support reconciliation but reconciled forecasts may not be temporally smooth.
  - hierarchicalforecast (Olivares et al., 2024) in Python — similar limitation.
- Smoothing-only packages:
  - smooth (Svetunkov, 2024) and forecast (Hyndman et al., 2024) for R; statsmodels (Seabold and Perktold, 2010) for Python — provide smooth forecasts but lack constraint-imposition functionality.
- Conclusion: To our knowledge, no package supports simultaneous reconciliation and smoothing.

### Contribution: macroframe-forecast Python package
- Introduces macroframe-forecast, a Python package that generates forecasts that are temporally smooth and meet user-defined constraints.
- Two-step approach a la Ando and Kim (2023) and Ando (2024):
  1. First step (model-agnostic): users generate unconstrained forecasts using any forecasting model (compatible with sktime (Loning et al., 2019)), including machine learning pipelines or econometric methods.
  2. Second step: forecasts undergo reconciliation a la Ando (2024) to enforce equality or inequality constraints and smooth the forecast trajectory over time.
- Contrasts with Chan et al. (2025), who jointly forecast and reconcile but require Gaussian distributions for all variables.
- Benefits of Python implementation: compatibility with rich machine learning packages.

### Package design and user experience
- Plug-and-play simplicity and flexibility: main class MFF lets users operate without explicitly providing mathematical inputs for the reconciliation problem.
- Constraints can be specified using strings rather than matrices.
- Forecasters can fine-tune both steps independently:
  - Use sophisticated machine learning models in first step or supply first-step forecasts exogenously.
  - Experiment with different constraints, inequality constraints, or smoothness parameters in the reconciliation step.
- Use cases: experiment with different target values, inequality constraints, or smoothness before finalizing forecasts.

### Demonstrations and paper structure
- Two demonstrative examples:
  - Single-variable example: U.S. GDP forecasted subject to constraint that growth rate at forecast horizon end matches a predefined value.
  - Multivariable example: Forecasting fiscal variables (revenue, expenditure, interest expense) subject to accounting identity, with the path of the primary balance given exogenously.
- Paper structure:
  - Section 2: conceptual framework of methods implemented in the package.
  - Section 3: instructions for installing and using the package, with illustrative examples.
  - Section 4: conclusion.

*IMF Working Papers — 1. Introduction (wpiea2025172-source-pdf)*

### 4. Ordinary Least Squares (OLS) using the first feature.

### 4. Ordinary Least Squares (OLS) using the first feature.

### Key options and parameters
- Default forecaster when Grid Search cannot run due to too few non-NaN observations:
  - DefaultForecaster() is NaiveForecaster() from sktime.
- equality_constraints: A list of equality constraints expressed as strings.  
  - Can bind at a certain period or use wildcards to extend to all forecast horizons.  
  - Specified as a list of strings using the column names of df and time periods.  
  - If no input is provided, no constraints will be imposed and the second-step forecast will be the smoothed version of the first-step forecast.  
  - If an island value is provided in df as in Table 1, users do not need to specify it in equality_constraints.
- inequality_constraints: A list of inequality constraints, analogous to equality_constraints.
- parallelize: A Boolean indicator of whether dask’s parallel computing is utilized for generating forecasts.
- n_forecast_error: The number of time series cross validation splits to generate the first-step forecast error covariance matrix Σ̂.  
  - Default: 5.
- shrinkage_method: Method for shrinking the sample covariance matrix estimated from first-step forecasts.  
  - Default: Oracle Approximating Shrinkage (OAS) Estimator (Chen et al., 2010).  
  - Other options: Oracle Approximating Estimator with Diagonal target (OASD), identity matrix, monotone diagonal method.
- default_lam: The value of lambda 휆i used for smoothing forecasts in the second step.  
  - Default: -1 (lambda selected by minimizing mean-squared error from time-series cross validation).  
  - When 0, no smoothing is applied.
- max_lam: The maximum value that the lambda parameter 휆i can take if estimated optimally (default_lam = -1).  
  - Default: 129600.

### Main outputs and stored attributes
- Main output:
  - df2: A dataframe with the same size as df, with NaN replaced by the forecasts that satisfy specified constraints.
- Additional attributes capturing intermediate steps:
  - df0: The original input dataframe, with all island values set to NaN.
  - df1: A dataframe with first-step forecasts filled in the NaN cells of dataframe df0.
  - df1_model: A dataframe where each cell corresponds to the model used for generating the corresponding forecasted value in df1.
  - pred: A dataframe with predicted values from in-sample predictions generated using pseudo-historical datasets (used to generate first-step forecast errors).
  - true: A dataframe with actual values of variables corresponding to pred. First-step forecast errors = pred − true.
  - model: A dataframe containing the models used for generating in-sample forecasts stored in pred.
  - C_eq: Left side of equality constraint equation C_eq y − d_eq = 0 in matrix form from equation (3).
  - d_eq: Right side of equality constraint equation C_eq y − d_eq = 0 in matrix form from equation (3).
  - C_ineq: Left side of inequality constraint equation C_ineq y − d_ineq ≤ 0 in matrix form from equation (3).
  - d_ineq: Right side of inequality constraint equation C_ineq y − d_ineq ≤ 0 in matrix form from equation (3).
  - W: The weight matrix implied by the first step forecast error.
  - Phi: Smoothing matrix used for generating smooth second-step forecasts.
  - shrinkage: The shrinkage parameter 휌 associated with the second-step adjustment, corresponding to 휌_OAS when OAS is used and 휌_OASD when OASD is used. np.nan when identity option is used.
  - smoothness: A series containing the smoothing variable 휆i* used for each variable i.

### Examples: Single-variable forecasting (US nominal GDP)
- Data setup:
  - US nominal GDP in trillion US dollars from World Economic Outlook database.
  - Historical data: 1950-2024.
  - Forecast horizon: 2025-2030 (rows 2025-2030 are NaN).
- Example df0 fragment:
  - year GDP  
    1950 0.30  
    1951 0.35  
    ... ...  
    2024 28.18  
    2025 NaN  
    2026 NaN  
    2027 NaN  
    2028 NaN  
    2029 NaN  
    2030 NaN
- Constraining GDP growth for 2030 to around 4 percent:
  - Constraint written as: GDP_2030 - 1.04 * GDP_2029
  - MFF call example:
    - m = MFF(df =df0, equality_constraints = ['GDP_2030 - 1.04 * GDP_2029'])
    - m.fit()
  - Outputs:
    - First-step forecasts in m.df1 (may be unsmoothed).
    - Second-step forecasts in m.df2 (satisfy growth-rate constraint and are smoothed by default).
  - If default_lam set to 0, reconciliation occurs without smoothing (growth rates may not be smooth).
- Accessing models used for first-step forecasts:
  - m.df1_model
  - m.df1_model.loc['2028', 'GDP']

### Customizing second-step reconciliation (example workflow)
- Replace first-step forecasts with external forecasts:
  - GDP_forecasts_external = pd.DataFrame({"GDP": [29.0, 31.5, 33, 30.2, 36.8, 39]}, index = [2025, 2026, 2027, 2028, 2029, 2030])
  - y1_exo = m.y1; y1_exo[:] = GDP_forecasts_external['GDP']
- Replace weight matrix with identity:
  - W_exo = m.W; W_exo.iloc[:,:] = np.eye(len(m.y1))
- Halve smoothness parameter 휆∗ and update smoothing matrix Φ:
  - from macroframe_forecast.utils import GenSmoothingMatrix
  - smoothness_exo = m.smoothness * .5
  - Phi_exo = GenSmoothingMatrix(m.W, smoothness_exo)
- Reconcile with user-specified inputs:
  - from macroframe_forecast.utils import Reconciliation
  - alternative_y2 = Reconciliation(y1= y1_exo, W = W_exo, Phi = Phi_exo, C = m.C, d = m.d, C_ineq = m.C_ineq, d_ineq = m.d_ineq)

### Examples: Multivariable forecasting (US fiscal data)
- Variables (expressed as ratios to nominal GDP): exp (expenditure), rev (revenue), int_payments (interest payments), pb (primary balance).
- Identity:
  - Primary balance = revenue – expenditure + interest payments.
- Data setup (April 2025 WEO historical to 2024 and projections for 2025-30):
  - Example rows:
    - year exp rev int_payments pb  
      2001 32.80 32.26 3.26 2.71  
      2002 33.70 29.88 2.89 -0.93  
      2003 34.04 29.27 2.66 -2.11  
      2004 33.72 29.48 2.56 -1.68  
      2005 33.93 30.85 2.70 -0.37  
      ... ... ... ... ...  
      2024 37.59 30.33 4.20 -3.06  
      2025 NaN NaN NaN -2.16  
      2026 NaN NaN NaN -1.06  
      2027 NaN NaN NaN -0.90  
      2028 NaN NaN NaN -1.25  
      2029 NaN NaN NaN -1.14  
      2030 NaN NaN NaN -1.32
- Constraints example:
  - Fiscal identity across all horizons plus an explicit target for expenditure in 2030:
    - fiscal_constraint = ['pb? - rev? + exp? - int_payments?', 'exp_2030 - 37']
    - The question mark “?” is a wildcard that expands the constraint across all forecast horizons.
  - Inequality constraint example for interest payments staying between 2.5 and 3.5 percent:
    - int_constraint = ['int_payments? - 3.5', '2.5 - int_payments?']
- Using a user-defined OLS-based forecaster within sktime:
  - Wrap LinearRegression in DirectReductionForecaster inside a ForecastingPipeline:
    - ols_model = ForecastingPipeline(steps=[('ols', DirectReductionForecaster(LinearRegression()))])
  - Call MFF with user forecaster:
    - m = MFF(df = fiscal_data, equality_constraints = fiscal_constraint, inequality_constraints = int_constraint, forecaster = ols_model)
    - m.fit()
- Observed outcomes in examples:
  - Second-step forecasts are smoother than first-step forecasts.
  - Expenditure forecast in second step equals the target in the equality constraint.
  - Interest payments remain within predefined bounds.
  - First-step forecasts do not satisfy the accounting identity; second-step forecasts do.

### Conclusion and further development directions
- The package separates model estimation from imposition of constraints and smoothness adjustments, allowing any statistical or machine learning models for initial forecasts while ensuring final forecasts are internally consistent and smooth.
- The macroframe-forecast package is a beta version; planned or suggested extensions include:
  - Extending support to mixed-frequency time series.
  - Speeding up parallel processing.
  - Developing an Excel add-in.

*A Python Package to Assist Macroframework Forecasting — Working Paper No. WP/2025/172*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2025/english/wpiea2025172-source-pdf.pdf_