## _wp0566 — Methodological Details ## Source details **Canonical URL:** [_wp0566 — Methodological Details](https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2005/_wp0566.pdf) ## Other formats - [Markdown version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2005/_wp0566.pdf.md) - [Structured JSON version](/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2005/_wp0566.pdf.json) --- ### I. Introduction and summary — methodological framing - Purpose: Compare Canadian central government budget forecasting with benchmark group (most other G-7 countries, plus Australia and New Zealand, and the Netherlands, Sweden, and Switzerland). - Approach: - Structural and quantitative comparison. - Sections II and III: institutional environment and forecasting processes. - Section IV: budgetary forecast outcomes. - Section V: statistical analyses testing for forecast bias and links between institutional characteristics and forecast errors. - Exclusion note: Japanese fiscal policy in the mid- to late 1990s excluded because implementation via supplementary budgets complicates comparisons. ### Major findings on institutional and methodological factors affecting forecasting - Institutional strength and practice: - Canada’s fiscal forecasting is governed by one of the strongest institutional frameworks relative to benchmark countries. - No formal fiscal rule, but a de facto policy of “budget balance or better”. - Budgeting approach: conservative with explicit prudence and contingency factors; strong transparency and accountability. - Explicit use of macroeconomic projections from a wide range of private forecasters is a particular strength. - Fiscal variable forecasts compiled by the Department of Finance with little participation of non-governmental agencies. - Information and transparency: - Canada could improve understanding of budgetary forecasts by providing more information on assumptions and methods used to translate macroeconomic outlooks into fiscal projections. - Quantitative summary: - Since the mid-1990s, Canadian budget projections of macroeconomic and fiscal aggregates have been more cautious than in other countries. - Measures for distance between budget projections and actual outcomes were among the highest within the benchmark group. - Forecast errors for revenue and expenditure aggregates consistently conservative; Canada on average most strongly underestimated its fiscal balance since 1995. - Empirical tests indicate forecast errors are significantly different from zero; stronger-than-expected growth explains a considerable part of fiscal overperformance. - The relatively volatile macroeconomic environment and institutional factors likely contributed to Canada’s conservative forecast bias. ### II. Institutional environment for budget forecasts — methodological implications - Three institutional factors examined: - Distribution of fiscal authority between legislature and executive. - Fiscal relations between central and sub-national governments. - Presence of fiscal rules and other constraints limiting fiscal policy discretion. - Distribution of fiscal authority — observations and Canadian specifics: - If legislature holds substantial fiscal authority, executive forecasts may diverge from enacted measures; executives may produce biased forecasts to influence legislature. - Canada: legislature focuses on optimizing budget process rather than active formulation. - OECD/WB (2003): 19 out of 20 key aspects of Canadian budget process regulated by constitution or law; only the United States scores similarly among benchmark countries. - Canada adheres to ten out of 13 OECD Best Practices in budget reporting (matched only by New Zealand and the United States). - Timing and amendment constraints: - Parliament receives the budget relatively late: <2 months before start of fiscal year; a quarter of fiscal year typically elapsed by approval. - Only about 30−40 percent of spending requires new appropriations; compares to 70–80 percent in the United Kingdom and 90−100 percent in Continental Europe. - Parliament can reduce but not increase funding for line items. - Executive would customarily have to step down if parliament voted against any single aspect of the budget (as in Australia, New Zealand, United Kingdom). - Methodological conclusion: executive-legislative relations unlikely to disproportionately affect Canadian forecast accuracy relative to other countries. - Fiscal relations with sub-national governments — observations and implications: - Structural facts: - Combined, Canada’s sub-national governments are about as large as the central government. - Provinces have high own-source revenue share: 85 percent (including shared tax revenues). - Provinces can determine overall fiscal aggregates and most expenditure allocations; only Sweden and the United States among benchmarks have similar leeway. - Provincial borrowing without federal limits (as in France, Netherlands, New Zealand, Sweden). - Transfers and uncertainty: - Transfers to provinces account for an important share of central government spending; transfers more important in Canada than in other benchmark countries. - Equalization transfers and transfers for health and social spending amount to 1 and 3 percent of GDP, respectively. - Prior to 2004 agreement, equalization transfer payments subject to considerable uncertainty due to statistical revisions of provincial tax bases and population size. - Ex-post adjustments arise from federal government collecting tax revenue for some provinces and the Canada Pension Plan (CPP); collections on behalf of provinces represent about 35 percent of federal revenue. - Gross income and payroll tax revenues are divided on a preliminary basis throughout the year; actual split known after all returns are assessed—usually toward the end of the following fiscal year. - Methodological implication: large and partially uncertain intergovernmental transfers and delayed finalization of revenue splits increase volatility and forecasting challenges. - Fiscal rules and other constraints — observations: - Rules landscape: - Canada has no fiscal policy rules legislated by constitution or law. - “Fiscal Spending Control Act” in force only between 1991 and 1994. - Most advanced countries have adopted some form of rule (overall balance targets, expenditure targets, medium-term frameworks). - De facto rule: - Canada adopted a de facto fiscal rule of budget balance or better; beginning in 1998 authorities defined specific fiscal targets aimed at achieving balance or better. - Political commitment to target produced asymmetric incentives likely influencing forecasting behavior. - Methodological trade-offs: - Fiscal rules may improve discipline and forecast quality, but asymmetric consequences of missing targets may induce prudence (explicit or implicit) in forecasts. - Presence/absence of legislated rules affects monitoring, reporting, and embedding of fiscal plans in medium-term frameworks. ### III. Quantitative methods and definitions used - Time series and notation: - Budget data constructed with notation exemplified in equation (1) (source uses Tttttt xxxx ,...,,,{,},{},{,}=−− 1012). - x denotes projected variables (real GDP growth, tax revenue, fiscal balance); subscript t denotes budget year; superscript denotes year relative to budget year (example: 2 2001 ˆ − y denotes value for real GDP growth in FY 1999 reported in FY2001 budget). - Forecast error definitions (exact formulations in source): - One- and two-year ahead forecast errors: - )log()log( 2 2 00 − + − = ttt xx e - )log()log( 2 3 11 − + − = ttt xx e - Difference between estimated value in base year and actual value reported one year later: - )log()log( 2 1 11 − + − − − = ttt xx e - Logarithmic notation: percentage-point expressions for nominal variables; growth rates in first differences. - Decomposition of nominal GDP forecast error: - One-year nominal GDP forecast error approximation shown in source as: - 0 , ˆ 0 , ˆ 1 , 0 t p t y t Y t Y e e e e − + + = (equation (4)) - Highlights errors in base year nominal GDP and one-year projections of real GDP growth and GDP inflation. - Accuracy metrics: - Mean error (ME) and root mean squared error (RMSE) defined (equation (5)); squared RMSE decomposition: 222 i e ii MERMSE σ += (equation (6)). - Bias tests: - Nonparametric median tests: binomial sign test, Wilcoxon signed ranks test, van der Waerden test. - Mean tests via regression on a constant with Newey-West residuals allowing AR(1) serial correlation. - Efficiency tests: - Regression of actual on constant and projected value (Nordhaus, 1997) shown as equation (7); joint hypothesis α = 0 and β = 1 tested. - Forecast errors tested for autocorrelation. - Comparing budget vs private-sector forecasts: - RMSE difference test following Ashley et al. (1980) (equation (8)). - Encompassing tests (Fair and Shiller, 1990) regress actual on both forecasts (equation (9)); White covariance matrix used for heteroscedasticity. - Notes and measurement conventions: - Forecast error for fiscal balance defined as difference in projected and actual value scaled by average of government revenues and expenditures. - Unemployment and fiscal GDP ratio errors expressed as first differences. - Small sample sizes and structural breaks limit statistical power; tests serve as robustness checks. ### IV. Macroeconomic forecast findings (1995−2003) - Sample focus: 1995−2003 (period when current Canadian forecasting methodology was in force). - Error signs and magnitudes: - Economic growth in Canada on average ½ percentage point higher than budget projections in recent years. - Canadian projections of nominal GDP and real GDP growth show higher RMSEs than most other countries; Canadian mean errors at negative end among benchmarks. - Canadian forecasters underestimated GDP inflation by 0.2 percentage points on average. - One-year unemployment rate forecasts: lower RMSE and (positive) mean error than for other countries. - Canadian forecasters underestimated base year GDP by about one percent on average—the largest negative value in benchmark group. - Macroeconomic prudence adjustment through 1998 budget estimated to account for 0.1 percentage points of mean real growth forecast error, and half as much of mean GDP inflation error. ### V. Fiscal forecast findings and statistical analysis - Fiscal forecast characterization: - Canada among group with relatively weak forecast accuracy (as measured by RMSE); aggregate forecast errors relatively conservative/negatively biased. - Canada records one of largest negative average errors for revenues and one of largest positive average errors for expenditures. - Canada has the largest negative mean error for the overall deficit forecast among benchmark group, even allowing for prudence and contingency factors. - Revenue and expenditure specifics: - Personal income tax and GST/MST projections contributed most to overall revenue forecast error. - Tax revenue subcomponent RMSEs generally not as large as aggregate revenue RMSEs; all subcomponent mean errors for Canada are negative. - Expenditure deviations partly driven by smaller-than-expected debt servicing costs: interest payments on average 2 percent lower than projected, leading to average forecast error of 0.1 percent of GDP. - When expressed as GDP ratios, Canada still shows largest negative mean error versus benchmark group; RMSEs more moderate. - Bias, efficiency, and comparisons: - Bias and efficiency tests (1995−2003): - Tests reject zero mean/median for several Canadian variables: nominal GDP, total and nontax government revenue. - Canada grouped with Germany, New Zealand, Sweden, United Kingdom in exhibiting consistent bias in macro or aggregate fiscal forecasts. - Many countries (Australia, France, Italy, Netherlands, United States) largely free of such findings. - Aggregation of small unidirectional errors drives aggregate bias in Canada: individual tax component mean errors not significant at 10 percent, but aggregate revenue error significant. - Including nominal GDP forecast errors in revenue regressions removes much apparent revenue bias; for Canada the null of unbiased forecasts no longer rejected once nominal GDP errors included—implying measured revenue elasticity close to tax base behavior (measured elasticity among significant countries between 1¼ and 2, with Canada at 1½). - Tests indicate Canadian forecasts may not have used all available information (rejecting forecast efficiency for growth and revenue estimates); Canada exhibits strong autocorrelation in both tax and nontax revenue errors. - Budget vs private sector: - One-year budget forecasts compared with Consensus projections from month of budget release. - Consensus projection errors generally close to government errors; neither dominates across countries. - In Canada, private sector forecasts show slightly smaller RMSE for growth and fiscal forecasts than government forecasts; RMSE equality test rejects equality at relatively high confidence levels for growth though differences minor in absolute terms. - Fiscal forecast differences narrow when underlying budgetary balance (excluding prudence and contingency reserve) used—RMSE differences become statistically insignificant. - Encompassing tests often inconclusive; clear dominance cases: private sector better in Italy and New Zealand (fiscal), government better in France. - Factors affecting forecast errors: - Macroeconomic volatility: - Canada registered the third highest output volatility among benchmark countries between 1990 and 2003. - Short-term interest rates fluctuated relatively strongly; CPI inflation, business wages, and nominal effective exchange rate comparatively stable. - Fiscal aggregates not significantly more volatile than other countries; expenditure-to-GDP ratio volatility higher for Canada, while revenue volatility (as share of GDP) lower than any of the other ten countries (exception: corporate income tax revenue). - Regression findings (bivariate OLS of mean errors and RMSEs on structural and volatility indicators): - Stronger accountability associated with reduced RMSE for growth and tax revenue forecasts; federal structure associated with opposite effect. - Earlier budget presentation (longer lead time) associated with harder-to-forecast revenues. - Weak evidence deficit and expenditure ceilings coincide with conservative revenue estimates. - Fiscal rules associated with overly optimistic forecasts; higher share of voted appropriations similarly associated with optimistic forecasts. - Higher share of mandatory expenditure positively correlated with forecast error for government spending. - Volatility measures: greater real GDP growth volatility pushes growth and revenue forecast errors downward (more pessimistic forecasts) while leaving expenditure forecasts largely unaffected. - Combined regressions: growth volatility and prudence indicators jointly explain mean errors and RMSEs better than structural variables alone. Mean error for Canada’s nominal GDP forecasts close to predicted value given volatility and prudence indicators, but Canada has a large residual for RMSE (second highest), suggesting RMSE less explained by volatility. ### Box 1 — Equalization Transfers in Canada (methodological highlights) - Purpose: reduce disparities in tax-raising capacity between provinces via general purpose block grants. - Revenue-raising capacity: compared per capita to representative provinces (British Columbia, Manitoba, Saskatchewan, Ontario, Québec). - Formula elements (as given in source): Eij, Bj, P, Bij, Pi, τj used in entitlement calculation (exact formula displayed in source). - Size, uncertainty, ex-post adjustments: - Inputs initially based on current-year estimates; revisions (census, final tax data) lead to ex-post adjustments. - Over FY 2000−01 to FY 2003−04, ex-post adjustments ranged between -21 percent to 8 percent of annual transfers, equivalent to a margin of up to 1/8 percent of GDP. - 2004 framework change: - October 2004 announcement: new Equalization framework including a legislated level of overall entitlements starting in 2005-06 with built-in growth rate of 3.5 percent annually. - Timeline of 2000-01 equalization calculation (in billions of Canadian dollars): - Payments through February 2001: Nfld. 1.0; P.E.I. 0.2; N.S. 1.2; N.B. 1.1; Que. 4.3; Man. 1.1; Sask. 0.1; Total 9.0 - Third estimate (February 2001): Nfld. 1.1; P.E.I. 0.3; N.S. 1.3; N.B. 1.2; Que. 5.4; Man. 1.2; Sask. 0.2; Total 10.8 - Fourth estimate (October 2001): Nfld. 1.1; P.E.I. 0.3; N.S. 1.3; N.B. 1.2; Que. 5.4; Man. 1.2; Sask. 0.3; Total 10.8 - Fifth estimate (February 2002): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.2; Man. 1.3; Sask. 0.3; Total 10.8 - Sixth estimate (October 2002): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.3; Man. 1.3; Sask. 0.2; Total 10.9 - Seventh estimate (February 2003): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.3; Man. 1.3; Sask. 0.2; Total 10.9 - Final estimate (September 2003): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.4; Man. 1.3; Sask. 0.2; Total 10.9 ### VI. Policy implications and recommendations (methodological focus) - Diagnosed drivers of Canada’s conservative forecasting: - Macroeconomic volatility and unexpectedly strong outperformance in late 1990s. - Prudence adjustments in late 1990s increasing forecast conservatism modestly. - Institutional factors, including ex-post uncertainty over provincial transfers and tax-sharing arrangements. - Rational behavior under asymmetric costs of missing fiscal targets. - Suggested improvements: - Improve transparency of budgetary forecasts. - Involve private forecasters in producing revenue estimates to broaden information inputs. - Provide more information on critical parts of forecasting process—assumptions and methods used to transform macroeconomic forecasts into fiscal projections—to invite outside scrutiny and potentially improve forecast quality and public confidence. - Noted international examples: Australia and New Zealand (transparency legislation); Germany and Netherlands (involvement of academic bodies or independent agencies). ### Appendix I and Appendix II — data coverage and methodological caveats (high-level points) - Data overview: - Country-specific notes for Australia, Canada, France, Germany, Italy, Netherlands, New Zealand, Sweden, Switzerland, United Kingdom, United States. - Consensus forecasts from Consensus Economics used for private-sector comparisons; drawn from month in which authorities released budget. - Methodological caveats: - Forecasts evaluated against subsequent budget “actuals” to avoid retroactive data revisions. - Time series of consistent forecasts and outcomes often short (often <10 observations), reducing statistical power. - Structural breaks from early/mid-1990s budget format and method changes. - Cross-country comparability limits (revenue coverage differences, nontax revenue heterogeneity, expenditure subcategory mismatch). - Revised mid-year forecasts and outside analyses excluded to preserve consistency. *Source: _wp0566 - 2. Methodological Details (IMF staff working paper content provided).* ### References ........................................................................................................... ### _wp0566 - References .............................................................................................................................42 ### Boxes - 1. Equalization Transfers in Canada ................................................................................11 - 2. Fiscal Forecasting Arrangements in Canada................................................................13 - 3. Forecasting, Performance, and Budget Debate in New Zealand .................................18 ### Tables - 1. Indicators of Relations Between Legislature and Executive .........................................5 - 2. Share of Spending by Sub-National Governments ........................................................7 - 3. Consolidated Central Government Expenditure Shares, 2003 ......................................9 - 4. Fiscal Policy Rules and Transparency Laws ...............................................................12 - 5. Key Institutional Characteristics of the Fiscal Forecasting Process ............................15 - 6. Fiscal Forecasting: Quality Assurance.........................................................................16 - 7. Descriptive Statistics of One-Year Budget Forecast Errors, 1995–2003 ....................24 - 8. Results of Forecast Error Median and Mean Tests ......................................................30 - 9. Results of Efficiency Tests ..........................................................................................32 - 10. Comparing Budget and Consensus Forecasts ..............................................................34 - 11. Potential Factors Affecting Forecast ...........................................................................36 - 12. Volatility of Macroeconomic and Fiscal Variables, 1990–2003 .................................37 - 13. Bivariate Regressions of Error Characteristics on Structural and Volatility Indicators .....................................................................................39 ### Figures - 1. Influence of Sub-National Governments .......................................................................8 - 2. Forecast Errors: Real GDP Growth .............................................................................20 - 3. Forecast Errors: Fiscal Balance ...................................................................................21 - 4. Descriptive Statistics for One- and Two-Year Macroeconomic Budget Forecast, 1995–2003 ..............................................................................25 - 5. Descriptive Statistics for One- and Two-Year Fiscal Budget Forecasts, 1995–2003 .............................................................................27 - 6. Decomposition of Mean Forecast Errors .....................................................................28 - 7. Budget and Consensus One-Year Growth Forecast Errors..........................................33 - 8. Fiscal Balance Forecast Errors.....................................................................................35 - 9. Impact of GDP Volatility on Forecast Quality ............................................................40 ### Appendices - 1. Data Overview .............................................................................................................44 *https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2005/_wp0566.pdf* ### 2. Methodological Details .............................................................................. ### 2. Methodological Details ................................................................................................47 ### I. Introduction and summary — key methodological framing - Purpose: Compare Canadian central government budget forecasting with that of a benchmark group (most other G-7 countries, plus Australia and New Zealand, and the Netherlands, Sweden, and Switzerland). - Approach: Two-pronged — structural and quantitative. Sections II and III compare institutional environment and forecasting processes; Section IV describes budgetary forecast outcomes; Section V presents statistical analyses testing for forecast bias and links between institutional characteristics and forecast errors. - Main methodological note: Japanese fiscal policy in the mid- to late 1990s was excluded from the benchmark group because implementation through supplementary budget requests would complicate comparisons. ### Major findings on institutional and methodological factors affecting forecasting - Institutional strength: - Canada’s fiscal forecasting is governed by one of the strongest institutional frameworks relative to the benchmark countries. - Canada has no formal fiscal rule, but the policy of “budget balance or better” functions as a de facto fiscal target. - Budgeting approach is conservative with explicit prudence and contingency factors, and strong transparency and accountability. - Explicit use of macroeconomic projections from a wide range of private forecasters is a particular strength. - Forecasts of fiscal variables are compiled by the Department of Finance with little participation of non-governmental agencies. - Information and transparency: - Canada could improve understanding of budgetary forecasts by providing more information on assumptions and methods used to translate macroeconomic outlooks into fiscal projections. - Quantitative summary (methodological implications): - Since the mid-1990s, budget projections of macroeconomic and fiscal aggregates in Canada have been more cautious than in other countries. - Measures for the distance between budget projections and actual outcomes were among the highest within the benchmark group. - Forecast errors for both revenue and expenditure aggregates were consistently on the conservative side; Canada on average most strongly underestimated its fiscal balance since 1995. - Empirical tests indicate forecast errors are significantly different from zero; both public and private forecasters were repeatedly surprised by the strength of the Canadian economy and fiscal performance, particularly in the late 1990s. - Stronger-than-expected growth explains a considerable part of fiscal overperformance given the close link between tax revenues and the macroeconomy. - The relatively volatile macroeconomic environment and institutional factors likely contributed to Canada’s conservative forecast bias. ### II. Institutional environment for budget forecasts — methodological implications - Three institutional factors examined: - Distribution of fiscal authority between legislature and executive. - Fiscal relations between central and sub-national governments. - Presence of fiscal rules and other constraints limiting fiscal policy discretion. #### Distribution of fiscal authority — methodological observations - Potential forecast impacts: - If the legislature holds substantial fiscal authority, executive forecasts may diverge from enacted measures, affecting forecast quality. - The executive may have incentives to produce biased forecasts to influence the legislature (e.g., conservative revenue forecasts to limit spending pressures). - Canadian specifics: - Legislature focused on optimizing the budget process rather than active formulation of the budget. - OECD/World Bank survey (OECD/WB, 2003) finds 19 out of 20 key aspects of the Canadian budget process are regulated by the constitution or by law; only the United States achieves a similar score among benchmark countries. - Canada adheres to ten out of 13 OECD Best Practices in budget reporting, a score matched only by New Zealand and the United States. - Timing and amendment constraints that affect forecasting: - Parliament receives the budget relatively late: <2 months before the start of the new fiscal year; a quarter of the fiscal year typically elapsed by the time the budget is approved. - Only about 30−40 percent of spending requires new appropriations (mandatory spending does not require annual funding legislation); this compares to 70–80 percent in the United Kingdom and 90−100 percent in Continental Europe. - Parliament can reduce but not increase funding for line items; generally constrained to approve or reject government spending proposals. - In Canada (as in Australia, New Zealand, and the United Kingdom), the executive would customarily have to step down if parliament voted against any single aspect of the budget. - Methodological conclusion: - Little indication that executive-legislative relations disproportionately affect the accuracy of Canadian budget forecasts relative to other countries. Stringent process rules and reporting requirements are conducive to forecast accuracy. #### Fiscal relations with sub-national governments — methodological observations - Structural characteristics: - Combined, Canada’s sub-national governments are about as large as the central government (Table 2: Share of Spending by Sub-National Governments). - Provinces have high own-source revenue share: 85 percent (including shared tax revenues). - Provinces can determine overall fiscal aggregates and most expenditure allocations; among benchmark countries, only sub-national governments in Sweden and the United States have as much leeway. - Provincial borrowing without federal limits, as in France, the Netherlands, New Zealand, and Sweden. - Transfers and forecasting uncertainty: - Transfers to provinces account for an important share of central government spending; transfers are more important in Canada than in other benchmark countries. - Equalization transfers and transfers for health and social spending are the most important transfers, amounting to 1 and 3 percent of GDP, respectively. - Prior to a 2004 agreement, the amount of equalization transfer payments was subject to considerable uncertainty due to statistical revisions of provincial tax bases and population size. - Ex-post adjustments arise from the federal government collecting tax revenue for some provinces and the Canada Pension Plan (CPP); collections on behalf of provinces represent about 35 percent of federal revenue. - Gross income and payroll tax revenues are divided on a preliminary basis throughout the year, but the actual split is known after all relevant tax returns are assessed—usually toward the end of the following fiscal year. - Methodological implication: - Large and partially uncertain intergovernmental transfers and delayed finalization of revenue splits increase volatility and introduce forecasting challenges for the central government. #### Fiscal rules and other constraints — methodological observations - Rules landscape: - Unlike many comparator countries, Canada has no fiscal policy rules legislated by the constitution or by law (Table 4). - The “Fiscal Spending Control Act” was in force only between 1991 and 1994. - Most advanced countries have adopted some form of rule (overall balance targets, expenditure targets, medium-term frameworks). - De facto rule: - Canada has adopted a de facto fiscal rule of budget balance or better, with performance observed on a relatively stringent basis. - Beginning in 1998, authorities defined specific fiscal targets aimed at achieving budget balance or better; the political commitment to this target produced asymmetric incentives that likely influenced forecasting behavior. - Methodological trade-offs: - Fiscal rules may improve discipline and the quality of budget planning and implementation, but asymmetric consequences of missing targets may induce prudence factors in forecasts (explicit or implicit). - The presence or absence of legislated rules affects monitoring, reporting, and the embedding of fiscal plans in medium-term frameworks — all important methodological components for forecast evaluation. ### Quantitative methods referenced - Statistical analyses described (Section V) test for: - Forecast bias (whether forecast errors are significantly different from zero). - Links between structural characteristics and forecast errors. - Empirical results summarized: - Canada’s forecast errors for revenues, expenditures, and fiscal balance since 1995 were significantly different from zero and biased toward underestimation of the fiscal balance. - Stronger-than-expected GDP growth accounted for a considerable portion of fiscal overperformance. _Italic: Source — _wp0566 - 2. Methodological Details ................................................................................................47_ ### Box 1. Equalization Transfers in Canada ### Box 1. Equalization Transfers in Canada ### Purpose and definition - Equalization transfers are designed to reduce disparities in tax-raising capacity between provinces. - Transfers are provided as general purpose block grants, channeling federal funds to provinces with below-average revenue raising capacity. - "Revenue raising capacity" is defined by comparing per capita revenue raised to the per capita revenue each province could raise if it levied national average tax rates on each source of provincial revenue. - Each province’s revenue raising capacity is compared on a per capita basis to the average of the five middle income provinces: British Columbia, Manitoba, Saskatchewan, Ontario and Québec. ### Formula for total entitlements - Total equalization entitlements are determined as: iiij j jj j ij PPBPBE⋅−= ∑∑ )//(τ where - Eij = entitlement under revenue source j in province i - Bj = the tax base for revenue source j in the representative provinces - P = the population of the representative provinces - Bij = the tax base for revenue source j in province i - Pi = the population of province i - τj = the national average tax rate for revenue source j ### Size, uncertainty, and ex-post adjustments - Inputs to the entitlement formula are initially based on estimates for the current fiscal year. - As data are revised in subsequent years (for example, new census data or final tax revenue data), entitlements are modified and positive or negative ex-post payments are made. - Over the four years between FY 2000−01 and FY 2003−04, the magnitude of ex-post adjustments ranged between -21 percent to 8 percent of annual transfers, equivalent to a margin of up to 1/8 percent of GDP. ### 2004 framework change - In October 2004, the government announced a new Equalization framework. - The framework included a new legislated level of overall Equalization entitlements starting in 2005-06, with a built-in growth rate of 3.5 percent annually. ### Calculation of 2000-01 Equalization Transfers (in billions of Canadian dollars) - Payments through February 2001: Nfld. 1.0; P.E.I. 0.2; N.S. 1.2; N.B. 1.1; Que. 4.3; Man. 1.1; Sask. 0.1; Total 9.0 - Third estimate (February 2001): Nfld. 1.1; P.E.I. 0.3; N.S. 1.3; N.B. 1.2; Que. 5.4; Man. 1.2; Sask. 0.2; Total 10.8 - Fourth estimate (October 2001): Nfld. 1.1; P.E.I. 0.3; N.S. 1.3; N.B. 1.2; Que. 5.4; Man. 1.2; Sask. 0.3; Total 10.8 - Fifth estimate (February 2002): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.2; Man. 1.3; Sask. 0.3; Total 10.8 - Sixth estimate (October 2002): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.3; Man. 1.3; Sask. 0.2; Total 10.9 - Seventh estimate (February 2003): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.3; Man. 1.3; Sask. 0.2; Total 10.9 - Final estimate (September 2003): Nfld. 1.1; P.E.I. 0.3; N.S. 1.4; N.B. 1.3; Que. 5.4; Man. 1.3; Sask. 0.2; Total 10.9 - Source for these calculations: Department of Finance. *Italic: Source: _wp0566 - Box 1. Equalization Transfers in Canada_* ### Box 3. Forecasting Performance and Budget Debate in New Zealand ### Box 3. Forecasting Performance and Budget Debate in New Zealand ### Fiscal framework and legal requirements - The 1994 Fiscal Responsibility Act requires the government to communicate its policy intentions and to quantify the short- and long-term effects of the associated spending and taxation decisions. - The law mandates a continuing review of policy plans and their financial implications, which are assessed against budget plans and actual developments. - The review process is enforced through the publication of two regular reports intended to enrich the budget debate by making the inherent risks to the fiscal forecast more accessible to the broader public. ### Primary budget documents and their roles - The Budget Policy Statement - Specifies the fiscal intentions of the government for the next three years, including strategic priorities and targets for spending, revenue, the fiscal surplus, and public debt. - Requires that policy goals be in line with the responsibility principles set out in the 1994 law. - The Fiscal Strategy Report - Published at the time of the budget—focuses on the quantitative implications of policies contained in the Budget Policy Statement. - Assesses whether the budget is consistent with the longer term policy plans. - Is required to identify deviations between the projected implications under previous policy plans and their original intentions. ### Effects on forecasting practices and public accountability - By requiring separate statements on overall policy goals and their fiscal implications, the public is better positioned to assess the government’s track record in meeting its fiscal goals. - Mandatory evaluations of the consistency between long-term goals and short-term plans have put greater emphasis on forecast accuracy and on the forecasting process. - With deviations of fiscal outturns from projections subject to greater scrutiny: - Information about sources of forecast errors is being disclosed. - The government has commissioned regular external and internal reviews of forecasting processes and methods. ### Assessing forecast accuracy: cross-country data limitations and recent comparisons - Data problems generally limit the analysis of fiscal forecasting performance across countries. - Most prior analyses have focused on macroeconomic forecast accuracy of private sector economists and international organizations (Artis, 1996; Artis and Marcellino, 2001; Ash, et al., 1998; Batchelor, 2001; Isiklar, et al., 2004, Loungani, 2000; Öller and Barot, 2000). - Most analyses of budget projections have focused on a single country, given difficulties in obtaining a cross-country data set of budget forecasts. - Two more recent studies analyzed budgetary forecasts for a group of relatively homogenous countries (euro zone members): - One study suggested that the size of forecast errors may depend on structural characteristics of a country’s budgetary framework (Strauch, et al., 2004). - The other called for independent budget forecasting agencies on the basis of significant forecast biases (Jonung and Larch, 2004). - Information obtained for this study provided sufficient detail to compare Canadian central government budget forecasts with benchmark countries in recent years. - Typical budget reporting practices noted: - At a minimum, most budgets provide 3−4 years of information for key macroeconomic and fiscal variables, including actual or estimated values for the preceding year, an estimate or projection for the current, and projections for one or two future fiscal years. - Most budgets are compiled near the beginning of a new fiscal year, so the values of economic and fiscal variables reported for the prior year are generally at or close to their final revision. This allows the use of historical data reported in the budget as basis for comparison with projections contained in earlier budgets. *Source: Box 3. Forecasting Performance and Budget Debate in New Zealand (excerpt).* ### Appendix I, and methodological issues are covered in Appendix II. ### _wp0566 - Appendix I, and methodological issues are covered in Appendix II. ### Methodology and data caveats - Budget projections are evaluated against subsequent budget “actuals” to avoid using revised historical data that were not available to forecasters at the time; data losses are limited to at most 2−3 observations around the time a revision was introduced. - Advantages of this approach: - Avoids retroactive application of data revisions to measure projection accuracy. - Focuses on information available to forecasters and relevant to expectation formation. - Key data limitations noted: - Time series of consistent forecasts and budget outcomes are relatively short (often with less than 10 observations), reducing statistical power. - Many countries updated budget formats and forecasting methods in the early to mid-1990s, producing structural breaks. - Cross-country comparability limits: - Revenue coverage differences (e.g., inclusion of social insurance contributions as government revenues in some countries). - Nontax revenue sources differ significantly (asset sales, royalties, frequency spectrum fees, etc.). - Expenditure subcategory comparisons (discretionary vs. mandatory; transfers to other government levels) are difficult or approximated. - Internal consistency and structural break checks eliminated many data points, but only obvious outliers were removed due to limited institutional knowledge. - Revised mid-year forecasts and outside analyses (e.g., Economic and Fiscal Update, convergence programs, U.S. Congressional Budget Office) are excluded to preserve consistency, recognizing that excluding mid-year policy shifts can make some forecast “errors” policy-driven rather than forecaster responsibility. ### Macroeconomic forecast findings (1995−2003) - Sample focus: 1995−2003 to match period when current Canadian forecasting methodology was in force. - Errors defined as projected minus actual values; negative implies outcome exceeded expectations. - Main findings: - Economic growth in Canada has on average been ½ percentage point higher than budget projections in recent years. - Canadian projections of nominal GDP and real GDP growth show higher RMSEs than in most other countries; Canadian mean errors are at the negative end among benchmark countries. - Decomposition indicates the large RMSE is mostly a function of a large mean error rather than large standard deviation of errors. - Canadian forecasters underestimated GDP inflation by 0.2 percentage points on average. - Short-term unemployment trends were anticipated quite well; one-year unemployment rate forecasts exhibited a lower RMSE and (positive) mean error than for other countries. - Canadian forecasters underestimated base year GDP by about one percent on average—the largest negative value in the benchmark group. - Macroeconomic prudence adjustment through the 1998 budget is estimated to account for 0.1 percentage points of the mean real growth forecast error, and for half as much of the mean GDP inflation error. ### Fiscal forecast findings - Overall characterization: Canada among group with relatively weak forecast accuracy (as measured by RMSE); aggregate forecast errors are relatively conservative/negatively biased. - Specifics: - Canada records one of the largest negative average errors for revenues, and one of the largest positive average errors for expenditures. - Taken together, Canada has the largest negative mean error for the overall deficit forecast among the benchmark group, even allowing for economic prudence and contingency factors. - On the revenue side, personal income tax and GST/MST projections contributed most to overall forecast error. - For tax revenue subcomponents, Canadian RMSEs are generally not as large as aggregate revenue RMSEs; however, all subcomponent mean errors for Canada are negative—an accumulation of small but persistently negative errors produces the conservative aggregate outcome. - Expenditure deviations partly driven by smaller-than-expected debt servicing costs: interest payments were on average 2 percent lower than projected, leading to an average forecast error of 0.1 percent of GDP. - When errors are expressed as GDP ratios, Canada still shows the largest negative mean error versus benchmark group, although RMSEs are more moderate. ### Statistical analysis: bias, efficiency, and comparisons - Bias and efficiency tests (see Appendix II for methods) for 1995−2003: - Tests reject zero mean/median for several Canadian variables: nominal GDP, total and nontax government revenue. - Canada grouped with Germany, New Zealand, Sweden, and the United Kingdom in exhibiting consistent bias in macro or aggregate fiscal forecasts. - Many countries (Australia, France, Italy, Netherlands, United States) largely free of such findings. - Aggregation of small unidirectional errors drives aggregate bias in Canada: individual tax component mean errors were not significant at 10 percent, but aggregate revenue error was significant. - Including nominal GDP forecast errors in regression tests removes much apparent revenue bias; for Canada the null of unbiased forecasts was no longer rejected once nominal GDP errors were included, implying a measured revenue elasticity close to tax base behavior (measured elasticity among significant countries between 1¼ and 2, with Canada at 1½). - Tests indicate Canadian forecasts may not have used all available information (rejecting forecast efficiency for growth and revenue estimates); Canada exhibits strong autocorrelation in both tax and nontax revenue errors. - Budget vs private sector forecasts: - One-year budget forecasts were compared with Consensus projections from the month of the budget release. - Consensus projection errors are generally close in magnitude to government budget errors; neither public nor private forecasters systematically dominates across countries. - In Canada, differences are relatively small: private sector forecasts show slightly smaller RMSE for growth and fiscal forecasts than government forecasts; test of RMSE equality rejects equality at relatively high confidence levels for growth, though differences are minor in absolute terms. - Fiscal forecast differences narrow once the underlying budgetary balance (excluding prudence and contingency reserve) is used—RMSE differences become statistically insignificant. - Encompassing tests yield often inconclusive results; clear dominance cases: private sector better in Italy and New Zealand (fiscal), government better in France. - Factors affecting forecast errors: - Canada experienced greater macroeconomic volatility than many benchmark countries: - Canada registered the third highest output volatility among benchmark countries between 1990 and 2003. - Short-term interest rates also fluctuated relatively strongly; CPI inflation, business wages, and nominal effective exchange rate were comparatively stable. - Fiscal aggregates were not significantly more volatile than in other countries; expenditure-to-GDP ratio volatility was higher for Canada, while revenue volatility (as share of GDP) was lower than any of the other ten countries (exception: corporate income tax revenue). - Bivariate OLS regressions of mean errors (MEs) and RMSEs on structural and volatility indicators find relatively few significant relationships overall, but notable results include: - Stronger accountability associated with reduced RMSE for growth and tax revenue forecasts; federal structure associated with opposite effect. - Earlier budget presentation (longer budget lead time) associated with harder-to-forecast revenues (possibly influenced by U.S. lead time coincidence). - Weak evidence that deficit and expenditure ceilings coincide with conservative revenue estimates. - Fiscal rules associated with overly optimistic forecasts; higher share of voted appropriations similarly associated with optimistic forecasts. - Higher share of mandatory expenditure positively correlated with forecast error for government spending. - Volatility measures show stronger explanatory power: greater real GDP growth volatility pushes growth and revenue forecast errors downward (more pessimistic forecasts) while leaving expenditure forecasts largely unaffected. - Combined regressions: growth volatility and prudence indicators jointly explain mean errors and RMSEs better than structural variables alone. Mean error for Canada’s nominal GDP forecasts is close to predicted value given its volatility and prudence indicators, but Canada still has a large residual for RMSE (second highest), suggesting RMSE less explained by volatility. ### Policy implications and recommendations - Canada has followed a cautious forecasting approach in recent years, producing small but consistently one-sided fiscal subcomponent errors and an aggregate conservative bias. - Drivers include: - Macroeconomic volatility and an unexpectedly strong outperformance in the late 1990s that forecasters underestimated. - Prudence adjustments in the late 1990s that increased forecast conservatism modestly. - Institutional factors including ex-post uncertainty over provincial transfers and tax-sharing arrangements. - Rational behavior in a regime where costs of missing fiscal targets were high and asymmetric during the period studied. - Suggested improvements (as discussed in the paper): - Improve transparency of budgetary forecasts. - Involve private forecasters in producing revenue estimates to broaden information inputs. - Provide more information on critical parts of the forecasting process—assumptions and methods used to transform macroeconomic forecasts into fiscal projections—to invite outside scrutiny and potentially improve forecast quality and public confidence. - Examples noted in the study: Australia and New Zealand have adopted transparency legislation; Germany and the Netherlands involve academic bodies or independent agencies in forecasts. *Source: Staff calculations.* ### References ### _wp0566 - References ### Data Overview (Appendix I) - Australia - Annual budgets are usually presented in May, two months before the start of the fiscal year in July. Forecast data begins with the 1984/85 budget. - Budgets present activities of the general government, which includes central, state/territory and local governments. - Beginning in the 1999/00 fiscal year, Australia moved from a cash to an accrual accounting basis, but subsequent budgets reported most items on both a cash and accrual basis. For the sake of consistency, the data set uses cash forecasts for all fiscal variables, except interest expenses which from 1999/00 to 2004/05 were only available on an accrual basis. In FY 1999/00 and FY 2000/01, individual, corporate, indirect, and other taxes are omitted from the data set because they were not reported on a cash basis. - Fiscal years 1984/85 through 1993/94 did not report revenue projections beyond the budget year, i.e. two-year projections are omitted. Projections for real GDP growth and unemployment are also limited to the next fiscal year. - Final outcomes for FY 1996/97 were not reported in the FY 1998/99 budget and had to be substituted with estimates reported in the FY 1997/98 budget. - Canada - Data were provided in electronic form by the Department of Finance. Canadian budgets are usually published in February, two months before the start of the fiscal year on April 1. - Projections for FY 2000-01 come from the Budget Update for FY 1999-2000, which was published in October 2000. - Mandatory expenditures includes transfer payments; discretionary expenses are defined as program costs. - Actual outcomes are generally taken from annual financial reports of the government. Annual financial reports are published sufficiently long after the close of the fiscal year to properly estimate accruals transactions. - France - Data were provided in electronic form by French national authorities. French budgets are usually published in September, with the fiscal year starting on January 1. - Forecast data begins with FY 1996. Personal income, corporate income, excise and other tax revenue data are not available for FY 1996 and FY 1997. - Germany - German budgets are published in September, with the next fiscal year starting on January 1. - Forecast data begins with FY 1990. Variables directly affected by the 1990 reunification have been omitted. - Data on mandatory expenditures comprises government wages and salaries and transfer payments. Discretionary expenditures include acquisition of goods and services and capital spending. - Italy - Italian budget proposals are published in the “Documento di Programmazione Economico-Finanziaria” (DPEF) between May and July, half a year before the start of the next fiscal year in January. Data provided by the national authorities reached back to FY 1989. - Personal and corporate income, excise, and other tax revenue data are not available. - Central government (“Bilancio”) data for FY 2000 and FY 2001 were not available. - For FY1990 - FY1998, DPEFs did not report final outcomes for either fiscal or macroeconomic variables, so estimated outcomes from the previous budget are used as the final outcomes. - Netherlands - Data were provided in electronic form by Dutch national authorities. Dutch budgets are published in September, with the fiscal year starting on January 1. - Forecast data begins with FY 1995, and covers general government. - Most projections were limited to the one-year time frame. - New Zealand - New Zealand publishes its “Budget Economic and Fiscal Update” (BEFU) in May, prior to the start of the fiscal year on July 1. Growth and unemployment data were pulled directly from BEFU documents; all other observations came from: http://www.treasury.govt.nz/fiscaldata/default.asp. - Projection data was available for fiscal years 1994/95 through 2004/05, except for growth and unemployment projections which begin with FY 1998/99. - Sweden - Outcome and projection data for Sweden were taken from “Appendix 2: Svensk Economi” of the annual budget bill. The bill is published in September, four months prior to the start of the next fiscal year on January 1. Data were available for FY 1997 through FY 2005, with the exception of FY 2000. - Revenues and the fiscal balance were provided on a general government basis. Budgetary expenditure is on a central government basis. - Data for personal income, corporate income, excise and other tax revenue were not available for FY 1997 and FY 1998. - Switzerland - Data were provided in electronic form by Swiss national authorities. Swiss budgets are published in October, with the fiscal year beginning on January 1. - Forecast data begin with FY 1990. - Data for personal income, corporate income, excise and other tax revenues were not available. - United Kingdom - The U.K. government usually publishes its “Budget Report” in March, shortly before the start of the fiscal year in April. Data only covers budgets published under the current framework since FY 1997/98. - The “Budget Report” refers primarily to the public sector, although general government aggregates are shown for most years. - The current U.K. fiscal framework separates the current and capital budget. For consistency purposes, current and capital expenditures were consolidated. Total outlays are the sum of current expenditure and net investment. - The headline balance concept used was “Net borrowing” inclusive of net windfall tax receipts and associated spending (WTAS), asset sales and depreciation. - United States - Federal government data was obtained from “Historical Tables: Budget of the U.S. Government”, which is usually published in February, 8 months before the start of the next fiscal year on October 1. - Interest expense is recorded on a net basis. - For FY 1984/85 through FY 1990/91, mandatory spending was defined to as “total, relatively uncontrollable outlays” and discretionary spending as “total, relatively controllable outlays.” - Prior to FY 1990/91, nominal output is reported as gross national product. Beginning with FY 1990/91, nominal output is reported as gross domestic product. - Consensus forecasts - Private sector forecast data for real GDP growth rate (calendar year basis) and the headline budget deficit value (in local currency) come from Consensus Economics, Inc. Consensus Economics publishes updated estimates for the current and next calendar/fiscal year every month. The data for this study are drawn from the month in which authorities released their budget documents. ### Methodological Details (Appendix II) - Time series construction - Budget data were used to create time series using the notation: - Tttttt xxxx ,...,,,{,},{},{,}=−− 1012 (equation (1) in source) - x stands for variables projected in budget documents (e.g., real GDP growth, tax revenue, or the fiscal balance). The subscript t denotes the budget year and the superscript denotes a year relative to the budget year. Example: 2 2001 ˆ − y denotes the value for real GDP growth in FY 1999 reported in the FY2001 budget. - Forecast error definitions - One- and two-year ahead forecast errors: - )log()log( 2 2 00 − + − = ttt xx e (equation (2)) - )log()log( 2 3 11 − + − = ttt xx e - Difference between the estimated value of a variable in the base year and the actual value reported one year later: - )log()log( 2 1 11 − + − − − = ttt xx e (equation (3)) - The logarithmic notation implies that projection errors for nominal variables are expressed in percentage points of actual outcomes, and errors for growth rates in first differences. - Decomposition of nominal GDP forecast error - The one-year nominal GDP forecast error approximation: - 0 , ˆ 0 , ˆ 1 , 0 t p t y t Y t Y e e e e − + + = (equation (4)) - Highlights that errors in estimating base year nominal GDP (e-1) and one-year projection errors of real GDP growth and GDP inflation (0, ˆ t y e and 0, ˆ t p e) affect the one-year forecast. - Accuracy metrics - Countries’ budget projections are compared using mean error (ME) and root mean squared error (RMSE): - 2/1 2 ,...,1,...,1 /1,/1 ⎟⎟⎠⎞ ⎜⎜⎝⎛ == ∑∑ == Tt i ti Tt i ti eTRMSE eTME (equation (5)) - The squared RMSE decomposition: 222 i e ii MERMSE σ += (equation (6)) - Bias tests - Bias (nonzero median or mean error) tested with three nonparametric median tests: - binomial sign test (proportion above and below zero equals one-half) - Wilcoxon signed ranks test (sum of ranks of absolute error sizes similar for subsamples above/below zero) - van der Waerden test (variation of Wilcoxon using normal quantiles) - Mean tests via regression of errors on a constant; Newey-West residuals used allowing for AR(1) serial correlation. - Efficiency tests - Regression of actual on a constant and projected value: - 20 2 log()log( ) ttt xx α β ε − + = ++ (equation (7)) - Joint hypothesis tested: α = 0 and β = 1 (Nordhaus, 1997). - Forecast errors also tested for autocorrelation. - Comparing budget vs. private sector forecasts - Difference in RMSEs tested following Ashley et al. (1980) with regression: - ttt ε βα + Σ − Σ += ∆ )( (equation (8)) - ∆t is the difference of government and private sector forecast errors in budget year t; Σt is their sum; the difference is significant if a Wald test rejects α = β = 0. Note: distribution of Wald statistic is nonstandard with serial correlation; p-values may be at most about half normal values for one-sided test once sign of mean errors is established. - Encompassing tests (Fair and Shiller, 1990) regress actual on both forecasts: - t C ttt xxx ε β β α ++= − + ,0 1 0 0 2 2 (equation (9)) - Coefficients β0 and β1 measure information content of government and consensus forecasts; insignificance of β0 with significance of β1 implies consensus forecast encompasses budget forecast. - Regressions run with White covariance matrix to account for possible heteroscedasticity. - Notes and measurement conventions - The logarithmic formulation means forecast error for fiscal balance is defined as the difference in projected and actual value, scaled by the average of government revenues and expenditures. Forecast errors for the unemployment rate and fiscal GDP ratios are expressed as first differences. - Remarks on limited sample sizes and statistical power: tests relying on complex distributional assumptions serve primarily as robustness checks. *Source: _wp0566 - References (IMF PDF content provided).* --- _Source: https://www.imf.org/-/media/websites/imf/imported-full-text-pdf/external/pubs/ft/wp/2005/_wp0566.pdf_