Alfred Kammer's Remarks at the ECB Forum on Central Banking: "Lessons from recent experiences in macroeconomic forecasting"
June 28, 2023
Check out the accompanying slides from Alfred Kammer.
Thank you very much, it’s a pleasure to be here and discuss some of the common challenges we have faced in trying to make sense of developments and project the economy over the past few years.
I will structure my remarks around three themes – first, the challenge of macroeconomic forecasting and how we approach it at the IMF. Then I’ll explain how we adapted our forecasting methods during the uncertain and volatile periods of a pandemic and energy crisis. Finally, I will conclude with a few thoughts on what lessons we can take forward. In my view, these are to (i) be balanced—emphasize top-down forecasting approaches when common forces are strong, but allow for idiosyncrasies as well; (i) be nimble—be ready to continuously develop and add new tools; and (iii) be modest—focus on avoiding misses that would cause major policy mistakes rather than marginal variation around the modal forecast.
With that, let me start by reminding ourselves that macroeconomic forecasting is difficult in the best of times. Economies are complex social systems as we all know. At the IMF we occasionally take stock of how well our forecasts perform and compare our performance with other major forecasters. We tend to find that at relatively short horizons (same year mainly) projections tend to do reasonably well. But moving out to even a 1.5 to two-year horizon both our and others’ forecasts deteriorate very fast (Celasun et al., 2021). The reason is of course that economies suffer constant shocks. Another standard insight into growth forecast accuracy is that the higher the volatility of GDP in a country, the larger the forecast errors. [once we control for GDP growth volatility, our forecasts for advanced economies are no better than those of low-income countries, for example].
The last few years have of course brought us big shocks, making forecasting particularly challenging. Lots of the standard macroeconomic relationships broke down following the pandemic. Think about Okun’s law, the link between unemployment and output which weakened due to job retention schemes, and possibly an increased need or preference to work fewer hours per week. Or think about the “famous” excess savings of the past years. Essentially, they represent a breakdown of the historically tight relationship between household disposable income and consumption as the left chart shows. Another challenge over the past year has been the massive volatility in energy markets since the Russian invasion of Ukraine . The right chart shows the breakdown of another relationship – that between European oil and gas prices in 2022.
What does all this imply for forecasting? One question is the tradeoff between focusing on country specific factors versus common forces.
What do I mean by that? Our main outlet for projections at the IMF is the World Economic Outlook (WEO), published quarterly. The WEO is a bottom-up exercise, aggregating forecasts for our member countries. 190 country teams prepare forecasts based on common assumptions for key global variables--such as commodity prices and benchmark interest rates—over the forecast horizon. As we go along the forecasting round, we perform many checks to ensure cross-country consistency in how these assumptions are incorporated. Nonetheless, country teams have substantial discretion, including in terms of deciding how much their economy is affected by a common factor, embedding country specific information, and the tools and models they use.
Giving country experts room to incorporate idiosyncratic factors is crucial, because domestic factors tend to be more important than foreign ones for the path of economic activity – on average. But it is also true that when we do ex-post evaluations, we find that common components are not sufficiently taken into account as they could be. For instance, Celasun et al 2021 find that growth projections for the US, the euro area, or China, or countries own terms of trade forecasts, can all help predict growth forecast errors for a significant share of countries. If we used all available information efficiently, there would be no such correlation, and our forecasts errors would be smaller.
Especially during times of large global or regional shocks, the common factor can be dominant enough to justify a more top down approach to forecasting.
As the chart shows, forecast errors for individual countries are usually broadly centered around 0 with both positive and negative surprises. But both during the GFC and pandemic, forecast errors were both very large and one-sided, highlighting the vast dominance of the common shock
To give a concrete example, during the early months of the pandemic it became clear that the expected progression of infections, mobility restrictions, and the sensitivity of output to mobility would dominate the immediate outlook. Translating these qualitative factors into a quantitative forecast faced three challenges.
First, in March 2020, when our Spring WEO forecast was being finalized, reported infections were still clustered in just a few countries. The path of infections was uncertain, though we knew this was a highly contagious virus and likely to spread rapidly.
Second, it was unclear how severe the mobility restrictions would be in different countries.
And third, there was little historical data to extrapolate how economic activity would respond to such restrictions.
We decided to take a centralized approach. Our Research Department engaged with epidemiologists and public health experts and then provided centralized guidance to teams on how many effective working days would be lost in the coming quarters. Depending on economic structures (for instance, the contact-intensive share of activity), country teams translated the lost days into GDP declines. Adding to these domestic disruptions, country teams factored in the impact of international demand and supply spillovers. Finally, based on available policy space, they factored in some offsetting policy support. Overall, this approach did quite well, with the “same year” forecast we made in the Spring of 2020 for the euro area being off only by about 1 percentage point. Not a large error given how massive the shock was.
Let me briefly turn to the elephant in the room, the repeated underestimation of inflation over the past 1.5 years, which then points to the need to be nimble. Of course, the war and the pandemic supply shocks were unpredictable. But even after the shocks materialized, it was challenging to quantify their effects on inflation. Forecast errors were very large.
Let me highlight two specific shortcomings of our models and how we tried to improve them.
The first issue is about the right amount of granularity. Most of our forecasting was previously done using international oil prices as a proxy for overall energy prices. This worked very well in the past. But as I showed you earlier, gas prices decoupled significantly from oil prices when Russia cut gas flows to Europe.
Not allowing gas prices to enter inflation projections separately thus was a problem for projecting energy inflation and ultimately the passthrough from energy to core inflation. And this is just one example of models needing some amendment—quickly—to deal with a new situation.
In response, we now project inflation in a much more disaggregated way (see, for example, McGregor and Toscani (2022)). As the red dashed line in the chart shows, adjusting the model structure of a Phillips curve based inflation projection framework and estimating it on data until 2019 produces significantly better out of sample forecasts than the previous model. But at the same time, that model still misses the most recent inflation surge by a meaningful margin. [Note that what I am showing is quite a demanding exercise, being a pseudo out of sample forecast from 2020q1 until 2023q1 – a highly unusual period]
The second key shortcoming - much harder to correct – has to do with non-linearities. Whereas a firm might not react strongly to a 20 percent increase in the prices of one of its minor inputs, it does have to adjust its prices given a 500 percent increase. And models missed the mark because such movements were well out of the range of the data the model was trained on. We found out that it is challenging to gauge such nonlinearities in real time.
So forecasting is a humbling process. What lessons can we apply going forward?
- First, after the experience of the past years, some enhanced role for top down guidance will probably stay with us at the IMF relative to pre-pandemic times. That helps react in real time to important developments, which in a possibly more shock prone World could prove important. But we need to be balanced—idiosyncratic factors are also key.
- Second, we will need to be nimble – it is clear that we need to continuously monitor and enhance our tools. Part of the way forward is also to exploit underused data sources and incorporating new data, including big data in a flexible way.
- Finally, we should be modest about forecasting. It is worthwhile also reminding ourselves that macroeconomic data remains difficult to collect, and the key series we rely on – notably GDP – are prone to large revisions. De-emphasizing point estimates and focusing on forecast ranges thus seems important, especially when forecasting informs policy making. We should focus on avoiding forecast errors that would yield gross policy mistakes rather than worrying excessively over marginal changes in the modal forecast. This also means scenario analysis has a clear role to play – allowing both to implement policies to avoid a downside scenario, and do contingency planning on how to react should it materialize.
To conclude, let me zoom back on today’s forecasting challenge. A key question now that inflation has peaked, is whether we can trust the disinflation paths that come out of most model forecasts given the dissipation of supply shocks and tightening in monetary policy. We have high uncertainty on the level of slack in the economy, many of the standard relationships are not yet fully normalized, and wage and price inflation exceed the range on which we have estimated our models. We also know from past evaluations that we tend to overcompensate for large overpredictions of growth by then turning overly pessimistic. While our recent experiences make it right to question the smooth disinflation paths coming out of our models, we also need to avoid turning overly pessimistic.
So it is important to always be aware of the potential shortcomings of tools, and use several of them jointly, and cross check them. For instance, in addition to predicting inflation components in a disaggregated way, we now also look at prices from the “income” side, examining the wage growth, import prices, and profit share behavior that is consistent with our projections (see Hansen et al., 2023).
I thus see an active role for us as forecasters, with a constant interaction between existing tools, new and refined tools and data sources and ultimately well-grounded to bring in considerations not embedded in the models.