The Structure of the Oil Market and Causes of High Prices

Prepared by Pelin Berkmen, Sam Ouliaris, and Hossein Samiei
Research Department
(In consultation with other Departments, in particular the International Capital Markets Department)
Approved by Raghuram G. Rajan

September 21, 2005


  1. Introduction
  2. Do Crude Oil Prices Reflect the Fundamentals?
  3. Futures Markets and the Role of Speculation
  4. What is the Impact of Refining Constraints?
  5. Concluding Remarks


  1. Spot and Futures Prices
  2. Spread Between the Arab Super Light and Arab Heavy Crude Oil Spot Price
  3. U.S. Crude Oil and Gasoline Spot Prices
  4. Short-Dated NYMEX Open Interest
  5. 6-Year NYMEX Open Interest
  6. The World and United States Refinery Utilization

I. Introduction

The average petroleum spot price (APSP)1 rose by about 31 percent during 2004 and by a further 50 percent in 2005 to reach a record high of US$65 in late-August (Figure 1). In contrast to previous episodes of oil price hikes, longer-dated future prices have also surged and become quite sensitive to daily movements in spot prices. Moreover, the price differential between light-sweet and heavy-sour crude oil has risen to historical highs, while, as in the past, petroleum product prices have increased broadly in line with crude oil prices (Figures 23).

Fig 1

Fig 2

Fig 3

Although many factors have contributed to higher crude oil prices, a combination of strong (and somewhat unexpected) global demand for oil since 2003 and expectations of continuing future tightness is the major cause. These demand/supply imbalances reflect robust global activity, an apparent shift in the demand for oil by China2 and other emerging economies, and limited investment in the oil sector in the past two decades. Naturally, given the tightness in the oil market and uncertainties about demand and supply, factors such as geopolitical developments, fears of potential supply disruptions, and speculation have also all played a part in price movements, but largely through their impact on expectations regarding future fundamentals. Refinery bottlenecks have put additional pressures on petroleum product prices—as demonstrated by the significant rise in gasoline prices following the 10 percent reduction in U.S. refinery capacity caused by Hurricane Katrina.3

This note examines how crude oil, futures, and petroleum product markets interact to determine market outcomes. It discusses: (a) the structure of the global crude oil market and the fundamental forces behind the higher prices; (b) the futures market, and the role of new entrants and speculators; and, (c) refinery bottlenecks, and the relationship between crude and wholesale gasoline prices.

II. Do Crude Oil Prices Reflect the Fundamentals?

On the supply side, the main players in the crude oil market are OPEC—which currently provides about 40 percent of world supply and holds about 70 percent of proven reserves—and non-OPEC producers. OPEC, as the marginal supplier, behaves as a semi-cartel in normal times by aiming to maintain excess extraction capacity in order to influence crude oil prices. In recent years, its policy has been to balance the market while allowing for an 'appropriate' level of crude oil inventories in consuming nations. Non-OPEC producers, on the other hand, have relatively limited reserves and spare capacity, and generally behave as price takers.

Under current circumstance—when quantity demanded is close to productive capacity—OPEC's ability to lower prices is limited. In an effort to meet strong demand, OPEC has kept production and official quotas at record levels for the most part in the past two years. As a result, crude oil inventories have risen significantly (while gasoline inventories remain below average—see below). However, this accommodative stance toward demand, together with limited investment in capacity in the past two decades, has resulted in a significant reduction in OPEC's excess capacity, currently estimated at 1.4 mbd excluding that in Iraq, thus limiting OPEC's monopoly power and its ability to influence global prices. As a result, the current price-output configuration in the crude oil market is essentially a competitive equilibrium in the short run.

The crude oil price increases of 2004 can be broadly explained by the unexpectedly rapid growth in consumption. Consumption in 2004 grew by 2.9 mbd (3.7 percent—of which China contributed 0.8-1 mbd) relative to 2003—the largest in the past 20 years. It was also higher by around 3.0 mbd (or 4 percent) relative to IEA's projections in mid-2003. With non-OPEC producers unable to increase production relative to original plans, the call on OPEC rose significantly. In view of OPEC's near-to-capacity production, implying a close-to-vertical supply curve, and inelastic short-run demand for oil, the 30 percent price increase in 2004 appears to have been well within the range implied by fundamentals in the physical market. Indeed, given estimated historical non-OPEC short-term supply elasticities, crude oil prices could have increased in excess of 60 percent during 2004 in the absence of increased production by OPEC.

Price increases in 2005, while still supported by current market fundamentals, appear largely to reflect the uncertain environment and expectations about future tightness in the market. Crude oil demand growth for 2005 has so far been broadly in line with IEA projections, while supply (with non-OPEC supply shortfalls offset by higher OPEC output) appears adequate given the growth in OECD commercial crude oil inventories. However, strong demand continues to put pressure on production capacity, thereby contributing to upward price pressures. It also appears that—unlike in the 1990s, when OPEC's ability to satisfy excess demand provided a stable anchor for expectations—even transitory events now seem to motivate precautionary or profit-seeking buying, resulting in price movements that at times do not appear justified by current market fundamentals. Indeed, a large part of the price increase (both spot and futures) appears to reflect uncertainty regarding future market conditions. In this context, geopolitical developments, fears of potential supply and refinery disruptions, and other factors may place upward pressure on spot prices by feeding into expectations.

The perception of a limited response of investment to higher oil prices has reinforced these expectations. Based on current investment plans, production capacity is unlikely to grow enough to outpace future growth in consumption and create adequate spare capacity. A significant capacity overhang (mostly amongst OPEC producers) in the 1980s, low oil prices prior to 2004, and environmental considerations in some countries have had an adverse impact on the growth in oil productive and refining capacity. While investment has picked up in the past two years and some oil exporting countries have announced major investment plans, the market does not appear convinced that adequate investment will be forthcoming. Limited openness to foreign investment and uncertainty about licensing terms in some countries, and caution on the part of both national and international oil companies appear to continue to impede investment.

Data problems have also added to uncertainties. Weaknesses are particularly prevalent in relation to data on supply, stocks, and exports, as non-OECD countries have no obligation to supply data to the IEA. While consumption data for OECD countries are reasonably reliable, as emerging market countries (such as China) have become increasingly significant in the oil market, assessing market conditions is becoming ever more difficult.

III. Futures Markets and the Role of Speculation

The significant rise in longer-dated futures prices reflects the perception of continued tightness in the physical market, and is facilitated by increased investor interest. The futures market in the United States has deepened considerably since 1990s, with short-dated contracts increasing from around 30 percent of the U.S. crude oil production in the 1990s to 80 percent in mid-2005 (Figure 4), and synthetic 6-year futures contracts reaching 9 percent of U.S. production (Figure 5) in 2005 compared to less than 1 percent in 1997. Longer-dated futures prices are also responding more to daily oil market news, suggesting that while market participants are more actively forming views about prospects for supply and demand, their assessment of the likely impact on future prices has become more uncertain. This has also created incentives for new players who, through hedging or speculative activities, can potentially benefit from the uncertainty surrounding future supply.

Fig 4

Fig 5

Key players in the energy markets span a diverse group of commercial and non-commercial investors. The set of so-called commercial traders—traditionally oil producers and energy companies that tend to hedge—has been expanded by the growing number of investment banks and hedge funds who own energy-producing facilities, and the emergence of specialized energy trading firms in the wake of deregulation. Furthermore, the distinction between commercial and non-commercial traders is increasingly blurred as non-commercial traders may enter into swap arrangements in which commercial traders act as their agent.

Recent entrants to energy markets (for example, pension and hedge funds) have added diversity to the market and can be a source of liquidity and price discovery. Many of these institutional investors have sought to diversify their investment portfolios by entering energy markets. Industry estimates suggest that approximately $100-120 billion of new investment in the past three years has been in active and passive energy investment vehicles. Hedge funds, which seek to arbitrage perceived inefficiencies in market valuations, typically employ more active investment strategies and could influence market outcomes in the short term. In contrast, the index-related vehicles used by passive investors tend to be strategic (i.e., seeking portfolio benefits such as diversification) and relatively long-term in nature, responding to perceptions of a structural component in recent price movements.

While the new investors could be instrumental in translating expected future fundamentals into current prices, excessive activity based on limited information may lead to a disconnect between the futures and physical markets. In particular, excessive activity by newcomers or herd behavior by investors may exaggerate the impact of concerns about current and future supply conditions at all points along the futures curve, including spot prices. Given that only about 5 percent of futures contracts are ever delivered as a physical product, increased uncertainty can encourage speculative behavior in the futures market. This, in turn, may push up futures prices beyond that warranted by future market fundamentals.

Causality tests suggest that speculative activity, as proxied by net non-commercial long positions, does not have a significant impact on spot prices, but it does moderately influence longer-dated futures prices. The results—which should be treated with caution owing to the definitional problems noted above—also suggest that speculative activity follows rather than leads spot prices, as do longer-dated future prices, which supports the argument that changes in the fundamentals affect, via spot prices, perceptions regarding future physical market conditions.4

IV. What is the Impact of Refining Constraints?

A number of key pricing relationships exist in petroleum product markets. Prices of petroleum products typically move together over time and all of these prices respond to movements in crude oil spot prices (see Figure 3), though often with a lag. As refinery utilization increases, operating costs tend to rise and stronger product demand allows refiners to pass-on a greater proportion of these costs (see IEA, Oil Monthly Report, July 2005).

Refinery bottlenecks remain a serious concern. While global refinery capacity utilization is presently below 90 percent (Figure 6), specialized refinery capacity is limited for particular types of oil (heavy) and in specific regions, requiring regional demand-supply imbalances to be resolved through imports of petroleum products. Although different product specifications for different regions hamper trade in products, the United States for example, imported almost 10 percent of its gasoline requirements in 2004. Given crude prices are determined in a global market, however, localized refining capacity constraints by themselves are unlikely to increase crude oil prices. Econometric evidence (see below) supports this proposition.

Fig 6

Even if refining bottlenecks were binding in an absolute and global sense, the impact on average crude oil prices would be limited. While, in general, the impact of refinery bottlenecks on crude oil prices depends on the structure of the oil market, an absolute global refinery bottleneck should constrain the demand for crude oil. This should, if anything, cap upward pressures on crude oil prices, particularly if the crude oil market is operating near a competitive outcome (as it presently is). Of course, OPEC could always elect to use its monopoly power to extract part of the increase in product prices but—given OPEC's recent stance towards demand—this does not appear to be the case. Nevertheless, refinery bottlenecks could have some modest impact on crude prices by raising the demand for crude oil at the margin—and operating costs—as higher refinery utilization necessitates the use of less efficient techniques (see IEA, July 2005). Bottlenecks could also affect crude prices through expectations by promoting heightened perceptions of generalized shortages. All in all, therefore, it is hard to support some market analysts' suggestion that refinery bottlenecks are a major cause of higher crude prices.

Limited availability of specialized refinery capacity may, however, influence the spread between specific grades of crude oil and that between petroleum products and crude oil prices. The demand for lighter grades of crude oil has increased in the past two years as the capacity to process additional heavy oil has declined. This has raised the price differential between the supply-constrained light and refinery-constrained heavy crude oils. Furthermore, as the experience with Hurricane Katrina clearly illustrated, a major shock to refinery capacity can have far greater impact on product prices than crude oil prices,

A statistical analysis of the relationship between U.S. wholesale gasoline and crude oil prices using weekly data for 1985-2005 reveals that causality runs predominantly from crude oil to wholesale gasoline prices. Moreover, cost pass-through from crude oil to gasoline prices increases as refinery utilization rises, with about 95 percent of crude oil price movements quickly reflected in wholesale prices compared to only 85 percent in the early 1990s (when refinery utilization in the U.S. was significantly lower).

This result likely reflects the fact that crude oil prices are determined in a global market, one that interacts with localized, relatively competitive product markets. Shocks in local product markets may not affect local product prices immediately, unless of course product inventories are insufficient—as in the aftermath of Hurricane Katrina. In general, crude oil prices respond, depending on global supply conditions, only after the shocks in individual product markets accumulate to become a large shock for the global crude oil market. Local product prices will eventually reflect crude oil price movements, but obviously with a lag. The predominance of supply shocks in the crude oil market during the 1970s and 1980s, furthermore, strengthened the causality from crude to product prices during that period.

V. Concluding Remarks

Market fundamentals, together with expectations of continued tightness, have been the primary influence on crude oil prices during the past two years.

  • It is, however, reasonable to assume that a more certain environment regarding current and future fundamentals would lower oil prices as expectations become better anchored.
  • The significant increase in activity in the futures markets does not appear to have contributed to higher spot prices, but may have had some impact on longer-dated futures prices. Also, speculative activity appears to follow, rather than lead, spot prices.
  • Refining capacity constraints, while contributing to perceptions of tight market conditions, mostly affect product rather than crude oil prices. In general, crude oil prices lead product prices, reflecting the global nature of crude oil price determination and OPEC's role as the marginal supplier of crude oil. Refinery bottlenecks have recently contributed to higher spread between light and heavy crude oil prices.

In the light of the arguments presented in this note, the functioning of the crude oil market could be enhanced and excessive upward pressures on prices reduced by better and more timely data collection, more efficient taxation, improved investment environment, and energy conservation.

  • Better quality and more timely data on current production, consumption and inventory (especially in non-OECD countries), oil reserves, and planned investment by both international and national oil companies would facilitate the adjustment of market participants' expectations to shifts in fundamentals, and improve investment and consumption decisions. Strengthening data quality involves coordinating efforts at both national and international levels. By extending data reporting to non-OECD countries, the Joint Oil Data Initiative (JODI) should improve quality and coverage. The IMF contributes to improving oil data by focusing on transparency issues, encouraging major oil trading countries to participate in the IMF's data initiatives. While international efforts are essential for strengthening data quality, success ultimately rests on strong and focused efforts by national authorities.
  • Adopting more efficient taxation policies to reflect social costs of oil consumption and reducing inappropriate domestic subsidies (explicit or implicit) for petroleum products would help promote demand adjustment. Keeping domestic product prices in line with international prices would encourage consumers to internalize market pressures.
  • Improving the regulatory environment and avoiding unexpected and frequent tax changes in oil exporting countries would encourage foreign investment, speed up the development of new fields, and provide access to advanced technology. Furthermore, international oil companies could contribute by basing their investment decisions on more realistic risk and return tradeoff analyses. Increased investment, by easing fears of future supply shortages, could reduce price pressures.
  • Energy conservation will further ease demand pressures. Encouraging alternative sources of energy, improving public transport, and appropriate efficiency standards and energy taxes are essential to reduce oil intensity.

1The IMF average petroleum spot price (APSP) is an equally weighted average of the West Texas Intermediate, Brent, and Dubai crude oil prices.
2While a portion of this reflected structural changes such as increase in transport demand, the remaining part resulted from electricity shortages—which could be transitory.
3Hurricane Katrina also reduced crude oil production by about 1.5 mbd (12 percent of total daily U.S. production), causing crude oil prices to rise. There is currently insufficient information to assess the lasting impact of Katrina on U.S. production and refining capacity, and oil prices.
4Haigh, et. al. (U.S. Commodity Futures Trading Commission Working paper, April 2005) find similar results using a different framework, while Merino and Ortiz (OPEC Review, vol. 29, Issue 2, 2005) suggest that speculation could have an impact on prices once the effect of inventories is taken out. Extending our analysis to include inventories, however, did not change our basic results.