In Hurricane Forecasting, Science is Far From Exact
By CARL BIALIK | The Numbers Guy | Wall Street Journal | June 2005
Hurricane season has officially begun in the U.S. -- and so has the annual guessing game, played out in the media, of just how many of the violent storms we'll see this year.
Each spring, climatologists take a stab at predicting how many hurricanes will threaten the Caribbean and Southeastern U.S. between June 1 and November 30. Routinely covered as heavily as Punxsutawney Phil's pronouncements, the forecasts were especially awaited this year with memories of 2004's hurricane devastation still fresh. Six hurricanes made landfall in the U.S. last year, causing more than $20 billion in damage by some estimates.
In recent weeks, CNN, MSNBC and CBS -- and dozens of newspapers around the country -- all reported forecasts by Colorado State professor William Gray, the pioneer of long-range hurricane predicting, of eight hurricanes this year (revised up from seven in April). Other forecasts from the National Oceanic and Atmospheric Administration, Florida State University and University College London project between six and eight hurricanes.
The Numbers Guy examines numbers and statistics in the news, business, politics and health. Some numbers are flat-out wrong, misleading or biased. Others are valid and useful, helping us to make informed decisions. As the Numbers Guy, I will try to sort through which numbers to trust, question or discard altogether. But my examination of forecasters' records over the past six years -- the period in which all four groups have been publicizing their predictions before the hurricane season begins -- shows that none do much better than a simple five-year average, a number that can be derived without expertise in climatology or statistics (more on this in a moment). Furthermore, despite all the press they generate, these forecasts aren't very useful to insurance companies, emergency planners and others who you might think would crave them. Researchers are recognizing this and moving away from the headline numbers to more complex forecasts that attempt to better capture hurricane season's impact.
First, it's important to understand what these numbers mean. The projections count all storms that form in the Atlantic basin -- the Atlantic Ocean north of the equator, the Caribbean and the Gulf of Mexico -- and that rise to the hurricane definition set by NOAA's National Hurricane Center: winds of 64 knots (73.6 miles per hour) that are sustained over a one-minute average period. The projections include storms that never touch ground.
To predict the future, hurricane forecasters look to the past. They analyze various weather parameters to see which have correlated best with hurricane activity. These include water temperature, wind speed and the size of the El Niño effect. Forecasters I spoke with said they create statistical models that best fit the historical data, and then turn the resulting crystal ball toward the future. This method could be replicated by a statistician who knows nothing about meteorology -- the models don't ask why a certain temperature creates a certain effect, they merely say that in the past it has been likely to do so. (Dr. Gray says he adjusts his results based on other climatological factors not included in his statistical model, like long-term changes in ocean conditions.)
Historical weather data are surprisingly detailed, researchers told me. Some forecasters go back to 1944, when pilots began flying into storms to measure them. (Since the 1960s, such data have been supplemented by satellites.) Newspaper accounts and other sources can help forecasters look back even further. Charles Watson, a University of Central Florida professor who predicts the probability that hurricanes will touch down in cities around the country, says that shippers and insurers had an incentive to create state-of-the-art measurements: "When there's money involved, people pay attention."
Researchers' methods may be complex and rigorous, but their results over the past six years -- at least for the total number of hurricanes -- haven't been much better than an educated guess. To evaluate the forecasts, I measured the average error for each one -- the difference each year between predicted hurricanes and the actual number. (Thanks to Iowa State statistics professor Philip Dixon and National Center for Atmospheric Research scientist Barbara Brown for advising me on this evaluation, and reader Paul Haskins for suggesting it.) I looked at forecasts made in May or earlier, before the start of hurricane season -- some researchers continue to update their predictions later in the year. It turns out that all four forecasts have missed by between 1.3 and 1.5 hurricanes each year. But a more simplistic method, a five-year moving average of hurricane counts, does just as well, missing by an average of 1.4 hurricanes each year. (To arrive at a five-year moving average, you simply add the counts for the previous five years and divide by five. For instance, to get a prediction for the number of hurricanes in 2000, I averaged the actual counts from 1995 through 1999. For 2001, I used counts from 1996 through 2000. I did this for each of the six years I evaluated.)
Even if forecasters improve their ability to predict hurricane counts, they may not be accomplishing a whole lot. There's little correlation between the number of hurricanes and the number that hit land in the U.S. (see NOAA's hurricane count and list of hurricanes that touched ground). There were at least eight hurricanes in each of 1990, 2000 and 2001, but none hit land. Each year between 1991 and 1993, there were four hurricanes and one hit land each year. Last year's devastating season featured nine hurricanes and six landfalls. Even the number of landfalls doesn't tell all; 1992's lone strike was named Andrew, and it caused roughly $20 billion in damages. Also, the forecasts don't say anything about when the storms will hit.
As a result, it's hard to find industry groups or government agencies who base planning decisions on the forecasts. Ayana McIntosh-Lee, a spokeswoman for BP Plc, said the energy company relies more on three-to-five-day forecasts for decisions about personnel on its offshore drilling operations. Same goes for the U.S. Navy's Second Fleet, Lieutenant Mike Kafka told me: "While we view long-range forecasts with interest, they do not play a large role in our decision to move assets." Coast Guard spokesman Ron Mench said, "We wait until there is actually an active storm."
Carolyn Gorman, vice president of the trade group Insurance Information Institute, told me insurers used to fund hurricane forecasting by researchers, including Dr. Gray. (She said insurers have "a lot of respect" for Dr. Gray's work.) But the companies have since pulled back. The forecasts come out too late for insurers to act on them: "They don't come out with anything like enough lead time to adjust insurance rates," she said. Even reinsurers can't benefit much because most of their contracts are signed by Jan. 1, says Richard Murnane, program manager at the Bermuda-based Risk Prediction Initiative, an insurance-industry-funded research group.
The forecasters are de-emphasizing the total hurricane count and instead working on projecting other, more-useful measures, like the number of landfalls and something called the accumulated cyclone energy index, which provides a measure of total storm intensity and duration. A Narragansett, R.I., company called Accurate Environmental Forecasting Inc. and Dr. Watson's team at Central Florida focus on the likelihood of storms touching down in various regions. (The most endangered cities have just a 10% chance of being struck by a hurricane this year, according to the forecasts. Central Florida stands at 4.9%.)
Ben Nelson, state meteorologist with the Florida Division of Emergency Management in Tallahassee, says the landfall predictions aren't yet good enough. "That part of the science hasn't been accurate enough for us to base protective actions on them, months in advance," he says. However, like other emergency managers, he values the yearly media hoopla around hurricane forecasts: "Above all, the forecasts that Dr. Gray put out each year raise public awareness. We're supportive of anything that raises public awareness before the hurricane season begins."
Those may not be stirring words for scientists, who are in the business of describing the physical world, not crafting public-awareness campaigns. And the support of emergency managers can be contingent: Dr. Watson points out that officials don't much care for forecasts of below-average activity, which have less public-relations value.
With memories of last year's grim season fresh and the 2005 outlooks particularly foreboding, this summer will be something of a proving ground for the researchers. This year, their forecasts are mostly in line -- ranging from six to eight hurricanes. "That would give you confidence we all have it right," says Christopher Landsea, an NOAA meteorologist. "Either that," he laughs, "or we'll all going to bomb."
[Note: The text was slightly shortened to comply with ASF requirements]
By CARL BIALIK | The Numbers Guy | Wall Street Journal | June 2005
Hurricane season has officially begun in the U.S. -- and so has the annual guessing game, played out in the media, of just how many of the violent storms we'll see this year.
Each spring, climatologists take a stab at predicting how many hurricanes will threaten the Caribbean and Southeastern U.S. between June 1 and November 30. Routinely covered as heavily as Punxsutawney Phil's pronouncements, the forecasts were especially awaited this year with memories of 2004's hurricane devastation still fresh. Six hurricanes made landfall in the U.S. last year, causing more than $20 billion in damage by some estimates.
In recent weeks, CNN, MSNBC and CBS -- and dozens of newspapers around the country -- all reported forecasts by Colorado State professor William Gray, the pioneer of long-range hurricane predicting, of eight hurricanes this year (revised up from seven in April). Other forecasts from the National Oceanic and Atmospheric Administration, Florida State University and University College London project between six and eight hurricanes.
The Numbers Guy examines numbers and statistics in the news, business, politics and health. Some numbers are flat-out wrong, misleading or biased. Others are valid and useful, helping us to make informed decisions. As the Numbers Guy, I will try to sort through which numbers to trust, question or discard altogether. But my examination of forecasters' records over the past six years -- the period in which all four groups have been publicizing their predictions before the hurricane season begins -- shows that none do much better than a simple five-year average, a number that can be derived without expertise in climatology or statistics (more on this in a moment). Furthermore, despite all the press they generate, these forecasts aren't very useful to insurance companies, emergency planners and others who you might think would crave them. Researchers are recognizing this and moving away from the headline numbers to more complex forecasts that attempt to better capture hurricane season's impact.
First, it's important to understand what these numbers mean. The projections count all storms that form in the Atlantic basin -- the Atlantic Ocean north of the equator, the Caribbean and the Gulf of Mexico -- and that rise to the hurricane definition set by NOAA's National Hurricane Center: winds of 64 knots (73.6 miles per hour) that are sustained over a one-minute average period. The projections include storms that never touch ground.
To predict the future, hurricane forecasters look to the past. They analyze various weather parameters to see which have correlated best with hurricane activity. These include water temperature, wind speed and the size of the El Niño effect. Forecasters I spoke with said they create statistical models that best fit the historical data, and then turn the resulting crystal ball toward the future. This method could be replicated by a statistician who knows nothing about meteorology -- the models don't ask why a certain temperature creates a certain effect, they merely say that in the past it has been likely to do so. (Dr. Gray says he adjusts his results based on other climatological factors not included in his statistical model, like long-term changes in ocean conditions.)
Historical weather data are surprisingly detailed, researchers told me. Some forecasters go back to 1944, when pilots began flying into storms to measure them. (Since the 1960s, such data have been supplemented by satellites.) Newspaper accounts and other sources can help forecasters look back even further. Charles Watson, a University of Central Florida professor who predicts the probability that hurricanes will touch down in cities around the country, says that shippers and insurers had an incentive to create state-of-the-art measurements: "When there's money involved, people pay attention."
Researchers' methods may be complex and rigorous, but their results over the past six years -- at least for the total number of hurricanes -- haven't been much better than an educated guess. To evaluate the forecasts, I measured the average error for each one -- the difference each year between predicted hurricanes and the actual number. (Thanks to Iowa State statistics professor Philip Dixon and National Center for Atmospheric Research scientist Barbara Brown for advising me on this evaluation, and reader Paul Haskins for suggesting it.) I looked at forecasts made in May or earlier, before the start of hurricane season -- some researchers continue to update their predictions later in the year. It turns out that all four forecasts have missed by between 1.3 and 1.5 hurricanes each year. But a more simplistic method, a five-year moving average of hurricane counts, does just as well, missing by an average of 1.4 hurricanes each year. (To arrive at a five-year moving average, you simply add the counts for the previous five years and divide by five. For instance, to get a prediction for the number of hurricanes in 2000, I averaged the actual counts from 1995 through 1999. For 2001, I used counts from 1996 through 2000. I did this for each of the six years I evaluated.)
Even if forecasters improve their ability to predict hurricane counts, they may not be accomplishing a whole lot. There's little correlation between the number of hurricanes and the number that hit land in the U.S. (see NOAA's hurricane count and list of hurricanes that touched ground). There were at least eight hurricanes in each of 1990, 2000 and 2001, but none hit land. Each year between 1991 and 1993, there were four hurricanes and one hit land each year. Last year's devastating season featured nine hurricanes and six landfalls. Even the number of landfalls doesn't tell all; 1992's lone strike was named Andrew, and it caused roughly $20 billion in damages. Also, the forecasts don't say anything about when the storms will hit.
As a result, it's hard to find industry groups or government agencies who base planning decisions on the forecasts. Ayana McIntosh-Lee, a spokeswoman for BP Plc, said the energy company relies more on three-to-five-day forecasts for decisions about personnel on its offshore drilling operations. Same goes for the U.S. Navy's Second Fleet, Lieutenant Mike Kafka told me: "While we view long-range forecasts with interest, they do not play a large role in our decision to move assets." Coast Guard spokesman Ron Mench said, "We wait until there is actually an active storm."
Carolyn Gorman, vice president of the trade group Insurance Information Institute, told me insurers used to fund hurricane forecasting by researchers, including Dr. Gray. (She said insurers have "a lot of respect" for Dr. Gray's work.) But the companies have since pulled back. The forecasts come out too late for insurers to act on them: "They don't come out with anything like enough lead time to adjust insurance rates," she said. Even reinsurers can't benefit much because most of their contracts are signed by Jan. 1, says Richard Murnane, program manager at the Bermuda-based Risk Prediction Initiative, an insurance-industry-funded research group.
The forecasters are de-emphasizing the total hurricane count and instead working on projecting other, more-useful measures, like the number of landfalls and something called the accumulated cyclone energy index, which provides a measure of total storm intensity and duration. A Narragansett, R.I., company called Accurate Environmental Forecasting Inc. and Dr. Watson's team at Central Florida focus on the likelihood of storms touching down in various regions. (The most endangered cities have just a 10% chance of being struck by a hurricane this year, according to the forecasts. Central Florida stands at 4.9%.)
Ben Nelson, state meteorologist with the Florida Division of Emergency Management in Tallahassee, says the landfall predictions aren't yet good enough. "That part of the science hasn't been accurate enough for us to base protective actions on them, months in advance," he says. However, like other emergency managers, he values the yearly media hoopla around hurricane forecasts: "Above all, the forecasts that Dr. Gray put out each year raise public awareness. We're supportive of anything that raises public awareness before the hurricane season begins."
Those may not be stirring words for scientists, who are in the business of describing the physical world, not crafting public-awareness campaigns. And the support of emergency managers can be contingent: Dr. Watson points out that officials don't much care for forecasts of below-average activity, which have less public-relations value.
With memories of last year's grim season fresh and the 2005 outlooks particularly foreboding, this summer will be something of a proving ground for the researchers. This year, their forecasts are mostly in line -- ranging from six to eight hurricanes. "That would give you confidence we all have it right," says Christopher Landsea, an NOAA meteorologist. "Either that," he laughs, "or we'll all going to bomb."
[Note: The text was slightly shortened to comply with ASF requirements]
