Tracing The Origins of ESP
HEPEX Historical Hydrology Series, Edition 1
Contribution by Andy Wood, Tom Pagano, and Maury Roos, with special thanks to Mike Anderson
Every day in the western US and elsewhere in the world, reservoir release decisions are made based in part on seasonal reservoir inflow forecasts created using a operational technique called Ensemble Streamflow Prediction (ESP, which originally stood for Extended Streamflow Prediction).
ESP has become a key component of operational long lead prediction in a number of agencies (in the US, Russia, Czechoslovakia, Germany, Sweden, to name a few countries) in the last few decades. The earliest common references for ESP are a US Western Snow Conference presentation (Twedt et al., 1977) and Day et al. (1985) journal article.
But where did ESP really begin?
We have to go back a little over 40 years, when the State of California was about to grapple with a series of long lead ‘water supply forecast’ (WSF) busts in one of the worst droughts to hit parts of the western US in the 20th Century – the two year drought of 1976-77.
In the fall of 1977, California water resources were dire condition, leading to intense interest in predictions for the coming winter and 1978 water supply (i.e., snowmelt runoff) outcomes. California’s winter rain and snowfall typically provide the bulk of the water used by the state throughout the year.
The 1970s were an exciting period of experimentation with seasonal, regional climate forecasting. California was working with private consultants on a program called Project Hydrospect, the goal of which was to use long-range weather forecasts to help with water supply management.
Researchers at Scripps (including Jerry Namais, who in the 1960s had developed monthly and seasonal forecasts at the U.S. Weather Bureau ) were making innovative forecasts, leveraging new computer capabilities in linear algebra (eg, inverting matrices!) that opened doors to objective analysis techniques such as relating EOFs of Pacific Ocean SSTs to California climate. ENSO indices had not yet been defined … but that’s another interesting story, perhaps one for another blog.
The very first ESP river forecasts were simultaneously being developed in the mid-1970s by a joint federal-state operations center that housed personnel from the State of California Department of Water Resources (CADWR) and the California-Nevada River Forecast Center (CNRFC) of the National Weather Service (NWS).
Flood forecasting was a federal responsibility, and in theory, longer-range forecasts were a joint activity. The two parts of longer range forecasts were:
- a continuous soil moisture accounting model (the Sacramento model, developed by NWS; Burnash et al, 1973) and
- the concept of combining estimated initial hydrologic conditions (IHCs) with future meteorological forecasts based on historical sequences or seasonal climate predictions to drive hydrologic simulations extending out to seasonal lead times.
A depiction of a seasonal ensemble forecasts from one of the first publications in which it was called ESP is below.
The ideation of ESP and initialization date of the first ESP forecast in the US is unclear, but the likely first operational ESP forecaster was a woman named Helen Joyce Peters.
Ms. Peters headed the State side of the joint flood forecasting program at the time, and was running the long-lead forecasts as well. Anecdotal reports suggest that her forecasts predated those from the NWS side of the center (made by Bob Burnash) by several weeks to months.
Ms. Peters was a groundbreaker in more than one way: she was the California DWR’s first female engineer as well as an internationally-renowned groundwater specialist. Others involved in the effort included consultants Jack Hannaford, Joe Burns, Bob Zettlemoyer from the State, and John Schaake, who joined the NWS Hydrology Laboratory (HL) in 1974. In the same year, Dave Smith of NWS coined the term Extended Streamflow Prediction, and the NWS HL soon after began developing an NWS version of ESP for use in other RFCs.
The two first major applications of ESP were both drought related – predictions made during the California drought and also for a smaller but intense drought in Virginia, affecting the Occoquan Reservoir, which formed part of the water supply for the Washington DC area. Based on the ESPs and other analysis techniques, probabilistic predictions for reservoir target level failures could be compiled (figure below). Today, we take such approaches for granted, but at the time they were quite innovative applications of stochastic methods in hydrology.
It’s interesting that the first ESP forecasts were made just an hour up the road from the source of so much modern-day innovation at companies such as Google and Apple. Is there something special in the water there?
In any case, ESPs — born in Sacramento, and focusing on the water supply drainages of the California Sierras — were due in no small measure to Joyce Peters, a little-known original forecaster and program lead. Ms. Peters passed away in 2002 after a remarkable career with DWR lasting 37 years.
NB: This reconstruction of the ESP origin is based on notes from interviews Tom Pagano (AU BOM) conducted with John Schaake, former Chief Scientist of the NWS Office of Hydrology, and on correspondence between Andy Wood (NCAR), Eric Rosenberg (Hazen and Sawyer), and Tom Pagano with Maury Roos, the former Director of the California Department of Water Resources. California’s State Climatologist, Mike Anderson, was also extremely helpful in providing digitized copies of Hydrospect Project reports from the mid to late 1970s, which are a fascinating read. Like any history based on long-ago memories and incomplete documentation, it is likely to be just one possible version of the actual events, and may contain inaccuracies. It is also possible that ESP evolved independently in other countries. Please send comments if you know more details of this history, or a related forecasting history that would be of interest to HEPEX readers.
- Burnash, R.J.C., Ferral, R.L., and McGuire, R.A. (1973). “A Generalized Streamflow Simulation System – Conceptual Modeling for Digital Computers,” U.S. Department of Commerce, National Weather Service and State of California, Department of Water Resources.
- Day, G. (1985). “Extended Streamflow Forecasting Using NWSRFS.” J. Wat. Res. Plan. Mgmt., 10.1061/(ASCE)0733-9496(1985)111:2(157), 157-170.
- Hirsch, R, J Schaake and D Sheer (1977). Assessment of Current Occoquan Water Supply Situation. Pub: # ICPRB-M-4, Interstate Commission on the Potomac River Basin.
- Twedt, T. M., J. C. Schaake, Jr., and E. L. Peck (1977). National Weather Service extended streamflow prediction. Proc., Western Snow Conference, 52 – 57.
6 thoughts on “Tracing The Origins of ESP”
Thanks for a cracker of a post Andy, Tom and Maury. Interesting to note that drought is so often the catalyst for the development of seasonal water forecasting services. (Here in Australia seasonal water forecasting got a significant funding boost in the wake of the Millennium Drought.)
Thanks a lot for this post! I was looking for some background on operational streamflow forecasting techniques in the western U.S., and this was particularly useful in terms of context and references.
While Andy, Tom and Maury provided context for the early use of ESP, but I think they didn’t get the story quite right from my memory. The drought in the Potomic River basin in the mid-1970 resulted in the USGS (Bob Hirsch) and NSW (John Schaake, who just joined the Office of Hydrology from MIT) trying to forecast water supply availability from the Occoquan Reservoir (Table 1 of the blog). The USGS preferred to use streamflow timeseries (a la the Harvard Water program and Mike Fiering’s work) for the forecast, while John thought they should use historical weather data and Sacramento hydrological model from observed initial conditions, since this would capture non-linearlities that a timeseries model wouldn’t. (This collective analysis led to the Hirsch, Shaake and Peck 1977 report referenced in the blog.) As I remember all of this, ESP was really the creation of John Shaake, and after the Potomic study, applied it to the western snow packs for the California drought. It became semi-operational for assessing water supply from western snow packs since the April 1 snow pack had such an impact on the total season’s water supply.
Hi Eric — Thanks for adding your recollections to the narrative. I double-checked with John S, who had reviewed the post before it was published, and he confirmed that ESP predated his application of it. The first ESP forecasts were issued in California before he joined NWS in 1974. His role was not the creator, but he was instrumental in helping ESP become widely used and thus he can be credited for its success as a core operational technique in the US and other countries today. John noted that the Occoquan application was the first in the East and maybe even the first outside of California. Cheers, Andy
In-case anyone is interested, you can find a PDF of the Twedt et al. (1977) conference proceedings here: https://westernsnowconference.org/node/1106
Nice, Shaun. Thanks! It is interesting to see that the future directions in the paper highlight dealing with large amounts of data in larger catchment data sets and the minimization of uncertainty in the approach by improving selection, QPF or state estimation.