Making connections through HEPEX – A CSIRO experience

By QJ Wang, Andrew Schepen, David Robertson, James Bennett, Durga Lal Shrestha, Yong Song and Tony Zhao members of the CSIRO Columnist Team

HEPEX has a reputation as a strong community of like-minded, supportive researchers and practitioners who are collectively bent on significantly advancing the science and applications of ensemble hydrological forecasting. HEPEX provides a unique forum where meeting people and exchanging ideas has the very real potential to open up new, exciting collaborative research opportunities. Indeed, this has been the experience of the CSIRO water forecasting team. A number of joint research activities have been kick-started in the wake of HEPEX workshops.


Participants in the last HEPEX workshop in Quebec, Canada, in June 2016

Following the Beijing workshop in 2012, Florian Pappenberger from ECMWF and our team saw an opportunity to work together in applying our CBaM rainfall post-processing method to seasonal forecasts from ECMWF’s System4 global model. A PhD student visiting CSIRO subsequently evaluated CBaM and System4 for seasonal climate forecasting across the whole of China. This led to a publication in JGR: Atmospheres.

A relationship with the NCEP Climate Prediction Center (CPC) was seeded during the HEPEX workshop in Maryland in 2014. The well-connected HEPEXers Andy Wood, Qingyun Duan and D-J Seo kindly introduced QJ Wang to colleagues in the CPC. This led to a number of side meetings at the time and many follow-up communications in the subsequent two years. Now CPC has a two-year funded collaborative project with our team to apply CBaM to CFSv2 and NMME data for US seasonal climate forecasting.


QJ Wang (CSIRO) presenting at the HEPEX Workshop on seasonal hydrological forecasting in Norrköping in September 2015

The SMHI Norrkoping workshop was similarly fruitful. We are currently working with Michael Butts and his team at DHI on an initial evaluation of methods for seasonal hydrological forecasting in Europe. From presentations during the same workshop, it was clear that the method of quantile-mapping (QM) had become very popular for post-processing ensemble seasonal climate forecasts. Following discussions with Andy Wood and Maria-Helena Ramos at the workshop, we decided to conduct an evaluation of QM to highlight its strength and weakness. This has now resulted in a joint research paper, which is currently in review for possible journal publication.


Andrew Schepen (CSIRO) presenting the HEPEX seasonal streamflow intercomparison project at the HEPEX Workshop in Quebec in June 2016

Of course, there is also the HEPEX seasonal ensemble hydrological forecasting test-bed initiative, in which we are actively participating.

Regionally our team is also reaching out to connect to forecasting researchers and practitioners in Asia through HEPEX special sessions at conferences like AOGS and MODSIM.

These are examples of connections that we, the CSIRO water forecasting team, made through the HEPEX forum.

HEPEX is sometimes said to have a friendly, big-family atmosphere and there is no doubt that others have been making many connections too. We certainly encourage all to capitalise on opportunities to make new connections and advance our science.

Who have you met and what are you working on? We’d love to hear your stories in the comments.

Posted in columnist, meetings, projects | Leave a comment

The quest for better global precipitation data

Contributed by Hylke Beck, Albert van Dijk, Ad de Roo, Jaap Schellekens, Diego Miralles, Brecht Martens, and Vincenzo Levizzani

Information on precipitation is essential for almost any hydrological study. Unfortunately, precipitation is also one of the most difficult to estimate meteorological variables, due its tremendous spatio-temporal heterogeneity, particularly in tropical, mountainous, and snow-dominated regions.

Over the past decades, many precipitation datasets have been developed using different techniques and observation sources (see, for instance, International Precipitation Working Group, UCAR Climate Data Guide, and

So, which precipitation dataset should one use?

Well, that depends on the application, but in the context of global hydrological modelling it has never really been possible to identify the “best” dataset for several reasons.

  • Importantly, none of the currently available datasets exploit the complementary nature of satellite and reanalysis data, or fully use all the available gauge data.
  • Moreover, many datasets cover only part of the global land surface, and have a short temporal record, or a coarse spatial or temporal resolution.
  • Last but not least, the datasets rarely fully account for gauge undercatch and orographic effects, and therefore exhibit biases in snow-dominated and mountainous areas (see, for example, Herold et al., 2015).

The MSWEP dataset

GPD-Fig1To address these shortcomings, we recently developed a new global precipitation dataset, with higher spatial and temporal resolution than most available datasets (0.25° and three-hourly, respectively). Referred to as Multi-Source Weighted-Ensemble Precipitation (MSWEP; fully described in Beck et al., 2016), the dataset currently covers the period 1979–2015.

It is unique in that it uses an unprecedented range of data sources, including two datasets based solely on rain gauges (GPCC and CPC Unified), three on satellites (CMORPH, GSMaP, and TMPA), and two on reanalyses (ERA-Interim and JRA-55). The use of rain-gauge data ensures reliable precipitation estimates in regions with dense rain-gauge networks, while satellite and reanalysis data are used to enhance the precipitation estimates in sparsely-sampled tropical and snow-dominated regions.

Another useful aspect of MSWEP is that it accounts for gauge undercatch and orographic effects in a new way. Traditionally, equations that incorporate wind speed and air temperature have been used to correct for gauge undercatch, but we did not find these equations to perform well. In Central Russia, for example, precipitation was consistently overestimated, while there was insufficient precipitation in many mountainous regions. Instead, we developed a novel approach to infer the “true” precipitation from streamflow observations for ~14,000 catchments across the globe using a Budyko-type equation, after which we interpolated the resulting bias correction factors to obtain a global bias correction map (see figure below).


Figure 1: Bias correction factors based on streamflow observations.

The video below shows an animation of three-hourly precipitation data from MSWEP for the seven-month period of June to December 2006.


Figure 2: Three-hourly MSWEP precipitation, for the seven-month period from June to December 2006 [Click on the image to see the animation].

The patchy nature of precipitation in the tropics, where MSWEP relies mainly on data from instruments onboard the (now decommissioned) TRMM satellite, is due to the localised convective storms that develop in warm, moist tropical conditions.

By contrast, large frontal systems occur when cold and warm air masses collide in poleward regions, where MSWEP relies primarily on reanalysis datasets produced using atmospheric models, which are designed to simulate large-scale weather systems. Finally, large mountain ranges, for example along the Pacific Coast of North America, show the occurrence of orographic storms due to the uplift of moist air.

MSWEP is freely available via For more information, see our open-access publication in HESS Discussions here.

Applicability of MSWEP

MSWEP is useful for a broad range of hydrological applications from local to global scales.

  • The value of MSWEP for streamflow simulation has been assessed by Beck et al. (2016). To this end, we calibrated the popular HBV hydrological model against daily streamflow observations with different precipitation estimates, including MSWEP. The calibration was performed for 9,172 catchments across the globe.

Figure 3: Median NSE scores obtained after calibration of HBV using different precipitation datasets. Catchments are grouped on the y-axis according to mean distance to the closest rain gauge.

The figure on the right presents median Nash-Sutcliffe Efficiency (NSE) scores obtained after calibration for catchments grouped according to their mean distance to the closest rain gauge. MSWEP (dark blue) obtained higher scores for all groups, suggesting that MSWEP performs well in regions with dense as well as sparse rain-gauge networks.

The results for catchments with a mean distance of more than 25 km to the closest rain gauge are of particular interest, since ~84% of the global land surface falls into those categories!

  • The value of MSWEP for evaporation and soil moisture simulation was also assessed. For this purpose we used the Global Land Evaporation Amsterdam Model (GLEAM), a relatively simple, observation-driven model designed to estimate terrestrial evaporation from satellite-based inputs.

For 64 FLUXNET tower stations around the globe, the GLEAM-based evaporation and soil moisture simulations obtained using precipitation from MSWEP and CPC Unified (which is based on rain-gauge data) were validated against observations. MSWEP was found to perform at least as well as CPC Unified, even though FLUXNET tower stations are concentrated in regions with dense rain-gauge networks, where the quality of CPC Unified is known to be very high.

Based on these results, and on the perceived higher quality of MSWEP in sparsely-gauged regions, MSWEP has been  selected as the core precipitation forcing for the GLEAM v3.0a dataset, a global 0.25° 35-year record of terrestrial evaporation and soil moisture (freely available here). See the paper by Martens et al. (2016) for more details on the GLEAM-derived datasets.

What’s next?

We plan to continue updating the MSWEP estimates to take advantage of new and improved data sources such as ERA5 (the successor of ERA-Interim; see this presentation by Dick Dee from ECMWF), IMERG (the successor of TMPA; more details and near real-time precipitation maps available here), and SM2RAIN (a precipitation product derived from satellite-based surface soil moisture; see Brocca et al., 2014).

We also plan to include ocean areas, to make MSWEP a truly global dataset, and to correct the daily precipitation distribution data calculated using gauges – mainly to reduce the “drizzle effect” (explained in Maraun, 2013).

Finally, we are developing a near real-time (NRT) version of MSWEP. This product, called MSWEP-NRT, would be the first NRT product to provide fully global precipitation estimates with demonstrated good performance in densely gauged areas as well as tropical and polar regions. MSWEP-NRT should be particularly useful to improve the initialization of the model state variables in global or regional forecasting systems, that is, the amount of water stored in the snowpack, the soil, the groundwater, and other water stores. The Global Flood Awareness System (GloFAS), for example, currently uses the 24-h lead time ECMWF forecast from yesterday to initialize the forecast of today, which means that initialization is solely based on weather model data. The use of MSWEP-NRT should improve initialization in densely gauged regions by taking advantage of the rain gauge data, and in tropical regions by taking advantage of the satellite data. That way, better estimates of precipitation will help achieve better streamflow forecasts.


Posted in data systems | 5 Comments

The Current State of Operational Global (and continental) Scale Flood Forecasting

contributed by Rebecca Emerton, PhD Student at the University of Reading and ECMWF

After reading Chantal’s recent blog post discussing some of the limitations for international flood forecasting, and last week’s interview with Thomas Adams and Tom Pagano discussing their new book describing national and regional flood forecasting systems around the world, I thought it would be interesting to add to this dialogue with a brief overview of the current state of large-scale (global and continental) operational flood forecasting.

It has only really become possible to produce flood forecasts at the global scale in recent years, largely due to the integration of meteorological and hydrological modelling capabilities, improvements in data, satellite observations and land-surface hydrology modelling, and increased resources and computer power. Two of the key scientific and technological advances that have enabled the forecasting of floods at the continental and global scale include the significant increase in the skill of precipitation forecasts in recent years, and the move from deterministic to probabilistic (ensemble) numerical weather prediction (NWP).

Key factors

Three key factors highlight the need for large-scale, international flood forecasting systems:

  1. to provide information on floodiness (Stephens et al., 2016) across areas larger than a catchment, for example, to indicate where flooding during the rainy season will be worse than normal; information that is of high importance to humanitarian organisations (Braman et al., 2013)

  2. to provide forecasts in basins across the globe where there are currently no forecasts available, aiding disaster risk reduction and flood early warning efforts globally

  3. to support existing capabilities, for example, by using ensemble forecasting techniques to enable probabilistic flood forecasts, or at longer lead times for earlier warnings, as these can be computationally expensive to run.

At present, these forecasting systems are able to produce coarse-scale discharge forecasts covering entire continents or the globe. As previously discussed by Chantal, Thomas and Tom, despite recent advances, there are several grand challenges faced by international forecasting systems at present and moving forward, from data availability to resources and effective communication of forecasts.

At present, there are four operational continental scale flood forecasting systems, and two global scale. Below, I have given a very brief summary of each. These systems are continuously evolving, and are already proving to be key in supplementing national and local forecasting capabilities while supporting global-scale activities. Thanks to my co-authors at the centres running each of these systems, you can read more about each system, such as the components of the model and their operational uses, and the current state of large-scale flood forecasting (including the advances that have led to where we are now, the grand challenges facing the community, and future directions), here: Continental and Global Scale Flood Forecasting Sytems (Emerton et al., 2016). For anyone interested in reading further around national and regional forecasting systems and their histories, I also include a link to the “Flood Forecasting, A Global Perspective” book discussed in last week’s post.

The European Flood Awareness System (EFAS)

EFAS has been fully operational since 2012, with the aim of providing transnational, harmonised early warnings of flood events and hydrological information to national agencies, complementing local services. Four different meteorological forecasts are used as input to the Lisflood hydrological model, producing simulated ensemble hydrographs; however, these alone do not constitute a flood forecast – a decision-making element needs to be incorporated (Thielen et al., 2009). As such, historical meteorological data are run through Lisflood to calculate a 22-year time series of river flow, providing reference thresholds for minor or major flooding. EFAS sends warning emails to national authorities responsible for flood forecasting across Europe, and daily overviews are sent to the Emergency Response Coordination Centre (ERCC) of the European Commission.

European Hydrological Predictions for the Environment (E-HYPE)

H:\Review_Paper\Long Version\ehype_interface.pngE-HYPE is a multipurpose model based on open data, which is used for various applications such as water management, research experiments and flood forecasts. The E-HYPE Water in Europe Today (WET) tool compares the current hydrological situation with climatological data and past modelled events. E-HYPE currently uses a single deterministic NWP input, although ensemble forecasting is intended for future system developments. Alongside natural processes, HYPE also takes into account anthropogenic influences including irrigation and hydropower.

The BoM Flood Forecasting and Warning System

H:\Review_Paper\Long Version\bom_graphics.png

The BoM has been producing flood forecasts operationally for several decades, with the technology and systems used to produce these forecasts continually evolving. Recently, short-term continuous streamflow forecasting has been introduced, alongside event-based hydrological modelling and nowcasting using radar rainfall estimates. The BoM system uses various resolutions (each covering a different spatial scale) of their own deterministic NWP model as input to the Génie Rural á 4 Paramètres (GR4) hydrological models, and forecasters are also able to produce “What If” precipitation scenarios to force the hydrological models. For event-based forecasts, local models are used in support of the continental scale system.

The U.S. Hydrologic Ensemble Forecast Service (HEFS)

H:\Review_Paper\Long Version\hefs_screenshot_2(locationsinflood)_copy.png

The HEFS has been running experimentally at several regional River Forecast Centres (RFCs) across the U.S. since 2012, and is currently being rolled out operationally at all 13 RFCs. The HEFS aims to produce ensemble streamflow forecasts that seamlessly span lead times from less than 1 hour up to several  years, and that are spatially and temporally consistent, calibrated and verified. The HEFS takes four meteorological inputs; two NWP models, alongside the RFC forecasts and historical observations, and each RFC may use a different combination of 19 components within the Hydrologic Processor suite (which contains various hydrologic, hydraulic, reservoir and routing models). Also included in the HEFS are pre- and post-processing steps which allow both the uncertainties in the meteorological input and the hydrology to be taken into account.

The Global Flood Awareness System (GloFAS)

H:\Review_Paper\Long Version\glofas_interface.png

GloFAS (Alfieri et al., 2013) has been producing probabilistic flood forecasts for the global river network with up to 2 weeks lead time in a pre-operational environment since 2011, and can provide downstream countries with early warnings and information on upstream river conditions alongside global overviews of upcoming flood events in large river basins. GloFAS takes the surface and sub-surface runoff from the ECWMF ensemble prediction system as input to the Lisflood river routing model, with runoff from the ERA-Interim reanalysis archive also run through Lisflood, in order to compute return periods for the global river network. Unlike EFAS, GloFAS does not directly disseminate flood warnings, as each country has national procedures to follow, but anyone is able to access and analyse the forecasts for decision-making purposes and research.

The Global Flood Forecasting Information System (GLOFFIS)

H:\Review_Paper\Long Version\gloffis_vis.png

GLOFFIS is a research-oriented operational system based on Delft-FEWS (56), and similarly to several of the continental-scale systems, uses several meteorological inputs to drive two hydrological models; PCR-GLOBWB and W3RA. The idea behind this is to validate, verify and inter-compare real-time rainfall products as they become available. The first version of GLOFFIS went live in 2015, used internally at Deltares and by their customers. GLOFFIS is intended to be a research tool on predictability and interoperability first and foremost, but will be suitable for a wide range of applications once fully operational.

Further reading

Stephens E, Day JJ, Pappenberger F, Cloke H. Precipitation and floodiness. Geophys Res Lett 2015, 42(23): 10316–10323

Braman LS, van Aalst MK, Mason SJ, Suarez P, Ait-Chellouche Y, Tall A. Climate forecasts in disaster management: Red Cross flood operations in West Africa, 2008. Disasters 2013, 37:144–164

Thielen J, Bartholmes J, Ramos MH, de Roo A. The European Flood Alert System – part 1: concept and development. Hydrol Earth Syst Sci 2009, 13:125–140

Alfieri L, Burek P, Dutra E, Krzeminski B, Muraro D,Thielen J, Pappenberger F. GloFAS – global ensemble streamflow forecasting and flood early warning. Hydrol Earth Syst Sci 2013, 17:1161–1175

Posted in ensemble techniques, floods, operational systems | 2 Comments

Flood forecasting systems around the world – Interview with Thomas Adams and Tom Pagano

As the 1st Edition of the book “Flood Forecasting, A Global Perspective” has just been released, Thomas Adams (UCAR) and Tom Pagano (BoM) [see their bio below], editors, recall how the idea of the book was born and what they have learned when editing contributions describing national and regional flood forecasting systems from 11 different countries, as well as continental scale systems. This reference book points out challenges and ways forward to improve flood forecasting around the world.


Australian hydrologist David Wilson at the flood desk of the Victorian Regional Forecasting Centre. He is using the Bureau of Meteorology’s next generation Hydrological Forecasting System (HyFS) for operations during a large East Coast Low system in early June 2016 (© BoM)

Maria-Helena Ramos (MHR): What is the book about?

Thomas Adams: The book presents overviews of national and large regional operational flood forecasting systems as they exist right now around the globe and the operational challenges they face. Our intent is that the authors would provide some historical context that brought the systems to the point they are now. Tom Pagano and I felt it was critical that these systems should be described from the perspective of those that developed and use the system in their countries. We also included chapters on what appears to be the direction flood forecasting is advancing with continental and global forecast systems and the challenges they face moving forward.

MHR: Where does it cover?

Tom Pagano: Every continent is represented except Antarctica. We included developed and developing countries, from China to Colombia to the Congo. It also includes trans-national systems.

MHR: What was the original inspiration for the book? What were you trying to achieve?

Adams: The book began from something of a chance encounter with representatives from Elsevier wanting to offer to the hydrologic community a book covering topics in hydrology following the American Meteorological Society Annual meeting in 2014. I was on the agenda to moderate a technical session and was asked my thoughts for a book. I had thought for some time that something along the lines of the book was long overdue and suggested a book with this format, which Elsevier enthusiastically supported.

MHR: What did you learn in writing the book, what were your most surprising findings?

Adams: (Laughing) How relieved I would be with the completion of the book! With having seen many different approaches to the design and implementation of flood forecast systems, there are not many surprises. At the same time, it is gratifying that there are so many similarities between systems. This, again, is not very surprising since many of the problems are universal. The most troubling issue for me, which the book hopes to address, is the relative lack of understanding of National or regional scale systems and seemingly poor communication between those developing, implementing and operating these forecast systems.


Rapid growth in Chinese monitoring networks in the recent decade (chapter 3, page 72)

Pagano: Like Tom, it’s surprising how universal some problems are, such as data. Water is both a necessity and a hazard; for forecasters, the same is true for data. Data are our lifeblood and yet they can also be our downfall, such as when they are poor quality. I was also impressed with some of the real changes that are underway. For example, in just the past few years China has nearly tripled its already large raingauge and river station networks.

MHR: How are operational systems doing with respect to ensembles?

Adams: A few countries have ensemble products, including public products, but generally it has been an uneasy transition from deterministic to probabilistic products. I would liken it to the transition from point-based to graphical products in weather forecasting; initially there were difficulties and frustrations and questions about if this is what the user wanted and how we can keep experts in the loop… But now it is almost taken as given. There still is a major challenge with end-users accepting and understanding probabilistic forecasts — much needs to be done on this front.

MHR: Is there anything countries could be doing better?

Pagano: Given the common nature of our problems, there are so many opportunities for collaborations and exchanges. This might be easier in other fields – when one country produces an El Nino forecast, many countries can benefit and are interested. However, hydrology is very local and why would other countries care about the forecasts for a catchment in Australia? This leads to systems being developed in isolation, reinventing solutions. Here is where initiatives like HEPEX can come in, to give a sense of the big picture and to make connections throughout the international community forecasters, researchers and developers. Deltares has also built a community around its flood forecasting platform FEWS.

Adams: I agree fully with Tom. We need to be far more effective in opening communication between those that have developed National and large regional forecast systems. We do a very good job at communicating scientifically, but discussions of operational forecast system development and related issues are, too often, treated as secondary topics. An annual international conference dedicated to operational forecast systems aimed at sharing advances is needed.

MHR: The recent Hepex workshop held in Quebec made us wonder that although advanced methods and techniques exist (for ensemble prediction, data assimilation, post-processing, etc.), these are not always already implemented in operational systems. What advice would you give, for researchers and operational forecasters, to foster the science-operations dialogue in flood forecasting?

Adams: This is an excellent question. It goes without saying that there should be something of cooperative effort in the development and implementation of ensemble forecast systems in operational environments. What is missing is demonstration of clear and unambiguous evidence of the superiority of ensemble forecasts over single-valued, deterministic forecasts. This requires a paradigm shift in our thinking and will necessitate significant education and training of forecasters, but also the general public and decision makers.

Pagano: If ensembles are the answer, what was the question again? Do researchers and forecasters know who are going to use these new products, and for what? Operational agencies want to reap the benefit of their investments in their existing systems and are eager to innovate if there is a value proposition. This generally means fewer resources needed to make better products more reliably.

My other advice is to recognize that crises are important catalysts for change in institutions. We read several examples where major floods (or serious droughts) motivated stakeholders to see what more could be done to prevent the next disaster. This led to historic changes in arrangements, practices, technologies and so on. Clearly it helps to have good relationships established between scientists and practitioners before such events happen.

MHR: What is next? A Global perspective on operational drought forecasting systems? Any other projects in your minds?

Adams: I would love to plan for a 2nd edition, which would give current contributors the opportunity to update their chapters, but, more importantly, we would like to include additional contributors that were either missed or were unable to fit into their schedules the time to write a chapter. We also would like to strengthen sections on the future of flood forecasting and the problems facing the enterprise, specifically as it relates to human impacts and warnings. I have been approached by Elsevier to consider another project, which I suggested a similar volume dealing specifically with flash flood monitoring, warnings and prediction. I would encourage anyone who has interest to please contact me.

MHR: Any final thoughts to share with the Hepex community?

Pagano: I think there’s something for everybody in this volume, whether you are a researcher, forecaster, student or even just interested in history and institutions.

MHR: Thank you both for taking your time to answer these questions!

  • Where you can find the book: hard copy and electronic versions are available from Elsevier, Amazon and Google.
  • About the editors:

This is Thomas Adams

Thomas was recently a Visiting Scientist with the University Corporation for Atmospheric Research (UCAR) at the U.S. National Oceanic and Atmospheric Administration (NOAA), National Weather Service (NWS) National Water Center. His work focused on developing forecast visualization, analysis, and verification methods for the UCAR/NCAR WRF-Hydro based National Water Model and techniques to de-bias ensemble forcings and hydrologic forecasts. Previously, he spent a brief period with the Australian Bureau of Meteorology, where he befriended Tom Pagano (and the world changed forever), thus putting into motion a life-long collaboration! Currently, Tom is forming, a NGO to bring flood and flash flood forecasting to under-capitalized, developing and emerging countries: “A critical aim is to build technical expertise and a culture of understanding to sustain hydrometeorological monitoring, forecasting, and warning locally.Read more about Thomas Adams


This is Tom Pagano, getting his feet wet at the sandbag wall.

The unifying theme of Tom Pagano’s career has been using operational river forecasts to bring the benefits of science to the public. He leads a national flood modelling group within the Australian Bureau of Meteorology. He was also an operational river forecaster with the USDA Natural Resources Conservation Service in Portland, Oregon. He has designed, developed and helped implement forecasting systems on several continents. He also has a special interest in forecast verification and is the lead of a Bureau group for the communication of verification information. Read more about Tom Pagano


Posted in case-studies, interviews, operational systems | 1 Comment

What to look for when using forecasts from NWP models for streamflow forecasting?

By Durga Lal Shrestha, James Bennett, David Robertson and QJ Wang, members of the CSIRO Columnist Team

There have been a few posts on NWP performance lately, and so we thought we’d add our perspective. We’ve been working closely with the Bureau of Meteorology to extend their new 7-day deterministic streamflow forecasting service to an operational ensemble streamflow forecasting service. One of the fundamental choices we have to make is the source of quantitative precipitation forecasts (QPFs). This is not as straightforward as it may seem – QPFs that show strong performance in some ways may not be suitable in others, while even if a hypothetical ‘ideal’ QPF exists, it may not be useable e.g. because it is inconsistent with other forecasts the Bureau issues.

Here’re some of the things we look for in QPFs from Numerical Weather Prediction (NWP) models:

  • QPFs that perform best after post-processing. We use a post-processor to produce ensemble forecasts from deterministic QPFs (we call it the Rainfall Post-Processor or RPP; see Robertson et al., 2013). It’s a little similar to EMOS-type post-processors (see, e.g. Gneiting et. al; 2005) with which you may be familiar. Post-processors are able to correct many of the deficiencies of rainfall forecasts, so we are looking for QPFs that add extra information.
  • QPFs that produce skilful forecasts in a range of catchments. The goal is to apply a single system across all of Australia, so we need to handle all the conditions Australia throws at us: from tropical/monsoonal climates, through arid areas, to cool temperate climates.
  • QPFs that produce skilful forecasts of accumulated rainfalls. Yes, the timing of rain storms is very important, and we of course consider this when we evaluate QPFs. But for streamflow forecasting, the total amount of rainfall (irrespective of timing) forecasted for all lead-times can be crucial. Assessing the performance of accumulated rainfall forecasts – particularly for measures of reliability – is not commonly considered in literature published on evaluating QPFs.

In Australia, the Bureau of Meteorology uses the ACCESS suite of NWP models and the ‘Poor Man’s Ensemble’ (PME) to provide guidance to issue official weather forecasts. The PME is simply a collection of 8 deterministic NWP models. The global version of ACCESS (ACCESS-G) is developed and operated by the Bureau to produce 10-day deterministic rainfall forecasts, while the PME form the major basis of operational weather forecasting products issued by the Bureau. The Bureau asked us to evaluate if ACCESS or PME would be preferable for an ensemble streamflow forecasting system.

After post-processing there is little to separate the performance of PME from ACCESS-G across a range of catchments (Figure 1) – even where there are big differences in the performance (particularly bias) of the raw forecasts, such as for tropical Queensland. As you can see CRPS is almost identical for raw ACCESS-G and PME, but the bias is vastly different. The PME has a tendency to predict too many small rain events because of “smearing out” rainfall from component models in the ensemble mean. These small positive errors help cancel out a few significant negative errors (underestimation) which leads to smaller bias in PME. In contrast, the small positive errors add up with the negative errors when computing CRPS (absolute error for deterministic forecast).

D:\My Publications\Blogs\fig1_bias_crps\fig1_bias_crps.png

Figure 1. Catchment average rainfall and CRPS in the raw and calibrated rainfall forecasts in Tully (Queensland – Tropical) and North Esk (Tasmania – Temperate) catchments

Calibrated QPFs from ACCESS-G and PME are equally reliable at individual lead times (Figure 2). When we consider accumulated rainfall totals, the story becomes more interesting: reliability of calibrated PME QPFs is worse than calibrated ACCESS-G QPFs (thick lines in Figure 2). We suspect that this could be due to autocorrelation in raw PME forecasts. Taking the average of any set of deterministic forecasts will generally give a smoother (i.e., more autocorrelated) rainfall forecast than an individual NWP forecast. This autocorrelation isn’t compatible with the Schaake Shuffle (Clark et al., 2004), which we use to instil temporal and spatial correlations in the QPFs. (Though we’re not sure this is the cause – anyone else come across this problem?) Of course, the benefit of averaging many deterministic NWP models to produce the PME should be an increase in forecast accuracy – but in this case this advantage is only slight.

D:\My Publications\Blogs\fig2\fig2.png

Figure 2. Probability Integral Transform – uniform probability diagrams (PIT plots) for the catchment average calibrated QPFs in Tully and North Esk catchments. Thin lines are for individual and thick lines – cumulative, yellow lines = calibrated ACCESS-G, violet lines – calibrated PME

Of course, in a brief blog post it’s not possible to cover all the aspects of NWP QPFs that interest us: for example, we are also very interested in the streamflow forecasts that are produced by our calibrated QPFs. So we’ll leave you to fill in the gaps:

  • What makes a good QPF for you? What’s important to you when you assess QPFs for streamflow forecasts? Do you look for something others don’t?

North Esk River in Launceston, Tasmania (Temperate/Perennial) (Creative Commons License)

Tully River in Far North Queensland (Tropical/Perennial) (Creative Commons License)

Ord River downstream of Lake Argyle in North East of Western Australia (Grassland/Intermittent) (permission from


Posted in columnist, verification | 3 Comments

FcstVerChallenge: will you join the HEPEX team?

You may have read Florian’s recent post on the WMO’s “forecast verification challenge”. In short: the WMO’s World Weather Research Programme set a challenge to develop new user oriented verification scores, metrics, diagnostics or diagrams. Any entries have to be submitted by the end of October and the winning entry will be awarded with a “keynote” presentation at the 2017 WMO verification meeting in Geneva as well as free passage into that event.


Some HEPEX-ers got together last week and discussed a joint effort to develop a new metric. While in the past, brilliant verification metrics have been developed by individuals (Brier´s probability score) as well as by small groups (CRPS decomposition), we realized that we’re likely to do better if we made this a team effort. More ideas, more points of view, more -and more varied- experience and expertise should all contribute to a better idea. Or to multiple ideas even, maybe.

So, our question to you:


It’ll be a great way to meet new people. It’ll be good to further sustain the HEPEX community. It’ll be a fantastic story if we get a result. In any case, it’ll be fun.

To organise the team effort, we’ve set up a Slack team space that we can use for team communication. Slack is essentially a virtual space where the team can have a conversation. It’s searchable. It can be used in conjunction with tools like Skype video calling, Trello and file sharing tools like Google Drive and Microsoft OneDrive. It’s everything we need to work across space and time.

If you’re interested, drop either Florian, Julie or myself an email and we’ll set you up on Slack so you can start the ride!

Posted in activities, verification | Leave a comment

Ensemble forecasting experiments in a medium size tropical basin using MASTER rainfall forecasts

Contributed by Adalberto Meller and Fernando Fan, members of the LSH Research Group Guest Columnist Team

As we have seem in our first post, great advances occurred in Brazil in the last years concerning the development of flood alert systems, but still today there is a low number of operational systems in terms of territory coverage, especially those using a probabilistic approach.

One of the first researches in Brazil to evaluate ensemble flood forecasts in South America was presented by Meller (2012) and Meller et al. (2014), but only published in Portuguese. Today we are using HEPEX space to present it to the broader community.


Figure 1. Paraopeba basin location.

The research assessed the performance of a short term ensemble flood forecasting system in a medium size tropical basin, the Paraopeba River Basin (Figure 1), based on data and streamflow forecasting tools available in operational mode in Brazil. The methodology consisted in the use of the MGB-IPH hydrological model, coupled to an ensemble of rainfall forecasts generated by several models with different initial conditions and parameterizations.

The weather forecast database used in the study comprised 50 outputs of NWP (numerical weather prediction) models, which differs in the type of model (global or regional), spatial resolution, parameterization, initial and boundary conditions. The preparation of this database came from an initiative of the Meteorologia Aplicada a Sistemas de Tempo Regionais Laboratory (MASTER-IAG/USP Lab) – in cooperation with other centers aiming conducting an intercomparison and combination activity of NWP models (Silva Dias et al., 2006).

Weather forecasts were issued by the centers of many countries once or twice a day (00:00 UTC or 12:00 UTC), with lead times between 48 and 168 hours, generally accumulated every six hours. Upon receipt, weather forecasts are interpolated by the MASTER-IAG laboratory to points corresponding to the latitudes and longitudes of a wide network of surface monitoring gauges in South America (Figure 2).


Figure 2. Location of surface stations where weather forecasts are provided by the MASTER Lab.

Besides the ensemble, a single deterministic streamflow forecast is also given by MASTER Lab, based on a quantitative precipitation forecast derived from the optimal combination of several outputs of NWP models . It was used as the reference forecast to assess the performance of the streamflow ensemble forecasts at the Paraopeba basin.

The aim of the reduction technique applied by MASTER Lab is to get a deterministic forecast with better performance than the arithmetic mean of the whole or any of the ensemble members. The methodology used by MASTER Lab is to assign different weights to each of the ensemble members in the composition, according to their performance in a period of 15 days prior to the forecast.

Using MASTER Lab data, flood forecasts were performed for three rainy seasons (austral summer) between 2008 and 2011. Figure 3 shows the results of forecasts from December 2009. The results from the ensemble flood forecasts were assessed by deterministic and probabilistic performance metrics.


Figure 3. Ensemble forecasts from December 2009 at the Paraopeba river.

General deterministic assessments showed that the ensemble mean have similar performance to those obtained by the deterministic reference forecast (the best forecast according to MASTER Lab), although showing better performance over most of the ensemble members. Based on the probabilistic performance measures, however, results showed the existence of an ensemble overforecasting and underspread of the members in regard to observed values, especially during initial lead times.

Results for predictions of dichotomous events, which tested the exceedance or not of a flood warning thresholds, showed that the 9th decile of the ensemble over performed the deterministic forecast and even the ensemble mean. In most cases, it was observed an increase in the proportion of correctly forecasted events while keeping false alarm rates at low levels. This benefit was generally higher for higher flow thresholds and for longer lead times, which are particularly important situations for flood mitigation.

Figure 4 shows results of ROC diagrams for 12, 24, 48 and 72 hours lead time and for the flow threshold of 623 m³/s, which is an estimated situation of lower channel extravasation of the river Paraopeba.


Figure 4. ROC diagrams at 12, 24, 48 and 72h lead times and flow 623 m³/s (threshold). Squares represent the ensemble percentiles 1º, 5º and 9º. The Red dot is the deterministic reference and the black dot is the ensemble mean.


  • Meller, A. (2012). Short Term Ensemble Flood Forecasting (Previsão de Cheias por Conjunto em Curto Prazo). PhD Thesis.  Federal University of Rio Grande do Sul. Hydraulic Research Institute. 224p.
  • Meller, A., Collischonn, W., Fan, F. M., Buarque, D. C., Paiva, R. C. D., Dias, P., Moreira, D. Short Term Ensemble Flood Forecasting. Revista Brasileira de Recursos Hídricos, v. 19, p. 33-49, 2014.
  • Silva Dias, P.L., Moreira, D.S., Dolif Neto, G. (2006). The Master Super Model Ensemble System (MSMES). Proceedings of 8th ICSHMO, Foz do Iguaçu, Brazil, April 24-28, p.1751-1757.
Posted in case-studies, columnist, ensemble techniques, floods | 2 Comments

A user-oriented forecast verification metric competition

H:\Downloads\wwrp.jpg Forecast performance is one of the most central themes not only in day-to-day weather forecasting, but also in HEPEX.

It is so important that we have devoted an entire chapter in our science and implementation plan to it (see here). I am, in particular, often forwarding the link to these blog posts when I am explaining (or trying to explain) forecast properties to a forecast user.

Nevertheless, many of the scores remain abstract. Whilst a forecast bias may still be easy to communicate, trying to get across what a root mean squared error is, is already far more challenging – and I haven’t even started with probabilistic scores.

There is no question that we need these scores to optimize and develop our forecasting systems, however, they are a “communication nightmare”. In HEPEX, we have developed games in trying to easy this communication (remember the peak box game from the HEPEX meeting in Maryland and, recently, in Quebec?).

Therefore, it is great that this communication nightmare is now also recognized by the verification component of the World Weather Research Program. They issued the challenge to develop and demonstrate the best New User-Oriented Forecast Verification Metric.

The challenge is cross-cutting and cross-disciplinary. It considers all applications of meteorological and hydrological forecasts that are relevant to user sectors such as agriculture, energy, emergency management, transport, etc. The metrics can be quantitative scores or diagnostics (e.g., diagrams), but they must be new to be considered for the prize.

  • Do you have an idea to propose?
  • Do you already use a score which would be ideal to the WWRP challenge?
  • Do you have a very specific user who would benefit from a very specific score?

If so, then join in and submit your entry here. For more details on the challenge visit the webpage here.

Posted in announcements-events, forecast users, verification | 1 Comment

Forecasting over international borders: limitations and solutions for large-scale or continental forecasting systems

Contributed by Chantal Donnelly (SMHI), member of the SMHI Guest Columnist Team

Global and continental forecasting schemes already exist and are used to inform disaster management in countries without sufficient national forecast systems of their own, as inputs to operational oceanographic models and for the general interest of citizens. I have been lucky enough to have worked with two operational European forecasting systems (setting up of E-HYPE and the WET tool, as an operational EFAS forecaster and testing E-HYPE in EFAS). My colleagues have also just recently set up a forecast system for the Arctic basin, Arctic-HYPE. So, I thought I’d reflect on some of the challenges specific to international forecasting.

Unlike national or subnational forecast systems, which often have access to their own country or region’s collected hydrometeorological data and expertise, international forecasting systems have to do with inhomogenous collations of data from different countries. There can be huge differences in how neighbouring countries contribute to international databases!

Historical collations of daily precipitation and temperature observations are improving, for example the E-OBS product in Europe, but data coverage both in time and space tends to be fairly uneven. Similarly, the global runoff data centre (GRDC) provides a fantastic service in collating and disseminating river discharge data around the world, but again, not all discharge data is available for all periods in this data set either.

So, what are some of the limitations for international forecasting and how can these be solved? Here are just a few:

  • There is often no single forcing data set that is consistently better than others. For example, in Europe, variations in precipitation and temperature gauges used in different data sets mean that quality varies regionally (e.g. Fig 1)
Fig.1. Row 1: Percentage difference between the mean and coefficient of variation of precipitation between gridded observations (5 km) and WFDEI and, Row 2: for discharge simulated using the gridded 5 km observations and WFDEI for the period 1991 to 2010.

Fig.1. Row 1: Percentage difference between the mean and coefficient of variation of precipitation between gridded observations (5 km) and WFDEI and, Row 2: for discharge simulated using the gridded 5 km observations and WFDEI for the period 1991 to 2010.

  • The available historical forcing sets are often only available to a fixed period (e.g. 2013), so can’t be used for hydrological model spin-up and initialisation.
  • This leads to a mix of model based forcing (reanalysis/forecast) and observation based forcing (interpolated observations) to calibrate, spin-up/initialise and run forecasts. As a result, potentially 3 or more different forcing sets can be used to make a forecast, e.g. (i) calibration to best available historical forcing, (ii) spin up with reanalysis and/or saved forecasts (iii) forecasting with ensemble or deterministic forecast model.
  • Real-time discharge data is generally unavailable for assimilation into the forecast model, so initial states are only as good as the calibrated hydrological model and forcing data.

We have begun testing solutions to these issues in our operational international forecasting at SMHI.

To secure a continuous  forcing data set that can continue from an  historical reference period until near-real-time, we created GFD (or global forcing data). This in an operational product that can flexibly correct a gridded reanalysis or forecast grid to gridded observations. By flexible we mean that either the gridded model data or observations can be interchanged so that when an historical reanalysis such as ERA-INTERIM is not available (e.g. typically at t-3 months), this can be replaced by saved deterministic forecast data (DFD). GFD corrects both ERA-INTERIM and the saved forecasts to gridded monthly mean precipitation (e.g. GPCC), ensuring continuity of the forcing data set from calibration to model initialisation. We are also now testing to see if we could similarly exploit our knowledge of the biases between forecasts (saved) and best available historical forcing data to correct our meteorological forecasts (e.g. Fig. 2).

Fig 2. Percentage difference between gridded observations (5 km) and saved deterministic forecasts at the E-HYPE model subbasin scale for the period 2010-2014. Note the large positive biases in the forecast over northern Europe and local negative biases in extremes over most of continental Europe as well as strong local biases for more extreme precipitation events (P99).

Fig 2. Percentage difference between gridded observations (5 km) and saved deterministic forecasts at the E-HYPE model subbasin scale for the period 2010-2014. Note the large positive biases in the forecast over northern Europe and local negative biases in extremes over most of continental Europe as well as strong local biases for more extreme precipitation events (P99).

This could potentially be useful not only for homogenising input forcing data to international forecast systems, but possibly also for bias-correcting forecasts in all hydrological forecasting systems.

We are also investigating remote sensing solution for model initialisation including altimeter measurements of surface water bodies, and satellite derived snow extent and depths. Our ultimate goal is to make international forecasting more useful despite the limitations that international borders sometimes present!


  Calibration Spin-up Forecasts Comments
WET: Water in Europe Today WFDEI to 2013 Saved ECMWF deterministic forecasts ECMWF deterministic forecast

Currently being updated to GFD

Arctic-HYPE GFD GFD-GPCC first guess + saved ECMWFW det. forecasts corrected to GPCC ECWF deterministic forecast

The spin-up uses 3 different combinations of gridded reanalysis/forecast and gauge data as data becomes online/available

Read more here.

Posted in columnist, data systems, operational systems | 2 Comments

Reflections on the 2016 HEPEX workshop (6-8 June 2016)

Contributed by QJ Wang, Maria-Helena Ramos, Andy Wood, François Anctil, Antoine Thiboult, Dirk Schwanenberg and Rodolfo Alvarado Montero

What a fantastic workshop! And this was certainly thanks to all the participants that you can see in the group photo below.


Participants to the workshop

The workshop was attended by about 85 people from 16 countries, with about 60 oral and posters presentations. The programme and the presentations that were made available by the authors can be seen here.


François Anctil (Université Laval) welcome participants on the first day

Many of the workshop attendees were connected through WhatsApp, so they could easily meet outside the workshop hours. Some early morning participants, for instance, were ready to go running at 6am, although not everyone responded to these calls after late nights, plenty of relaxed conversations and some beers.

In the programme, there was, of course, the highly anticipated game, this year featuring the economic value of ensemble forecasts. It was presented by Micha Werner, and we are looking forward to hearing more about the results (note: he publicly promised to provide a blog post as soon as the game sheets collected are analysed!).

The game will also join the Hepex resources page soon, and thus be available for the community to play it during meetings and courses.


Micha Werner presenting “The game of making decisions under uncertainty: How sure must one be?”


Massimiliano Zappa in front of his poster

Talking about games, we also had the opportunity to play again the peak-box game, proposed by Massimiliano Zappa, which was played in a plenary session at the Hepex workshop in Maryland (USA) in 2014. This was a nice example of interactive poster, where 42 participants were able to try their chances (and test their wisdom) in forecasting the peak and timing of a real flood event. In this PPT you can follow the diversity of opinions and our winner (thanks, Massimiliano, for sharing it with us!).

It was also pleasing to hear the significant progress made, since the Norrköping workshop, in setting up a testbed for inter-comparisons of techniques, methods and systems for seasonal streamflow forecasting. The topic is progressing fast within HEPEX, with notably a Special issue being prepared in HESS (check here for open submissions). If you want to participate to the experiment, contact Andrew Schepen (CSIRO) or Andy Wood (NCAR).

Also about experiments and testbeds, this year the Hepex workshop was followed by a break-out session. About 15 people attended the side event called “Assimilate your basin”. It was organized by Dirk Schwanenberg, Albrecht Weerts, Rodolfo Alvarado Montero and Peter Krahe. The participants provided 8 case studies to a testbed for data assimilation in hydrological models, conducted hindcasting experiments and assessed the forecast performance. The group discussed the common interest in data assimilation and the possibility of setting up a related HEPEX inter-comparison experiment in the near future. Further news will come soon in the Hepex Portal (or contact the session organizers if you are interested to learn more about it).


Break-out-session “Assimilate your basin”

Science, operations and applications

Three themes emerged from the workshop:  science, operations, and applications.

  • The last theme, applications, featured very strongly compared with previous workshops. There were a number of presentations on the use of forecasts in decision-making, and how this resulted in actions and impacts. Methods for evaluating the value of ensemble forecasts were contrasted, and showed how social structure and communications play an important role in achieving impacts. Many case studies, including from hydropower corporations and flood emergency and water management agencies, demonstrated the benefits and complexities of using ensemble forecasts.
  • On forecasting operations, how forecasters interact with forecasting models drew much interest once again – If one was not “in the loop”, one could certainly be just as effective or even more so operating “over the loop”! Challenges and successes in forecasting operations and water management for trans-boundary water systems such as the Great Lakes were highlighted. Practical and yet significant issues were discussed on how to maintain consistency in streamflow forecasting services when weather and climate forecasting models were frequently updated. Attention was also drawn for consistency in forecasts and forecasting methods across regions for operating large energy grids, for which hydropower can play a significant role. What integration of sciences and technologies could achieve was ably demonstrated by success in developing highly sophisticated forecasting systems for complex urban regions.
  • On forecasting science, new methods and successes in real applications of data assimilation were presented. The possibility of doing streamflow forecasting within NWP models is getting real, especially when these NWP models could be calibrated by new solvers developed by hydrologists. Downscaling of weather and climate forecasts was always going to be an area of strong interests, but this year Shaake Shuffle became a ground for fertile research, with a number of variants proposed to make it working even better! Different ways to handle hydrological uncertainty, being through representing multiple initial conditions, multiple hydrological models, or total residues, were showing progress towards achieving reliable ensemble forecasts.

A perfect score


Quebec city on a sunny day

The workshop was a great success – with a perfect “energy score” of 100!

We thank all the key note speakers, speakers, poster presenters, and all attendees, for making the workshop such a success. In particular, we thank all those that participated to the workshop organization for running such a wonderful workshop – with perfect accuracy and little uncertainty!

We also thank the sponsors of the workshop for the excellent facilities and warm hospitality.

The charm of Quebec city will last in the memory of attendees for a long time.


Posted in activities, meetings | Leave a comment