Ensemble prediction: past, present and future

Posted on September 19, 2017 by Fredrik Wetterhall

Contributed by Fredrik Wetterhall and Roberto Buizza, ECMWF

The work of producing meteorological ensemble forecasts started 25 years ago at ECMWF and NCEP, and it sparked a revolution in both weather forecasts and its many applications. To celebrate this occasion, more than 100 people from across the world joined the 28 speakers at ECMWF’s Annual Seminar 11-14 September held in Reading, UK. The theme was “Ensemble prediction: past, present and future” and the four days where filled with presentations and discussions on what has been done, where we are and how we in the future can further improve the accuracy and reliability of ensemble-based forecasts.

Thanks to advances in models, data assimilation schemes and the methods used to simulate initial and model uncertainties, today ensembles are widely used to provide a reliable estimate of possible future scenarios. This is expressed for example in terms of probabilities of weather events or of risk indices. Increasingly, ensembles are routinely used to provide forecasters and users with the range of weather scenarios that could happen in the future. An example is given by the ECMWF ensemble-based strike probability of hurricane Irma, issued by ECMWF on 5 September.

The ECMWF ensemble-based strike probability that hurricane Irma would pass within a 120 km radius during the next 10 days, issued on the 5th of September (left panel).

Using ensemble forecasts

Different aspects of ensemble forecasting were discussed during the seminar, and they included the history and theory of ensemble forecasting, initial conditions, model uncertainties, error growth, predictability across scales, verification and diagnostics and future outlook. The full programme including recordings of the talks can be found here. The theme that may be of most interest for the HEPEX community was devoted to applications of ensemble forecasts. The session discussed the various ensemble products that now exist to help decision making (David Richardson, ECMWF), hydrological ensembles including the HEPEX experience (Hannah Cloke, Reading University) and observing and supporting the growing use of ensemble products (Renate Hagedorn, DWD). The session was testament as to how mainstream ensemble forecasts have become, not only in science but also in institutions and authorities that use probabilistic information in decision-making. There is still a lot to do to overcome some of the existing barriers, but the acceptance of ensemble forecast is truly a success story.

Panel discussions and looking forward

The seminar also included a panel discussion which provided an opportunity to explore and discuss in more detail some of the fundamental questions that are currently being tackled by the community, such as:

  • Should we be moving to small ensembles at high resolution, or large ensembles at more moderate resolution?
  • If the most cost-effective ensemble structure changes with lead time, should our ensemble be built so as to give a resolution and ensemble size that changes with lead time?
  • If an ideal ensemble consists of a set of equally likely members, is there a role for an unperturbed/central forecast?
  • What do we expect from the future in terms of our ability to represent model error in ensemble systems, and the representation of perturbations more generally?

It can be interesting to report some of the comments raised during the lively panel discussion:

  • Some users would react also at small probabilities: they would be the ones benefiting more from a size increase;
  • Ensemble size very is important both for the extended/long ranges and for high-resolution ensembles, to be able to capture the fine-scale details;
  • Considering the range of users of the ECMWF ensembles, overall, a size of 50 seems about right; although ECMWF principal aim should be to provide the best raw ensemble forecasts, it should work with the users to develop calibration methods, and understand whether the balance between ensemble size and resolution should be revisited once calibration methods are more widely used;
  • ECMWF should aim to provide the national meteorological services and its users with ensemble-based probabilistic forecasts that could be used by a wide range of users; it will be then up to the national meteorological services and/or third parties to design ‘tailored’ ensemble configurations that can address the needs of specific users;
  • We need more observation-based diagnostic to understand model error, and design better schemes;

Participants of the ECMWF Annual Seminar 2017. Photo: Simon Witter, ECMWF

The HEPEX community was an early advocator of using ensemble forecasts and it is important that we continue to push the boundaries of how ensembles should be used in research and applications. A good way of doing just that is to come to the HEPEX workshop in Melbourne next year!

Posted in activities, data assimilation, ensemble techniques, forecast techniques, forecast users, historical, meetings, operational systems, verification | 1 Comment

Final call for abstracts: 2018 HEPEX workshop in Melbourne, Australia

Posted on September 12, 2017 by Fredrik Wetterhall

As you may have heard, the 2018 HEPEX workshop in Melbourne is coming up soon (Feb 6-8, 2018). Abstracts are due for submission on Sep 30, 2017. The workshop will feature both oral and poster presentations. The theme for the workshop is ‘breaking the barriers’ to highlight current challenges facing ensemble forecasting researchers and practitioners and how they can (and have!) been overcome. We wish to highlight the following barriers:

  • using ensemble forecasts to improve decisions in practice
  • extending forecasts in space (including to ungauged areas) and across lead-times, from short-term to sub-seasonal to seasonal forecast horizons
  • using ensemble forecasts to maximize economic returns from existing water infrastructure (e.g. reservoirs), even as inflows and demand for water change
  • using ensemble forecasts to improve environmental management of rivers
  • applying ensemble forecasts for agriculture
  • searching for better/new sources of forecast skill
  • balancing the use of dynamical climate and hydrological models with the need for reliable ensembles
  • communicating forecast quality and uncertainty to end users

More generally, we welcome contributions on new and improved ensemble hydrological prediction methods, as well the application of existing methods in practical and operational settings.

Keynote speakers for the workshop have been finalised – you can check out this and other information on the workshop website.

The HEPEX workshop is a highly effective forum for exchanging ideas and experiences on all things hydrological forecasting, and registration is free. So get to submitting those abstracts!

Any questions? Please contact us!

Posted in activities, announcements-events, meetings | Leave a comment

End-To-End Probabilistic Impact Based Early Warning Systems for Community Resilience

Posted on September 5, 2017 by Fredrik Wetterhall

Contributed by Dr. Bapon Fakhruddin, New Zealand

Recently I attended the Fourth Pacific Meteorological Council (PMC) and Second Pacific Meteorological Ministers Meeting (PMMM), which was held in Honiara, Solomon Islands, from 14 to 17 August.

Figure 1. Dr Bapon Fakhruddin at the meeting. Photo credit: Jenny Davson-Galle

Reaching communities and ensuring that those most in need are provided with effective communications and technologies are top priorities for the Pacific Meteorological Council (PMC). Science has in-build uncertainty and is highly probabilistic. The probabilistic ensemble forecasting approach exposes the range of uncertainty associated with different predictions. They also allow the adoption of a risk-based approach for decision making and contribute to building confidence to help operational forecasters. Effective early warning systems (EWS) require a complete understanding of the populations and assets exposed to threats linked to the probabilistic ensembles forecasts.

With present extreme weather events, risk-based early warning systems are essential. Practice shows that people and communities at risk need to be involved in the understanding of their exposure and the vulnerabilities of different groups, including the disabled, the elderly, children and pregnant women. An effective system also relies on expert risk assessment, interpretation and communication. Currently, in many places, we do not use probabilistic risk assessment. Since science information is probabilistic, risk assessment needs to follow the same path.

Figure 2. Participants at the meeting. Photo credit: Jenny Davson-Galle

Understanding the forecast

Research has shown that, before deciding to take a disruptive – and often expensive – action such as evacuation, people must understand the forecast; they must believe that it actually applies to their situation and, most importantly, they must feel that they need to act because people, including their loved ones, are at risk. However, in many cases, common practice has been to prepare and release forecast messages without adequate concern on how they are received, understood and/or interpreted. Accurate, appropriate information that translates early warnings into early actions at community level is essential.

“We’ve always been talking about reaching the last mile, and that means getting to the people who haven’t got the message we are relaying. We’re talking about people with disabilities as well. They need to be included in our conversations and awareness efforts too,” said the Secretariat of the Pacific Regional Environment Programme’s (SPREP) Climate Change Advisor, Espen Ronneberg. “We do this to prepare those who are vulnerable to disasters as well, and that includes people with disabilities.” was added.

Acting on the risks

Disabled and elderly people are particularly at risk from natural disasters as, even with strong family and community support systems, it takes longer for them to reach designated safety zones. Likewise, extra forward planning is required for the evacuation of hospital patients and other health care facilities. Ronneberg believes that the way forward lays in encouraging disabled people to join in on EWS (Early Warning Systems) discussions. “I think the best way to include them would be through the People with Disabilities’ Forum, and it will be great if we can get them to take an interest in meteorology as well,” he said.

The PMC session included a discussion on new forms of risk assessment, such as the shift from deterministic to probabilistic risk estimation. Deterministic approaches are used to assess the impacts of a specific natural hazard scenario, whereas probabilistic methods are used to produce more refined estimates of how often a hazard is likely to happen, and the potential damage it will deliver, with the help of modelling tools. Probabilistic assessments work with uncertainties, partly due to the random nature of natural hazards, and partly because of our incomplete understanding of natural hazards and the limited ways of measuring hazards, exposure and vulnerability (OECD, 2012).

Communicating probability information

As hazard information is always probabilistic, the risk information and risk communication also need to be probabilistic. When any new probabilistic forecast product is introduced, it can be mis-communicated to the affected people. For people to make good decisions, the capacity to generate an early warning with an acceptable lead-time is essential. For example, advances in tropical cyclone (TC) forecasts using ensemble methods have been widely used for operational TC tracking. By using simple, weighted, or selective methods, TC tracking forecasts tend to have smaller positional errors than single model–based forecasts.

Figure 3 End-to-end early warning systems.

The impacts of climate variability and change were also recognised at the meeting as major challenges to island nations. Of particular concern to the Pacific region were sea level rise, salt water intrusion, drought, flooding, coastal inundation, ocean conditions (tides, swells, waves, acidification) and impacts on health (e.g. malaria and dengue), water resources, agriculture and fisheries (invasive species, etc.).

WMO’s approach to Climate Risk and Early Warning System (CREWS) initiatives and the requirements for disaster loss data standardization is crucial for impact-based early warning system, which offers more accurate risk assessment. See also a summary the outcomes of both the Multi-Hazard Early Warning System and the Disaster Risk Reduction Global Platform Meetings (http://www.wmo.int/earlywarnings2017/) held in May in Cancun, Mexico.

The next Pacific Meteorological Council (PMC) meeting will be held in Samoa in 2019. The PMC consists of members of the Pacific National Meteorological and Hydrological Services supported by its technical partners, regional organisations, non-government organisations and private sectors.

The 14-17 August meeting was co-hosted by the Government of Solomon Islands, the Secretariat of the Pacific Regional Environment Programme (SPREP) and World Meteorological Organisation (WMO).

 

Posted in meetings | Leave a comment

Quiz: Can you guess the city by looking at its river from space?

Posted on August 29, 2017 by Maria-Helena Ramos

Contributed by Calum Baugh, Maria-Helena Ramos and Florian Pappenberger

Here are eight cities (and their rivers) seen from Google Earth. Can you recognize them?

Since nobody seems to have guessed the quiz we had in a previous post, we provide for each city/river some clues. Additionally, keep in mind the general clue for all of them: there is at least one Hepex member living in (or very close by) each of these cities (you don’t need to guess who they are…)

River 1: ‘Flooding often coincides with high tides in this city, its barrier can protect against storm surges…but for how long?’ Check the answer here

River 2: ‘The river drains the second largest lake in this country, industry thrived along its banks leading to the nickname Little Manchester.’ Check the answer here

River 3: ‘This river drains the lake which is eponymous from this city, flowing northwest, its steep gradient means there are 10 hydroelectric stations along its route prior to its confluence.’ Check the answer here

River 4: ‘Apparently, here, (non-hydrologists) tourists are often confused about the terms “right bank” and “left bank” and may spend hours trying to figure out which side of the river they are standing on.’ Check the answer here

River 5: ‘Heavy rain and catastrophic flooding was particularly observed in September 2013. They say the event went quickly from bad to worse. It was eight days, 1,000-year rain, 100-year flood.’ Check the answer here

River 6: ‘The canals somewhat dwarf the river here; perhaps hepexers will be more familiar with the beer to which the river lends its name.’ Check the answer here

River 7: ‘Flooding occurred along this river in May 2017 when nearly double the April rainfall average fell; however this city was spared any damage, with the worst affected being the cities upstream.’ Check the answer here

River 8: ‘Heavy flood protection prevents many floods in this city, but in 2014 the flooding resulted primarily from moderate rainfall combined with 111 km/h winds ‘pushing’ water upriver.’ Check the answer here

So, how many cities have you guessed right?

Posted in activities | 2 Comments

Hydrologic similarity: Bridging the gap between hyper-resolution and hydrologic ensemble prediction

Posted on August 15, 2017 by Andy Wood

Contributed by:  Nate Chaney (Princeton University) and Andy Newman (NCAR)

The ever-increasing volume of global environmental data and the continual increase in computational power continue to drive a push towards fully distributed modeling of the hydrologic cycle at hyper-resolutions (10-100 meters) [Wood et al., 2011]. In principle, this has the potential to increase model fidelity and lead to more locally-relevant hydrologic predictions (e.g., soil moisture at the farm level).

However, for the foreseeable future, due to computational constraints this modeling approach will not be suitable for large ensemble frameworks—a pre-requisite for reliable operational applications given the unavoidable uncertainties in model structure, model parameters, and meteorological forcing.

However, this does not mean that hydrologic ensemble prediction should continue to rely on over-simplistic hydrologic models simply to maintain computational efficiency. It is undeniable that providing field-scale hydrologic predictions can have the potential to significantly advance the use of hydrologic models (e.g., precision agriculture).

Furthermore, the important role of the physical environment and human management in hydrologic response necessitates a more explicit representation of the spatial drivers of heterogeneity in hydrologic models. Therefore, there is a need for a modeling approach that can provide field-scale predictions while approximating the computational efficiency of existing hydrologic models used in ensemble frameworks. Contemporary applications of hydrologic similarity can satisfy both objectives.

Hydrologic similarity

Hydrologic similarity aims to harness the observed covariance between a system’s physical environment (topography, soil, land cover, and climate) and its hydrologic response to assemble robust reduced-order models.

Although originally limited to one-dimensional binning of over-simplistic metrics of hydrologic response (e.g., topographic index), recent advances have taken hydrologic similarity a step further; a system’s most representative hydrologic response units (HRUs) are defined by clustering the high dimensional environmental data space [Newman et al., 2014]—the petabytes of readily available high resolution global environmental data make this feasible over the globe. Semi-distributed models can then be built to simulate these HRUs and their spatial interactions (e.g., HydroBlocks, Chaney et al., 2016).

Within the clustered spatial domain, each fine-scale grid cell (~30 meters) is associated with a specific HRU through its environmental characteristics. This then makes it possible to map out the HRU simulations onto the fine-scale grid to approximate the fully distributed simulation (see Figure 1 for an example).

Using this approach, ongoing work continues to show that the fully distributed simulation can be closely reproduced using around 1/1000 of the number of HRUs; each grid cell is a unique HRU in the fully distributed simulation. In other words, the semi-distributed model can effectively provide the same hydrologic information as the hyper-resolution fully distributed model with only a fraction of the computation.

This equates to being able to run roughly 1000 ensemble members of the semi-distributed model in the time it would take to run the fully distributed model once; all while being assured that each ensemble member closely approximates its corresponding fully distributed solution.

In summary, contemporary implementations of hydrologic similarity provide a unique opportunity to bridge the gap between physically-based hyper-resolution modeling efforts and hydrologic ensemble prediction. It enables robust ensemble frameworks to provide locally-relevant information while ensuring they can robustly characterize the unavoidable uncertainties due to model structure, model parameters, and meteorological forcing.

References

  • Chaney, N., P. Metcalfe, and E. F. Wood (2016), HydroBlocks: A Field-scale Resolving Land Surface Model for Application Over Continental Extents, Hydrol. Process., doi:10.1002/hyp.10891.
  • Newman, A. J., M. P. Clark, A. Winstral, D. Marks, and M. Seyfried (2014), The Use of Similarity Concepts to Represent Subgrid Variability in Land Surface Models: Case Study in a Snowmelt-Dominated Watershed, J. Hydrometeorol., 15, 1717–1738.
  • Wood, E. F. et al. (2011), Hyperresolution global land surface modeling: Meeting a grand challenge for monitoring Earth’s terrestrial water, Water Resour. Res., 47(5).
Posted in ensemble techniques, forecast techniques, hydrologic models | Leave a comment

Quiz: Can you guess the river from the space?

Posted on August 1, 2017 by Maria-Helena Ramos

Contributed by Calum Baugh, Maria-Helena Ramos and Florian Pappenberger

Here are four rivers seen from Google Earth. Can you recognize them?

River 1:

Check the answer here

River 2:

Check the answer here


River 3:

Check the answer here

River 4:

Check the answer here

 

Posted in activities | Leave a comment

Which scales matter for water resources management?

Posted on July 18, 2017 by Maria-Helena Ramos

Contributed by Andreas Hartmann, Axel Bronstert, Bettina Schaefli

The discussion about which scale is the most relevant for water resources management is an increasingly important debate of hydrological modelling principles over the last decade.

The session “(Ir‑)relevant scales in hydrology: Which scales matter for water resources management?” convened at this year’s EGU General Assembly 2017 tried to put some new light (and fire) into this ongoing debate.

Photo taken during the session at EGU 2017 by Axel Bronstert

Solicited speakers representing the plot scale, hillslope and catchment scale, and the large-scale provided valuable insights into their research experience and provided opinions about the session topic.

  • Hans-Joerg Vogel provided a list of questions that may be answered at the plot scale. For instance, to what detail do we need to know the basic soil hydraulic properties? And how useful are sophisticated lab measurements to predict what is happening in the field? Also, methods to handle the highly non-linear change in flow paths as a function of the hydraulic state and its history may be developed at the plot scale. However, how soil water dynamics integrate over the scales of hillslopes or catchments is still an open question cannot solely be answered at the plot scale. How far can approaches like Richard’s equation that were developed at the plot scale be transferred to those larger scales?
  • This question was picked up by the following presenter, Theresa Blume, who advocated for more research for understanding the link between plot and catchment scales. Nested monitoring programmes with monitored catchments that envelop different monitoring plots, as applied within the Catchments As Organised Systems (CAOS) project, may provide promising advances in our understanding how plot scale dynamics integrate to hillslopes and catchments. When such measurement design is applied in a comparative approach, generalized knowledge about hydrological processes across different types of hydrological landscapes could be obtained. But how to integrate the derived understanding into models that can be applied at test sites with less information?
  • The modeller’s point of view was picked up by Jens Christian Refsgaard. Stating that it It is recognised that there is a mismatch of spatial scales between our process knowledge, the modelling grids of our distributed catchment models (50 – 500 m) and the water management problems, where the relevant scale often is claimed to be the catchment (e.g. 10 – 5000 km2). Groundwater pollution may even occur on larger scales (several 10,000 km²) while mitigation measures have to be applied locally. Hence, relevant scales vary from one issue to another (plot to large scale) and for some issues (e.g. mitigation measures) are in the order of 100 m when for instance the European Water Framework Directive (EWFD) is applied. Although distributed models are technically able to simulate the hydrological behaviour on such small scales, how can we evaluate them for their realism if now operations are available?
  • The challenge of evaluating hydrological models that operate on scales even larger than the catchments scale was also picked up by Thorsten Wagener. We increasingly build and apply hydrologic models that simulate systems beyond the catchment scale. Such model can provide opportunities for new scientific insights, for instance, the consideration of intercatchment flow. Also, large-scale models can help us to understand changes to water resources from larger scale activities like agriculture or from hazards such as droughts. However, these models also require us to rethink how we build and evaluate them given that some of the unsolved problems from the catchment scale have not gone away. So what opportunities for solving these problems are there? Are there possibilities that have not yet been utilized?
  • An increasing source of information for large-scale model applications are global archives of hydrological observations, as stated by Lena Tallaksen. These may allow a more detailed development and evaluation of hydrological models for the purpose of water resource assessments and climate change impact studies at the global and continental scale. Recent research has been providing improved knowledge of the present state of global water resources and variability across large spatial domains, the role of terrestrial hydrology in earth system models, the influence of climate variability and change on continental hydrology (including extremes), and the representation of subsurface hydrology and land-surface atmosphere feedback processes. Large-scale models are more and more adapted to include multiple types of input data such as remote sensing processes. However, a lack of ground truthing especially in less developed regions, the representation of hydrological variability below the modelling scale and uncertainties in downscaling large-scale climate forcings to the model scale still limit the applicability of large-scale models. Despite these challenges, large-scale models may represent a useful source of information for continental-scale hydrological assessments and evidence-based policy making. To increase their reliability, transfer of knowledge across scales is essential to improve hydrologic predictions at different spatial scales in an ever-changing world.

Overall, by the excellent presentations and discussion during the session, we found that each scale has its own relevance for water management. Experimental research at the plot and catchment scales brings advances in hydrological process understanding that can improve our simulation tools, while comparative hydrology and large scale modelling can provide quantitative information at larger scales to support water governance and policy making.

The key question in the near future is certainly how to further improve collaboration and foster discussions across all hydrological scales, especially in the context of ever more complex models at all scales. One option would certainly be to organize regular cross-cutting conference sessions. Another interesting initiative that has been presented at EGU2017 is the data and model sharing platform https://www.hydroshare.org/.

Posted in activities, announcements-events, water management | Leave a comment

The role of Early Career Scientists in community research

Posted on June 28, 2017 by Florian Pappenberger

Contributed by Florian Pappenberger and Maria-Helena Ramos (both considerably beyond the early career stages, they admit)

(this post can also be seen in the Young Hydrologic Society Portal)

Science and forecasting practice are the foundations of the HEPEX community. These are certainly the routine of many of us during our office hours and while spending time in front of your computers.

But this community is also based on individuals, and this is often what really makes it fun to go to meetings, workshops and conferences. Face-to-face interactions often bring new ideas into form (see also this previous post from CSIRO team), while also helping us to further develop interpersonal skills.

It is thus not really a surprise when members of the community get together after meetings and write (sometimes successful) research proposals together. One example is the IMPREX project, funded under the EU H2020 programme, where many partners already knew each other from HEPEX before participating in IMPREX.

The ‘EX’ that HEPEX and IMPREX have in common does not express the same idea at all, but several challenges of HEPEX are part of the research tasks of IMPREX:  for instance, improving hydrological models and data assimilation for forecasting extremes, or estimating the economic value of forecasts in the water sector.

IMPREX stands for IMproving PRedictions and management of hydrological EXtremes. The project targets improving the quality of short-to-medium hydro-meteorological predictions, enhancing the reliability of future climate projections, applying this information to strategic sectoral and pan-European surveys at different scales, and evaluating and adapting current risk management strategies.

But this post is not about IMPREX. What in fact has (gladly) attracted our attention in this project and we would like to talk about here is the active participation of early career scientists (ECS).

Intergenerational engagement in research projects

Many research programmes call for the multiple benefits of stakeholder and public engagement, but what about “intergenerational engagement” between the early career scientists and the well-established ones?

If we look closely, the age group distribution in IMPREX, and in HEPEX as well, is extremely diverse. It ranges from early career scientists (usually MSc or PhD candidates) to well-established researchers, both sides with particular skills (from analytical to computer programming skills) and viewpoints (for carrying out science experiments but also succeed in multi-cultural team leadership, for instance). This diversity is not always fully explored in research projects, but the case seems to be different in IMPREX.

ECSs in IMPREX have tasks and are in charge of presentations in all project meetings. They have their place in the agenda and are encouraged to get involved in the consortium. They also have their place in the project’s online blog to communicate anything they want: their science achievements, activities, new discoveries, participation in meetings or just general reflections.

They produce posts about topics which are interesting for them and describe them from their perspective. New perspectives are always exciting to read. And new perspectives which are untainted by ‘old’ ideas (sometimes disguised under what is called ‘experience’) are even more interesting.

So far, there have been a number of IMPREX ECS blog posts that are closely related to HEPEX topics and certainly worth reading:

  1. An ecologist’s viewpoint of hydrological forecasting (as far as we can remember, we never had a blog post in HEPEX from that angle – read the discussion!)
  2. An interesting post presenting an insider’s point of view of four forecasting services active in Germany, The Netherlands, Spain and at the pan-European scale.
  3. A flood decision-making experiment, where we can step in the boots of a flood manager with the help of IMPREX ECSs.
  4. And a friendly report of their participation in the 2017 EGU General Assembly (notably, check the very nice way they present their photos at the end of the post!)

We are eagerly awaiting the next ones!

Posted in activities | 2 Comments

Risk aversion and decision making using ensemble forecasts

Posted on June 6, 2017 by Marie-Amélie Boucher

by Marie-Amélie Boucher and Vincent Boucher

Assessing the value of forecasts is a very popular topic among the HEPEX community. The assessment of forecast value is highly dependent on the purpose served by the forecasts. For the specific problem of decision-making related to flood mitigation, Murphy (1976, 1977) proposed the use of the cost-loss ratio framework. The vast majority of papers related to the assessment of forecast value for flood mitigation adopt this framework, so one could think that everything is pretty much solved… except that the cost-lost ratio has very important flaws!

One such flaws is the fact that the cost-loss ratio assumes that the decision maker is risk neutral. Risk neutral individuals only care about the expected outcome, and disregard the spread of the distribution of outcomes. Risk neutral individuals are very rarely encountered in real life. Indeed, most of us are risk averse. That is, for the same expected outcome, we prefer less risky distributions. This is (among other things) why we buy insurance. Informally, most people dislike risk and would be willing to spend resources (e.g. money) in order to reduce the amount of risk faced.

Utility theory

Economists (and statisticians and mathematicians) have been studying those issues for a long time, and came up with many models and concepts which could be used in hydrology, as an alternative to the cost-loss ratio. The central framework is based on “utility theory”. To put it’s development a bit in a context, here is a (very) pseudo-historical adaptation of a conversation between Nicolau I Bernoulli and his cousin Daniel [1]:

It didn’t happen exactly like that, but the general idea is preserved. The game that Nicolau is referring to is now known as the St-Petersburg paradox and was first described in a letter. The first person to present it more formally was Daniel Bernoulli in 1738. He was also the first to suggest the idea of risk aversion and exposed (without any mathematics) that different persons faced with the same decision problem and the same information could take different decisions, because of different preferences.

It is only much later, in 1944, that the concept of risk-based individual preferences was formalized as a mathematical theory, by John von Neumann and Oskar Morgenstern [2]:

Utility theory is not perfect (and indeed, many generalizations and extensions exist), but it has at least two advantages over the cost-loss ratio:

  1. It allows for decisions to consider a finite number or a continuum of decisions (e.g. “protect” vs “don’t protect” for floods, or alternatively, “amount spent in protection”).
  1. It allows to account explicitly for the decision maker’s level of risk aversion in the assessment of forecasts value.

In hydrology

Krzysztofowicz suggested using utility theory in hydrology as early as in 1986. It is not a miraculous solution to the problem of assessing the value of forecasts for flood mitigation, but it would be an improvement over the cost-loss ratio. It can explain real-life behaviors that the cost-loss ratio cannot. This was the case in our recent application for the assessment of forecasts value on the Montmorency River in Canada (Matte et al. 2017). Other examples of accounting for risk-aversion for decision-making in hydrology and meteorology include Shorr (1966), and Cerdá and Quiroga Gómez (2008).

References:

  • Bernoulli D. (1738) Specimen Theoriae Novae de Mensura Sortis, Commentarii academiae scientiarum imperialis Petropolitanae, 5, 175-192.
  • Cerdá Tena E. and Quiroga Gómez S. (2008) Cost-Loss Decision Models with Risk Aversion, Working paper no. 01, Instituto Complutense de Estudios Internaciolales, 28 pages.
  • Krzysztofowicz R. (1986) Expected utility, benefit, and loss criteria for seasonal water supply planning, Water Resources Research, 22(3), 303-312.
  • Matte S., Boucher M-A, Boucher V. and Fortier-Filion T-C (2017) Moving beyond the cost-loss ratio, economic assessment of streamflow forecasts for a risk-averse decision maker, Hydrology and Earth System Sciences (Accepted)
  • Murphy A.H. (1977) Value of climatological, categorical and probabilistic forecasts in cost-loss ration situation, Monthly Weather Review, 105(7), 803-816
  • Murphy A.H. (1976) Decision-making models in cost-loss ratio situation and measures of values of probability forecasts, Monthly Weather Review, 104(8), 1058-1065
  • Shorr B. (1966) The cost/loss utility ratio, Journal of Applied Meteorology, 5(6), 501-803.
  • von Neumann J. and Morgenstern O. (1944) Theory of games and economic behavior, vol. 60, Princeton University Press Princeton, 625 pages.

Figure captions:

[1] The original pictures were taken from these websites: http://www.famous-mathematicians.com/daniel-bernoulli/ (right) and http://www2.stetson.edu/~efriedma/periodictable/html/Bi.html (left). The schematic drawing of D. Bernoulli’s blood pressure experiment apparatus was taken from https://plus.maths.org/content/daniel-bernoulli-and-making-fluid-equation

[2] The original pictures were taken from these websites: http://www.tinbergen.nl/oskar-morgenstern-and-john-von-neumann-shelby-white-and-leon-levy-archives-center/ (left) and http://www.karbosguide.com/books/pcarchitecture/chapter02.htm (right)

Posted in decision making, economic value | 2 Comments

The elusive optimal decision rule and the impact of forecasting

Posted on May 23, 2017 by Maria-Helena Ramos

Contributed by Micha Werner, IHE Delft and Deltares

This winter just passed I was faced with the kind of dilemma of the type I am sure many find all too familiar. I was the designated driver one weekend for my daughter’s hockey team away match.

Parents take turns to drive to away matches, and so I was all set to head off with four excitedly chatting thirteen year-olds. At breakfast, I saw on the internet that KNMI, the Dutch national meteorological agency had earlier that morning issued an amber alert. It had been well below zero for a couple of days and the roads were cold. A warm front from the West was now pushing its way through, and there was a high risk of black ice, leading to very slippery roads and potentially dangerous situations.

All non-essential trips were strongly discouraged, particularly on motorways in the west of the country that had been as yet little used that Sunday morning. The warning indicated that the risks would diminish by 9:30. I had to leave by 9 to make it on time and my route included a section of motorway. What was I to do? Should I heed the warning? Was this trip essential or was it non-essential?

Having professionally dabbled with the concept of costs and losses, the decision seemed obvious. The risks were high, and the consequences of a potential accident severe. So the obvious rational decision was not to go. But there were also consequences to not going. The match would need to be called off if the four girls I had with me did not show. Would the opposing team claim victory for a no-show? That could be embarrassing, and moreover could knock our team out of the top of the pool. Would other parents decide not to go; or rather would they go and be let down by my decision to stay at home?

A furtive discussion on our team WhatsApp group ensued. Finally an agreement was reached. Yes, I explained to a concerned mother, yes I had invested in winter tyres. The decision was made. We were off.

Making decisions based on warning information

This little story reflects the difficulty of making decisions based on warning information. Even when provided with quite specific information such as I received that morning, which not only targeted the road type but also provided specific details as to the timing. This is the level of detail that impact based warnings set out to achieve. Information contained in the forecast that is personally relevant to the consumer of the forecast. And yet making the decision that would seem obvious on paper is not easy in practice.

There has been some research on the process of an individual making a decision based on an advisory of flooding, or severe weather, or other (natural) hazards.

  • Does the warning apply to me? Do I trust it?
  • Am I inherently risk averse, or rather risk acceptant?
  • How is my decision influenced by the potential losses?
  • And in making a decision are these losses balanced against the cost of taking measures?

A decision making “serious-game”: the Shopkeepers Dilemma Game

The latter question listed above is the reasoning that underlies the rational decision making principle. It was also the question what was asked participants in a decision making “serious-game” we played at the 2016 European Geophysical Union during the Ensemble Hydro-meteorological Forecasting session. Some readers may also recall playing the game at the 2016 HEPEX conference in Quebec City, Canada, or at the Royal Meteorological Society Conference in Manchester, England. It was also played with a group of students in a flood management course at IHE Delft.

Micha Werner and his poster at EGU 2017

This year’s EGU saw a poster presenting the results, and a paper will further explore the details but is still in preparation (I had of course intended to write the paper earlier, and may have found more time to do so if only I had heeded more warnings and stayed at home, instead of going to weekend sports matches with my kids!)

In the game, three different shop-keepers were presented with a series of seven forecasts, each providing participants with the forecast probability that they and their shop could be flooded. They were asked to make one of three decisions for each forecast; choosing between taking no action; raising temporary defences on the embankment between their shop and the river; or moving their inventory. Except for doing nothing, all actions came at a cost. But flooding also caused a loss.

What was also important was that if the shop could stay open for business then a profit could be made. Obviously the shop could not stay open for business if flooded, or if the decision to move the inventory had been taken. However, if the demountable defences were raised, and these were not subsequently overtopped, then it was business as usual and a profit could be made.

The three shops selected in the game had quite different costs and losses:

  • One shop sold Ferraris, with significant losses when flooded, as well as high costs when moving the inventory. But profits when the shop stayed open for business were also high.
  • The second type of shop was a grocery store, with lower losses when flooded; but also lower costs to move inventory, and lower profits when staying open for business.
  • The third shop sold gravestones. Losses due to flooding gravestones were low; and lower or equal to the cost of raising the defences or moving the inventory. Of course when raising the embankments the shop could stay open for business if the embankments were not overtopped, making a modest profit.

At each step, after the decision on the bulletin was made, the actual outcome of the event was shown and participants asked to tally their results. The winner was the participant who had the lowest expected expense at the end of the game, due either to taking measures, which at times were taken in vain when the forecast flood did not materialise, or due to losses incurred when either no action, or inadequate action was taken.

The game material and publications are publicly available here in the Resources Page of the Hepex Portal, together with other games and lectures provided by the community.

A couple of interesting patterns

Reviewing the results of the 215 participants that played the game across the four sessions revealed a couple of interesting patterns. I will leave the full assessment of those results to the full paper, but some interesting thoughts came forward that may be worth mentioning.

  1. Choices made by those selling Ferraris were more or less as expected. They were keen to act on the forecasts; and the most likely to take action in response to the forecasts, with a clear relationship between the probability of forecasting and the inclination to take action. The forecast was of clear value to them.
  2. This was also more or less the case for those with grocery stores, though they did not take action quite as often.
  3. What was interesting though was the behaviour of those selling gravestones. Though these was little need to respond to the forecasts for them, as the cost of unnecessary action was very high compared to the losses when no action was taken, the gravestone sellers seemed overly keen to respond to the forecasts. Their behaviour was more or less the same as the grocery store participants. It would appear that they were prepared to take the risk of getting it wrong despite the cost, preferring to invest to avoid being flooded.  Such behaviour is often considered to be risk adverse, influencing decisions to be made that stray from the rule that would be prescribed by the rational cost-loss theory.

But not all decisions made in risky situations are necessarily risk adverse. The quite well known prospect theory, first described by Kahneman and Tversky (1979) outlines how decision makers may be risk seeking, or rather how they may be risk averse, depending on how information informing the risk based decision is framed.

Generally, in the game it seemed that the prospect of being flooded caused the decision to take protective action to be taken more often than necessary. This could also be considered loss aversion. People are prepared to invest in taking measures, just to avoid the displeasure of being flooded, which given the disgruntled mutters of participants in the room when flooded after deciding not to take action, clearly gives rise to negative emotions.

The role of emotion in the framing of a decision

Emotion is clearly an important factor in the framing of a decision, as outlined by Druckman and Mcdermott (2008). They go on to discuss how different emotions may influence decisions. Maybe a bit too much to discuss in a blog post that started short but just kept growing; and definitely a topic that is increasingly out of reach of a hydrologist such as myself. However, they also shed some light on my little dilemma. Clearly, the negative emotion connected to not going to the match, as well as my own preference confidence led to the taking of what was a risk-seeking decision.

Emotions may also lead to paralysis in decision making, resulting in no decision being made. That reminded me of a discussion I recently had with a certain Dr Stephen Hussey. He leads an NGO called the Dabane Trust that works with communities in Southern Zimbabwe. What he has found somewhat confounding is that despite clear and acknowledged signs of oncoming drought conditions provided by a seasonal forecast product, that it was very difficult to change what was considered to be normal behaviour by the communities. They could easily plant a less drought sensitive crop than the traditional maize; but they had always planted maize and changing that was not easily done.  Changing behaviour could perhaps be done if a champion for change could be found in the community, but otherwise it proved to be very difficult.

Of course I need not tell the reader that the true value of any forecast will depend on how that forecast is used to influence behaviour and take protective action. Impact based forecasting has recently emerged as a step towards personalising forecasts and through this influencing what an individual should do to get themselves and their property out of harm’s way in the face of natural hazards. That is no doubt a good thing, but the impact of a forecast ultimately depends on the behavioural change that it influences.

This calls to my mind for branching out to research on behavioural change, working with psychologists, social scientists, and economists, as well as many others; on thinking how to frame forecast information. We cannot do this as hydrologists alone. I myself always like to find analogies from my own experiences in decision making in my personal life, wondering how this may help explain the behaviour of recipients of forecasts.

There were no real issues with the roads that Sunday morning. We left a bit earlier than planned, took caution and arrived well on time. That day the team did reasonably; of the two matches they had to play they won one, and lost the other.

Some game-addicted members of the Hepex community at EGU 2017 (from left to right): Micha Werner, Hannah Cloke, Louise Arnal, Andy Wood, Louise Crochemore, Maria-Helena Ramos and Ilias Pechlivanidis). The poster in the back can be found here.

References

Posted in decision making, forecast communication, forecast users | 1 Comment