What are the challenges and research needs in ensemble forecast verification?
Scientific and operational verification challenges include:
- to ensure that hindcasting and verification is an integral and routine component of hydrometeorological and hydrologic forecasting; while common in the atmospheric sciences, verification remains uncommon or cursory in hydrology;
- to compare methods developed by the atmospheric and hydrologic modelling communities, including methods that link single-valued forecast verification and probabilistic forecast verification;
- to consider jointly (but distinctly) the problem of improving the forecasting system, for which we need to evaluate the different sources of skill and uncertainty, and the problem of evaluating whether a forecasting system is useful, for which we need to know how a forecast is used to improve a decision-making process; this includes the selection of key verification metrics and summary scores that could effectively help forecasters and end user in their decision making, as well as techniques for verifying real-time forecasts (before the corresponding observation occurs);
- to propose methods which are appropriate for multivariate forecasts (e.g., forecasts issued for more than one location and forecasts providing values for more than one time step) and methods to analyze forecast quality on multiple space and time scales;
- to propose methods to characterize attributes of multivariate forecasts, such as timing error, peak error and shape error in hydrologic forecasts, and develop products for timing versus amplitude uncertainty information that are meaningful to forecasters and end users;
- to define a optimal set of benchmarks to 1) demonstrate the value of a hydrologic ensemble forecasting system compared to an existing (deterministic or probabilistic) forecasting system, 2) assess whether a hydrologic ensemble forecasting system is useful for decision-making purposes, 3) and analyze the different sources of forecast uncertainty and their interactions to help improve a forecasting system;
- to propose methods for verifying rare events and specifying sampling uncertainty of verifications scores;
- to understand how to account for statistical dependencies in hydrometeorological and hydrologic variables and to design verification measures that are sensitive to the correct representation of statistical dependencies in multivariate forecasts;
- to propose methods which take into account observational error (both measurement and representativeness errors).
This post is a contribution to the new HEPEX Science and Implementation Plan.
See also in “HEPEX Science and Challenges: Verification of Ensemble Forecasts”: