Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

Authors: HOLCOMB, KAREN - National Oceanic & Atmospheric Administration (NOAA); MATHIS, SARABETH - Centers For Disease Control And Prevention (CDC) - United States; STAPLES, JENNIFER - Centers For Disease Control And Prevention (CDC) - United States; FISCHER, MARC - Centers For Disease Control And Prevention (CDC) - United States; BARKER, CHRISTOPHER - University Of California, Davis; BEARD, CHARLES - Centers For Disease Control And Prevention (CDC) - United States; NETT, RANDALL - Centers For Disease Control And Prevention (CDC) - United States; KEYEL, ALEXANDER - New York Department Of Health; MARCANTONIO, MATTEO - Catholic University Of Leuven; CHILDS, MARISSA - Stanford University; GORRIS, MORGAN - Los Alamos National Research Laboratory; ROCHLIN, ILIA - Rutgers University; HAMINS-PUERTOLAS, MARCO - University Of California San Francisco (UCSF); RAY, EVAN - Mount Holyoke College; UELMEN, JOHNNY - University Of Illinois; DEFELICE, NICHOLAS - Mount Sinai School Of Medicine; FREEDMAN, ANDREW - North Carolina State University; HOLLINGSWORTH, BRANDON - Cornell University; DAS, PRAACHI - North Carolina State University; OSTHUS, DAVE - Los Alamos National Research Laboratory; Humphreys Jr, John; NOVA, NICOLE - Stanford University; MORDECAI, ERIN - Stanford University; Cohnstaedt, Lee; KIRK, DEVIN - Stanford University; KRAMER, LAURA - New York State Department Of Health; HARRIS, MALLORY - Stanford University; KAIN, MORGAN - Stanford University; REED, EMILY - Virginia Tech; JOHANSSON, MICHAEL - Centers For Disease Control And Prevention (CDC) - United States

Submitted to: Parasites & Vectors
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 12/20/2022
Publication Date: 1/12/2023
Citation: Holcomb, K.M., Mathis, S., Staples, J.E., Fischer, M., Barker, C.M., Beard, C.B., Nett, R.J., Keyel, A.C., Marcantonio, M., Childs, M.L., Gorris, M.E., Rochlin, I., Hamins-Puertolas, M., Ray, E.L., Uelmen, J.A., Defelice, N., Freedman, A.S., Hollingsworth, B.D., Das, P., Osthus, D., Humphreys Jr, J.M., Nova, N., Mordecai, E.A., Cohnstaedt, L.W., Kirk, D., Kramer, L., Harris, M.J., Kain, M.P., Reed, E.M., Johansson, M.A. 2023. Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction. Parasites & Vectors. 16(1):11. https://doi.org/10.1186/s13071-022-05630-y.
DOI: https://doi.org/10.1186/s13071-022-05630-y

Interpretive Summary: West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental United States. WNV occurrence has high variability and current approaches for targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement.

Technical Abstract: Background: West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental United States. WNV occurrence has high spatiotemporal variation and current approaches for targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. Methods: We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020, and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. Results: Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. Conclusions: Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g., current weather and preliminary human cases).