Introduction

In the US, the risks and complexities of wildfire have magnified dramatically, with changing socioecological fire regimes, expanding development in fire-prone areas, faster rates of fire growth, more destructive impacts to communities and landscapes, and escalating costs1,2,3,4,5. This is consistent with global patterns of environmental change and human activity that are interacting to increase likelihood of extreme wildfire events with grave implications for ecosystem services and public health6,7,8,9. Continued reliance on wildfire suppression as the dominant management action may exacerbate future hazard while increasing strains on an incident response system already challenged by increasing synchronous fire danger and resource scarcity10,11,12,13. Hence growing recognition of the need for more proactive approaches to wildfire management as well as risk and decision analytic frameworks to support them14,15,16,17,18.

A foundational need for proactive risk management is information on the likelihood of experiencing wildfire in any given location over a given time horizon. Likelihood can vary by orders of magnitude across landscapes and is therefore a significant driver of hazard and exposure as well as prioritization and efficacy19. Emergency managers, ignition prevention programs, hazardous fuels reduction programs, community hardening programs, land use planners, water and power utilities, and insurance providers, among others, all need reliable information on wildfire likelihood.

This information is typically provided through burn probability (BP) modeling, now widely used for operational and research purposes20,21,22,23. BP models differ by the intended application. Generally, all share the same workflow of iteratively simulating fire spread across landscapes while accounting for variability in factors such as ignition patterns and weather. Developing gridded BP datasets involves simulating multiple fire events under a range of conditions and calculating BP values as the number of times a given cell burns divided by the number of simulations. Some models are intended for near real-time incident support24, where ignitions are known, and generate conditional BPs corresponding to patterns of potential fire spread under forecasted weather over a defined planning horizon. Some models capture uncertainty in both ignition location and weather, and generate conditional BPs given user-defined scenarios25. Some models also account for uncertainty in ignition likelihood and generate annualized BPs that probabilistically simulate ignition location and timing as inputs into fire spread simulations26.

BP modeling is deployed extensively in the US for a range of decision support applications and products, particularly for land management agencies like the USDA Forest Service27,28,29,30,31. Our focus here is on FSim, a comprehensive large fire simulation system that generates tens of thousands of synthetic fire years and models fire occurrence and spread while accounting for stochasticity in ignitions, weather, and spotting26. One of its primary outputs is spatially gridded data of annualized BP values (120–270 m) that reflect average large fire potential based on input landscape conditions. Updating fuel conditions to account for large disturbance, as is done annually here, is intended to reflect dynamic spatial patterns of fire likelihood.

Although these results can be used to estimate expected annual area burned—as the sum-product of all burnable pixels and their corresponding BP values—this is not our primary focus as it does not directly lead to targeted, localized intervention; rather our focus is on FSim’s ability to spatially discriminate wildfire likelihood. We focus on FSim because its results have been integrated into multiple strategic assessment and planning efforts and informed budgetary allocations on the scale of billions. Notable nationwide examples leveraging FSim outputs include the National Cohesive Wildland Fire Management Strategy27, the Wildfire Crisis Strategy32, the Wildfire Risk to Communities project33, and the FEMA National Risk Index34. FSim results have also been widely deployed to support localized fuels management and operational response strategies35,36,37.

Despite growing use and sophistication of BP models, the wildfire science community has not widely adopted probabilistic forecast verification as a common practice. Few published examples exist comparing BP values against subsequent fire activity38,39,40,41. There are many challenges to measuring predictive accuracy of BP models, including insufficient or inaccurate fire observation data, very low BP values in some locations (i.e., < 1 in 1,000), limited ability to model how suppression alters large fire spread patterns, and, critically, changing landscape conditions29,42,43.

For spatially discriminating areas more or less likely to burn to support targeted intervention, accurately reflecting dynamic landscape conditions is essential. A longer post-simulation evaluation period enables better capture of interannual variability in fire activity, but the original BP map becomes increasingly outdated the longer the evaluation period extends. By contrast, more frequently updating BP maps enables better capture of interannual variability in the landscape’s ability to support or inhibit large fire growth, particularly salient in recently disturbed areas where prior fire activity can lead to self-limiting behavior44.

Recent work evaluating the performance of FSim BP maps across the conterminous US—based on a static 2014 landscape and contrasted against observed wildfires from 2016 to 2022—reported a moderate correlation of mean BP with observed burned area but underprediction of burned areas in some regions40. The authors suggest that discrepancies may have stemmed from an outdated fuels layer and argued that frequent updates to BP maps could improve their usefulness. Here, motivated in part by these recent findings, we present a novel performance evaluation of FSim, confronting annually updated BP maps with subsequent fire activity across the state of California over a 4-year period (2020–2023).

California presents a compelling and challenging use case—it contains diverse landscapes with high fuel loads influenced by fire exclusion and tree mortality and evolving fire regimes driven by human ignitions and extreme wind events, and has significant attention and need to proactively address growing wildfire risks45,46,47,48,49,50,51,52,53,54. California can also have significant interannual variability in wildfire activity. In 2020, the state set historical records for total area burned and in 2021 experienced the largest single fire in state history (the Dixie Fire), with significant impacts to communities and landscapes15,55,56,57. The fire years of 2022 and 2023 by contrast burned only ~ 10% of the amount burned in 2020 and 2021.

Our analysis is built on four pairwise comparisons of annual BP and fire activity, with each subsequent BP map updated in response to the prior year’s disturbances. Informed by prior work, we examine how the observed area burned varies with BP values, compare distributions of BP values inside and outside of observed burned areas, and calculate logarithmic skill scores39,40,58. We establish performance benchmarks based on percentiles of observed area burned and BP area mapped, discuss interpretations and risk management implications, and offer future directions in probabilistic forecast verification for wildfire modeling.

Results

Observed area burned and burn probabilities

Maps of simulated BP values (Fig. 1) reveal areas of higher fire likelihood throughout the Sierra Nevada, Northern and Southern Coast Ranges, Klamath Mountains, Transverse Ranges, and Peninsular Ranges, with lower likelihood in the Central Valley and Mojave Desert areas. Overlaid observed fire perimeters generally align with hotspots of high BP, particularly in the Northern Coast Range and Klamath Mountains. Interannual variability in spatial patterns of BP is evident in the wake of significant wildfires, particularly notable in the northwest portion of the state with widespread reductions in BP after the 2020 and 2021 fire years (Fig. 2). Within areas burned in 2020, the pre-burn mean BP was 0.054, reduced to 0.024 post-burn, a 56.19% reduction (median BP was reduced by 66.23%). Within areas burned in 2021, the pre-burn mean BP was 0.058, reduced to 0.018, a 69.2% reduction (median BP was reduced by 73.32%).

Fig. 1
figure 1

© Esri, USGS, NOAA. State boundaries from U.S. Census Bureau TIGER/Line® Shapefiles (public domain).

Burn probability values and observed fire perimeters. The state map shows 2020 BP and 2020–2023 fire perimeters. Inset panels contain BP and fire perimeters for respective years. The chronological inset map shows burned areas that are accounted for in subsequent BP maps. Map created in QGIS 3.34 using Esri World Terrain Base

Full size image
Fig. 2
figure 2

© Esri, USGS, NOAA.

Interannual variability in simulated BP due to prior large fire disturbance; location matches inset map from Fig. 1. Panels (a) and (b) show observed 2020 wildfires with pre-and post-simulation BPs (2020 and 2021 BP maps), respectively. Panels (c) and (d) show observed 2021 wildfires with pre-and post-simulation BPs (2021 and 2022 BP maps), respectively. The map reflects both the impact of wildfires prior to 2020 as well as wildfires in 2020 and 2021 on subsequent fire spread potential and BP values. Map created in QGIS 3.34 using Esri World Terrain Base

Full size image

Proportional comparisons of expected area burned (eAB) and observed area burned (oAB) reveal aligned patterns across five equal area BP classes (Fig. 3). BP class definitions vary by year; exact delineations of the BP bins are provided in supplementary materials. The general interpretation is that the model effectively predicts that most area burns in the higher BP classes, with slightly more area burned in the Medium and Low-Medium than predicted. The proportion of oAB in the Med-High or High BP classes was 90.3% in 2020, 80.9% in 2021, 93.6% in 2022, and 65.7% in 2023, notably higher than the ~ 40% that would result if area burned followed a random distribution across equal-area BP classes. Predicted proportions of eAB for the Med-High and High BP classes closely matched oAB values (90.1%, 89.7%, 89.7%, and 89.6%), except for 2023 where less area burned in the Med-High and more burned in the Med BP classes. The High BP class alone accounted for 69.2%, 56.8%, 79.8%, and 56.7% of oAB across respective years, varying from eAB (67.1%, 66.4%, 66.4%, and 66.4%). The mean observed area burned for all 4 years in the High BP class was similar to the expected area burned (65.6% vs 66.6%). The proportion of observed area burned in the Low and Low-Med classes was 2.3%, 5.8%, 3.4%, and 9.1% across respective years. These values are greater than expected area burned (2.1%, 2.2%, 2.3%, and 2.3%). The mean observed area burned of all 4 years in Low and Low-Med classes exceeded expected area burned (5.1% vs 2.2%).

Fig. 3
figure 3

Proportional comparison of expected area burned (eAB) and observed area burned (oAB) across five equal area BP classes. BP class definitions vary by year; exact delineations of the BP bins are provided in supplementary materials.

Full size image

Burn probabilities inside and outside of burned areas

Mean and median BP values were greater within burned areas than outside (Fig. 4). While distributions of BP values inside burned areas showed variability across years, BP values outside of burned areas were consistently concentrated near zero. In 2020, mean BP values in burned areas were 348.78% greater than unburned and median BP values in burned areas were 880.67% greater. In 2021, mean BP values in burned areas were 292.43% greater than unburned and median BP values in burned areas were 551.44% greater. In 2022, mean BP values in burned areas were 271.70% greater than unburned and median BP values in burned areas were 664.66% greater. In 2023, mean BP values in burned areas were 238.53% greater than unburned and median BP values in burned areas were 599.16% greater.

Fig. 4
figure 4

Violin plots comparing the distribution of burn probability values for burned and unburned areas. The horizontal lines represent the median (black) and mean (blue) within each respective class. Panels (a–d) correspond to years 2020–2023, respectively.

Full size image

Cumulative distribution plots comparing BP values for burned and unburned areas confirm consistent patterns across years of higher BPs in burned areas (Fig. 5). Kolmogorov-Smirnoff tests on a 0.01% random sample of burned/unburned areas yielded statistically significant differences for all 4 years (p < 0.01) with test statistic values of 0.5447 (2020), 0.4248 (2021), 0.6136 (2022), and 0.3738 (2023). In 2020, 95% of burned area occurred at or above the 50th percentile of BP values and 51% occurred above the 90th percentile. In 2021, 88% of burned area occurred at or above the 50th percentile and 33% above the 90th percentile. In 2022, 96% of burned area occurred above the 50th percentile and 38% above the 90th percentile. In 2023, 74% of burned area occurred above the 50th percentile and 36% above the 90th percentile.

Fig. 5
figure 5

Cumulative distribution plots comparing burn probability values in burned versus unburned areas. Panels (a–d) correspond to years 2020–2023, respectively.

Full size image

Logarithmic skill scores

Logarithmic Skill Scores (LSS; 58) were computed to provide a quantitative measure of forecast quality using two different reference logarithmic scores as the baselines for deriving the skill score, two different sets of data (burned only and all cells), and across all four years totaling 16 LSS estimates (Table 1). Mean LSS were 0.276 and 0.339 from 2020 to 2023 for burned cells using the naive reference model and the resampled reference model, respectively, which can be interpreted as the percentage improvements over these reference models. The LSS drops when both burned and unburned cells are considered with means of 0.114 and 0.280 against the naive and resampled reference models, respectively (Table 1). The decrease can largely be attributed to two years of negative skill in 2022 and 2023 for the all-cells LSS. The skill score is higher (0.296) for burned cells in 2022 compared to 2021 (0.264) even though the all-cells skill score is negative in 2022, a result from a well-below average year in total area burned but demonstrated skill in predicting the spatial location of the areas that did burn (c.f. Figure 2).

Table 1 Logarithmic skill scores of the BP results using two reference models: (1) The 15-year reference model is a spatially-constant, naive model that represents the 2006–2020 mean annual acreage burned and (2) the resampled BP spatially randomizes predicted BP values. The mean logarithmic scores used to derive the skill scores are also reported.
Full size table

To specifically test the value of annual updates, the LSS (the version derived from the 15-year mean reference model) were calculated using the 2020 BP predictions in comparison to the subsequent years (2021–2023) and compared to the LSS derived from the BP in matched years. LSS improved in five of the six comparisons by a mean of 0.028 (42%)–indicating superior performance of up-to-date BP maps relative to the continued use of the 2020 BP predictions for multiple years.

Discussion

Results generally showed strong predictive performance of annual BP maps as evaluated through multiple comparisons: markedly greater proportions of burned areas in higher BP bins; higher mean and median BPs in burned than unburned areas; statistically significant differences in cumulative distributions of BP between burned areas and unburned areas; and Logarithmic Skill Scores (LSS) for BP maps indicating outperformance of reference models. A general performance benchmark of observing 57–80% of area burned in the top ~ 20% of mapped BP area was established.

In addition to the general performance benchmark, the importance of interannual updates was evident. The largest cause of year-to-year model differences were from previous year disturbances represented in the fuel inputs, the most impactful of those being wildfire scars. We observed a mean percent reduction in BP of 69% in the following years (2021–2023) within the wildfire perimeters occurring in the study time frame (2020–2022). The model represented beyond -perimeter effects with a mean percent reduction in BP of 15% in a 2 km buffer from these wildfires. The BP maps responded on a landscape-level as well with total mean reductions in BP of 10% and 7% following the large fire years of 2020 and 2021, respectively, and only a 1% reduction in BP following the mild fire year in 2022. The average improvement of 42% in the LSS using updated inputs over relying on the increasingly-outdated 2020 BP map also highlights the added value of interannual updating.

Some relevant prior work evaluates the performance of conditional BP maps generated for individual wildfire events with known ignition locations, which significantly shrinks the uncertainty relative to simulating thousands of potential fire years and millions of potential events. Paz et al.41 established a similar performance benchmark for a single wildfire, reporting that 87% of observed burned area occurred in the top 30% of mapped BP area, which is within the range of results we present. Allaire et al.38 calculated Brier Skill Scores against a posteriori reference models based on the final footprints of seven observed fires. Although not directly comparable, our LSS were within the ranges they present (− 9.986—0.352) and generally closer to their best performing ensemble.

In two cases our LSS were negative—for low fire activity years (2022–2023) and when considering BP forecasts for all cells (not just burned cells) using the 15-year mean BP reference. This is largely due to the large number of higher BP predictions with a relatively low level of positive cases (burned cells). For these years, the analysis still shows model skill in all three analyses. For burned cells only, the LSS was generally similar to other years. The spatially randomized BP reference model intended on evaluating the spatial skill alone given the exact same distribution of BP was present and the corresponding LSS were 0.246 for 2022 and 0.191 for 2023. This highlights the difficulty in predicting burn probability in systems that can exhibit dramatic interannual variability (e.g., ~ 10% of area burned in 2022 and 2023 relative to 2020 and 2021). Analysis on BP modeling skill scores could be expanded to explore how they vary with fire activity or alternative reference models.

More directly relevant work includes evaluation of landscape- scale BP maps across larger regions with multiple years of fire observations, but without annual updates to fuels after disturbance. Evaluating simulation modeling in Alberta, Canada, Beverly and McLaughlin39 reported that distributions of BP in burned areas were not heavily skewed towards higher BP values, but that most of the burned area (75–80%) occurred in the top 50th percentile of mapped BP. Evaluating simulation modeling for the conterminous US, Carlson et al.40 established that 68% of the observed burned area occurred in the top 40% of mapped BP area, and for Mediterranean California that 62% of observed area burned occurred in the top 40% of mapped BP area. The performance benchmark discrepancy between40 and that established here could be attributed to several factors, including using more up-to-date and customized fuel input layers, more recent historical fire weather and occurrence data, differences and improvements in modeling methods and calibration over time, or simply narrowing the scope of analysis to a more fire prone region. The present analysis encompasses landscape scale evaluation of BP maps over multiple years while resolving issues of large disturbance effects through annual updates and generally indicates the strongest BP modeling performance published to date.

Several extensions and improvements are apparent. First, researchers could pursue validation across different geographic areas and across longer time horizons, especially in areas with comparatively less frequent fire activity, or for different BP systems used in operational decision support27,59. Second, validation could be performed on other FSim model outputs such as perimeters, fire progression maps, or magnitude of daily spread events60,61,62,63. Other efforts could focus on continued model improvement. For example, better information on fuel break and suppression effectiveness could improve containment modeling48,64,65. While temporal trends in fire occurrence were utilized in calibration, the reliance on historical data results in climate change not being directly considered and previous work has developed approaches for its incorporation in more out-year probability estimates66.

It is likely that application of machine learning-based approaches for BP prediction will increase67,68,69,70, highlighting a need for comparative validation and use case exploration. Costa-Saura et al.71 contrasted BP interpretations from fire spread and random forest models and noted that firefighting decision making might be more tightly linked with fire spread information. Applications of process-based fire spread simulation will persist, particularly for accompanying analyses based on the underlying simulated perimeters to analyze transmission into communities or watershed impacts14,72. This highlights the broader needs of accounting for user needs and experience with using risk-based information for decision support31,73.

In summary, the primary implication is that managers can have confidence in using well-calibrated BP maps to support ongoing wildfire risk assessment and planning in California and other fire prone regions, while sharing the sentiments expressed in Beverly and McLaughlin39 and Carlson et al.40 that caution is warranted. In fact, with growing wildfire activity and more communities at risk the use of BP modeling will likely increase, and rigorous evaluation of these predictions will increase confidence when prioritizing resources for prevention and mitigation. We hope this effort can catalyze broader collaborative efforts in probabilistic wildfire forecast verification with FSim and other BP models.

Methods

We generated annual burn probability (BP) maps for each year from 2020 to 2023 using a customized version of the Large Fire Simulation Model (FSim; version 1.0.9) in projects for the California Department of Forestry and Fire Protection. FSim is a stochastic , iterative model that simulates plausible fire ignition, growth, intensity, suppression, and containment throughout hypothetical years26,74. FSIM is developed by the United States Forest Service Missoula Fire Science Laboratory utilizing the foundational surface spread model75, crown fire spread and initiation models76,77 and methods for their spatially-explicit application78,79. Each iteration in the model represents an entire year and given enough iterations is intended to capture the full range of potential outcomes for a given set of inputs along with estimates of central tendency and variability.

Simulations were performed individually for unique pyromes, i.e., areas of relatively homogenous contemporary vegetation, climate, and fire regimes built to support landscape fire simulation80. Each pyrome was an individual modeling domain with unique calibration parameters and a 30 km buffer to allow ignitions to spread outside pyrome boundaries preventing edge effects. The BP estimates from each domain were then mosaicked by summing the overlapping buffer areas after normalizing by iteration count, which were a minimum of 10,000 per domain. Input parameters and outputs were reviewed in 2020 by fire experts in California and calibrated against historical fire occurrence data using best practices81. Input data were updated annually in response to observed disturbances based on fire observation and fuel treatment data and expert-defined fuel calibration rulesets that follow the general framework developed in the LANDFIRE program82. Calibration targets for fire size and occurrence distributions were developed using ordinary least squares regression over the prior 15 years of fire activity; more details on calibration are available in29.

We use a historical fire perimeters package to determine whether an area was burned by a fire in our given time frame (2020–2023). To create this package, we merged Welty and Jeffries83 perimeters with National Intergency Fire Center perimeters and clipped fires that burned only partially within California to the California boundary. We filtered the resulting geometries for large fires (> 100 acres) and turned them into binary rasters at a 30 m resolution. We excluded non-burnable land cover from the analysis as defined by LANDFIRE.gov, except where BP was greater than zero to retain urban and agricultural areas classified as burnable following methods in the Wildfire Risk to Communities project84. This method of ignoring non-burnable features such as water, barren, rock, and ice ensured that we did not give our BP models credit for predicting that non-burnable features would not burn.

We employ three methods to evaluate BP modeling performance results against subsequent annual fire activity. First, we compare the proportional values of expected area burned (eAB) and observed area burned (oAB) across five equal-area BP bins. To quantify eAB, we multiply the total land area mapped in each BP bin by the mean BP of that bin. Second, we compare mean and median BP values within and outside of burned areas using violin plots and compare cumulative distribution functions of burned and unburned areas by BP with Kolmogorov–Smirnov tests85.

Third, we calculate logarithmic scores (LS; 58) and derive logarithmic skill scores (LSS) using two reference models: one based on the previous fire activity produced by calculating the constant BP value that would generate the expected area burned that matched the latest 15-year mean area burned (2006–2020) in the Fire Occurrence Database86, and another based on spatially randomizing the predicted BP values across the entire domain. The former represents a standard, naive reference analogous to a climatological average in weather forecasting skill scoring and the latter a reference isolating the spatial skill by having the same distribution of BP values across the study area. We iteratively tested the variability in LSS following BP spatial randomization and found minimal variation in the LSS values. The LS followed the binary formulation as follows:

$$LS = \frac{1}{N}\sum\nolimits_{{i = 1}}^{N} { - \ln \left( {1 - |p_{i} - o_{i} |} \right)}$$
(1)

where \(p_{i}\) is the probability at cell \(i\) and \(o_{i}\) is the binary outcome burned (1) or not burned (0), and N is the number of cells. The LSS is then:

$$LSS = 1 - \frac{LS}{{LS_{ref} }}$$
(2)

where \(LS_{ref}\) is either the 15-year mean BP or the spatially randomized mean BP models. LSS are chosen over the ubiquitous Brier Skill scores (e.g., 38) to account for the rarity of wildfire occurrence, a known limitation58.