Scoring

Proper scoring rules let us evaluate predictive observation distributions w.r.t. data in ways beyond marginal likelihood.

dynestyx.evaluation.scoring defines the score objects themselves: BaseObservationScore, GaussianLogProbScore, DawidSebastianiScore, ObservationWiseCRPSScore, and EnergyScore. These scores consume canonical one-step-ahead predictive observation distributions produced by supported filters: the continuous-time CD-Dynamax Gaussian filters (ContinuousTimeKFConfig, ContinuousTimeEKFConfig, ContinuousTimeUKFConfig, and ContinuousTimeEnKFConfig) and the discrete-time Cuthbert EnKFConfig.

For the Cuthbert EnKF, prediction collection retains the incoming forecast ensemble, removes Cuthbert's leading dummy state, and projects each time-aligned state ensemble through the observation mean function. The resulting PredictedObservationOutputs carries the projected ensemble, its mean and sample covariance, the observation-noise covariance, and the total predictive observation covariance. Cuthbert does not provide an observation-noise-perturbed ensemble, so sample_source="auto" uses the projected latent ensemble plus reproducibly sampled observation noise for ensemble scores. The aligned prediction at index zero is the inflated pre-update initial-prior forecast for the first observation, with no ODE transition; downstream code must not shift the sequence again.

ObservationScoringConfig is documented on the companion Scoring Configs page.

Scoring rules for predictive observation distributions.

BaseObservationScore dataclass

Bases: ABC

Base class for predictive-observation scoring rules.

Subclasses define a per-time score array. site_name is available for integrations that want to record score arrays into named trace sites.

GaussianLogProbScore dataclass

Bases: BaseObservationScore

Per-time multivariate Gaussian log-probability score.

Uses predictive Gaussian moments and returns a score array of shape (*plate, time, 1). Higher values are better.

DawidSebastianiScore dataclass

Bases: BaseObservationScore

Per-time Dawid-Sebastiani score under Gaussian predictive moments.

Uses predictive Gaussian moments and returns a score array of shape (*plate, time, 1). Lower values are better.

ObservationWiseCRPSScore dataclass

Bases: BaseObservationScore

Per-observation-component CRPS under Gaussian predictive marginals.

Applies the scalar Gaussian CRPS to each observation component separately and returns a score array of shape (*plate, time, observation_dim). Lower values are better.

EnergyScore dataclass

Bases: BaseObservationScore

Per-time ensemble energy score with exponent beta.

If an explicit predictive observation ensemble is unavailable, this score can approximate one by drawing n_samples observations from the Gaussian predictive observation moments. Returns a score array of shape (*plate, time, 1). Lower values are better.

When synthetic sampling is needed, pass sample_seed to compute.

vectorized_pairwise=True is faster for moderate ensemble sizes but materializes the full pairwise distance tensor. Setting it to False uses a lower-memory lax.scan path at the cost of extra compute.