Scoring

Proper scoring rules let us evaluate predictive observation distributions with respect to data in ways beyond marginal likelihood.

dynestyx.evaluation.scoring defines the score objects themselves: BaseObservationScore, GaussianLogProbScore, DawidSebastianiScore, ObservationWiseCRPSScore, and EnergyScore. These scores operate on canonical one-step-ahead predictive observation distributions. Supported filters include the continuous-time CD-Dynamax Gaussian filters (ContinuousTimeKFConfig, ContinuousTimeEKFConfig, ContinuousTimeUKFConfig, and ContinuousTimeEnKFConfig) and the discrete-time Cuthbert ensemble Kalman filter (EnKFConfig). A continuous-time deterministic model can use the latter by nesting an ODE-flow Discretizer inside Filter.

For the Cuthbert EnKF, keep include_predicted_observations=True (the default). With ObservationScoringConfig(sample_source="auto"), moment-based rules use the predictive observation mean and covariance, while ensemble-based rules use the projected forecast ensemble with observation noise added reproducibly from sample_seed. See Observation scoring with a Cuthbert EnKF for a complete example. ObservationScoringConfig is documented on the companion Scoring Configs page.

Scoring rules for predictive observation distributions.

BaseObservationScore dataclass

Bases: ABC

Base class for predictive-observation scoring rules.

Subclasses define a per-time score array. site_name is available for integrations that want to record score arrays into named trace sites.

GaussianLogProbScore dataclass

Bases: BaseObservationScore

Per-time multivariate Gaussian log-probability score.

Uses predictive Gaussian moments and returns a score array of shape (*plate, time, 1). Higher values are better.

DawidSebastianiScore dataclass

Bases: BaseObservationScore

Per-time Dawid-Sebastiani score under Gaussian predictive moments.

Uses predictive Gaussian moments and returns a score array of shape (*plate, time, 1). Lower values are better.

ObservationWiseCRPSScore dataclass

Bases: BaseObservationScore

Per-observation-component CRPS under Gaussian predictive marginals.

Applies the scalar Gaussian CRPS to each observation component separately and returns a score array of shape (*plate, time, observation_dim). Lower values are better.

EnergyScore dataclass

Bases: BaseObservationScore

Per-time ensemble energy score with exponent beta.

If an explicit predictive observation ensemble is unavailable, this score can approximate one by drawing n_samples observations from the Gaussian predictive observation moments. Returns a score array of shape (*plate, time, 1). Lower values are better.

When synthetic sampling is needed, pass sample_seed to compute.

vectorized_pairwise=True is faster for moderate ensemble sizes but materializes the full pairwise distance tensor. Setting it to False uses a lower-memory lax.scan path at the cost of extra compute.