Walk-Forward Evaluation¶
elote.walk_forward() predicts each period from everything before it, then learns that
period. The returned elote.WalkForwardReport carries aggregate accuracy, log loss and
Brier score, and a reliability table showing which probability ranges are overconfident.
Reliability tables¶
report.reliability is a tuple of elote.ReliabilityBin records, one per
equal-width bin on [0, 1]. Choose the number of bins with calibration_bins (default 10; a
positive integer, booleans and non-integers raise ValueError).
Bins are left-inclusive and right-exclusive, except the final bin, which also contains 1.0.
Each prediction is binned before the log-loss clamp is applied.
The population is exactly the
report.predictionspopulation: decisive bouts between competitors already seen, after thewarmupperiods. Draws, missing outcomes, unseen competitors and warmup periods are excluded, so bin counts sum toreport.predictions.mean_predictedis the mean probability that the first side wins;observed_rateis the fraction of those bouts the first side won. Empty bins havecount == 0andNonefor both.Only bin aggregates are stored. This is an empirical expected-score calibration of decisive bouts, not a three-outcome draw-probability model.
from elote import EloCompetitor, group_by_period, walk_forward
periods = group_by_period(rows)
report = walk_forward(EloCompetitor, periods, warmup=2, calibration_bins=5)
for b in report.reliability:
if b.count:
print(f"[{b.lower:.1f}, {b.upper:.1f}) n={b.count} "
f"predicted={b.mean_predicted:.3f} observed={b.observed_rate:.3f}")
A bin whose mean_predicted is well above its observed_rate is overconfident.
Expanding development windows¶
Use elote.expanding_window_evaluate() to inspect a fixed configuration across
seasons or other explicit windows. Supply only development periods; keep a separately
reserved final population outside this helper. Boundaries are exclusive period indexes:
training is [0, train_end) and scoring is [train_end, test_end).
from elote import EloCompetitor, expanding_window_evaluate, group_by_period
development_periods = group_by_period(development_rows)
# This example requires at least six development periods.
folds = expanding_window_evaluate(
EloCompetitor,
development_periods,
[(2, 4), (4, 6)],
competitor_params={"k_factor": 32},
calibration_bins=5,
)
for fold in folds:
print(f"train=[0,{fold.train_end}) score=[{fold.train_end},{fold.test_end}) "
f"training_periods={fold.training_periods} scoring_periods={fold.scoring_periods}")
print(fold.report) # Includes the scored prediction count.
print(fold.report.skipped, fold.report.draws, fold.report.reliability)
Every fold starts with fresh arena state and replays its training prefix. All boundaries
are validated before evaluation: require integer indexes (excluding booleans),
0 < train_end < test_end <= len(periods), and ordered, non-overlapping scoring
windows. Gaps are allowed; empty fold lists are rejected. warmup is reserved for the
training boundary. Configuration, constructor, comparison, score and calibration options
have the same meanings as in elote.walk_forward().
Learning remains adaptive within each scoring window: predict a complete period, then learn its results before predicting the next. Earlier scoring results become training history in later folds, making the folds dependent. Inspect individual reports for poor later windows; this helper performs no search, automatic model choice, pooled aggregate, final-holdout management or confidence intervals, and makes no uncertainty claims.