Carbon & Cost-Aware Scheduling for Flexible AI Compute
Forecast uncertain grid conditions and schedule flexible compute across time and candidate zones using carbon, electricity cost, and risk-aware objectives.
Training and batch inference workloads are deadline-bound, not instant. Grid carbon intensity and electricity price both move by the hour, so the choice of window is a real decision with a real objective. GridShift forecasts what the operating day will actually look like, quantifies how wrong that forecast might be, and then optimises the schedule under a deadline constraint.
NYISO publishes a day-ahead implied marginal emission rate before the operating day. GridShift models the residual between that signal and the realised real-time value, so the benchmark to beat is the market operator's own forecast.
Split conformal prediction on a chronologically separate calibration window turns each point forecast into a distribution-free interval, which the risk-aware policy schedules against.
Carbon and cost disagree. Every feasible window is scored on both, the non-dominated set is computed, and a preference weight picks a point on the frontier.
Four public MIS reports are downloaded as monthly archives, verified by SHA256, parsed into UTC-safe timestamps and aligned into a single zone-hour mart with real SQL.
| Loading… |
Real-time dispatch intervals are stamped interval-ending and aggregated to zone-hours with an observation count and completeness ratio. Daylight-saving transitions are resolved by file order, so the repeated autumn hour and the missing spring hour are both handled explicitly rather than dropped.
XGBoost is given explicit lags, rolling statistics and calendar encodings. The causal PyTorch TCN is given the raw 168-hour history and the next day's known day-ahead covariates, and has to learn the structure itself. Both are scored on identical rows over the same chronological holdout.
| Loading… |
A real operating day from the chronological test window. The shaded band is the 90% conformal prediction interval; the highlighted block is the window the ML carbon scheduler selected using only information available before the day began. This is historical data, not a live grid feed.
These controls call the same POST /schedule endpoint the API exposes, which
runs the real deadline-constrained scheduler and Pareto optimiser over the prepared historical panel.
Every point is one feasible schedule for the scenario above. A schedule is Pareto-efficient when no other schedule is at least as good on both objectives and strictly better on one. Moving the preference slider moves the selected point along the frontier.
| Loading… |
| Loading… |
| Loading… |
Cross-zone scheduling assumes compute capacity is available in the evaluated zones. It does not imply that AI data-centre capacity actually exists in every NYISO zone, and it ignores the network transfer, data-locality and latency costs a real migration would incur. Read it as an upper bound on what spatial flexibility could be worth.
Neighbouring hours are strongly autocorrelated and all eleven zones share the same grid conditions, so shuffling rows leaks the answer through both time and space. Every partition boundary here is a calendar date, and every zone belonging to one operating date lands in the same partition. The calibration window sits chronologically between development and test, and the test window is strictly last.
NYISO already publishes a day-ahead implied marginal emission rate. Beating a naive mean would prove nothing; the only interesting claim is that historical temporal patterns improve on the market operator's own published signal. So the models forecast the residual against it, and the reported improvement is measured against that benchmark.
The schedule is committed at the start of the operating day. Allowed inputs: the published day-ahead IMER and LBMP for the coming 24 hours, the calendar, the zone, and real-time observations timestamped strictly before the day begins. Lag and rolling features are masked against that boundary — which matters on the 25-hour autumn day, where a plain 24-hour shift would reach into the operating day itself.
After the point model is frozen, absolute errors are measured on the held-out calibration window and the finite-sample conformal quantile of those errors becomes the interval half-width. It is distribution-free and requires no assumption about the error shape — only that the calibration and test windows are exchangeable, which is exactly the assumption a regime shift would break.
The blend weight is searched over {0, 0.25, 0.5, 0.75, 1} on development folds only, where 0 and 1 are the two pure models. The blend is adopted only if it genuinely beats both members out of sample. Whether it does is reported above either way.
The oracle policy picks the best window using the realised real-time outcome, which no scheduler can know in advance. It exists to bound how much of the available improvement any forecast could possibly capture — the gap to it is regret. It is never presented as deployable.