Your model was trained on last quarter.
Know when its inputs move.
Chalk compares each feature’s live values with the dataset you trained on, using a two-sample Kolmogorov–Smirnov test sized to the evidence in each window. Drag a model input away from its baseline and watch the statistic, its critical value and the alert respond.
Push a feature off its baseline
Five inputs of a fictional loan-approval model, each with a 5,000-row baseline. Shift or widen the live window, change how many rows it holds, then inject drift to watch a scheduled monitor catch it. Every number is computed by the same KS math Chalk’s monitors run.
The live windows are synthetic: a 1 vCPU Chalk ScalingGroup draws a fresh sample for each slider position and compares its 99-point quantile summary with the baseline’s. Its KS code reproduces the monitor implementation’s results exactly on a set of reference cases. The company, model and data are fictional.
How the KS monitor works
A model’s accuracy usually decays before anyone notices, because its inputs changed: a new marketing channel, a pricing change, an upstream bug. A KS monitor watches the inputs directly, one feature at a time.
A baseline you already have
Name the offline query you trained or validated on. Chalk keeps a quantile summary of every feature in it, so the baseline costs nothing extra to store.
A live window per evaluation
Each time the monitor runs, Chalk summarizes the values the feature actually took in production over the monitor's window, with the same 99 percentiles.
A test that knows its sample size
The KS statistic is the largest gap between the two CDFs. The critical value shrinks as the windows grow, and ks_test reports how far past it the gap is.
A baseline and one formula
Save the data your model was trained or validated on as a named dataset. Then create a chart monitor whose formula is ks_test(dataset, feature) with a threshold of 0, through the API or under Alerting → Metric Charts with the KS TEST formula.
Each evaluation compares the feature’s values in the monitor’s window with the baseline and opens an incident when the gap clears the critical value.
Feature drift docsfrom chalk.client import ChalkClient
client = ChalkClient()
# Any completed offline query saved under a name becomes a KS baseline:
# Chalk stores per-feature statistics for every dataset it writes.
baseline = client.offline_query(
input=reference_applications, # e.g. last quarter's loans
output=[
"loan_application.debt_to_income",
"loan_application.amount",
"applicant.credit_utilization",
],
dataset_name="loan-model-baseline",
recompute_features=True,
)
print(baseline.dataset_id)Drift monitoring built in
The features your model reads are already in Chalk, so the baseline and the live values are too. There is no second pipeline to build.
Any numeric feature
Continuous, discrete or capped. Repeated values are read at the top of their plateau, so integer counts and features pinned at a limit don't report phantom drift.
Sized to the evidence
The critical value comes from both window sizes, and is floored at the summary's resolution, so a busy feature isn't flagged for differences the summary can't resolve.
Alerts where you work
A KS monitor is an ordinary chart monitor: send it to Slack, PagerDuty or incident.io, mute it, and see its incidents next to every other alert.
Formulas you can compose
ks_stat, ks_threshold and ks_test are MQL functions. Chart them, combine them with other series, or pin the baseline to a specific dataset revision.
Baselines that move with you
Point the monitor at a dataset name and it follows the latest revision. Re-run the offline query after a retrain and the baseline updates with it.
Runs in your cloud
The statistics, the windows and the evaluation all stay in your Chalk environment, next to the features the model reads.
Catch drift before your metrics do
Monitor every model input against the data it was trained on. Talk to an engineer about your models.