Explore feature drift with Chalk monitoring
This demo shows how to run a KS comparison on Chalk CPUs to see how a feature distribution differs from its baseline. Shift a synthetic input with the sliders, then explore the code for configuring a Chalk drift monitor.
Make a distribution change large enough to trigger an alert
Compare the baseline and current distributions, then see how the same difference is evaluated over time.
Try this experiment · expected results
1Change one input
Start with Debt-to-income. Move Shift the live window and compare the two distribution plots.
Expected: The KS statistic measures the largest gap; the alert comparison depends on the threshold too.
2Run the monitor simulation
Select Inject drift and run and inspect the 24 evaluation windows.
Expected: Drift starts at window 12; compare the later scores with the alert boundary.
The live windows are synthetic: a 1 vCPU Chalk ScalingGroup draws a fresh sample for each slider position and compares its 99-point quantile summary with the baseline’s. Its KS code reproduces the monitor implementation’s results exactly on a set of reference cases. The company, model and data are fictional. Quantile summaries approximate the raw-sample test, so the nominal 5% significance level is not a guaranteed false-alert rate. A distribution change alone does not establish a drop in model accuracy.
Built on Chalk
Chalk’s chart monitors compare feature distributions with a named dataset baseline. This playground runs the same KS calculation in a CPU ScalingGroup using synthetic windows; it simulates the monitor’s evaluations.
Reuse a dataset baseline
Save an offline query as a named dataset. Chalk’s monitor uses its stored feature statistics as the reference distribution.
Compare a live window
A chart monitor compares the baseline with feature values observed in its evaluation window. Both sides use quantile summaries rather than retaining every raw value for the test.
Account for sample size
The KS threshold uses both observation counts and a floor for the quantile summary’s resolution. The ks_test formula reports the statistic minus that threshold.
Turn the comparison into an alert
Configure a chart monitor to alert when ks_test exceeds zero. Choose an evaluation schedule and an alert channel for the feature you want to watch.
Run it on your own
You will need Chalk credentials, a dataset of reference rows and a numeric feature with live observations. Save the baseline as a named dataset, then configure a chart monitor through the API or Alerting → Metric Charts.
Use ks_test(dataset, feature) with a threshold of zero and alert when the formula exceeds it. Select your evaluation window and alert channel. The snippets show the setup; replace the sample feature names, credentials and channel with your environment’s values.
from chalk.client import ChalkClient
client = ChalkClient()
# Any completed offline query saved under a name becomes a KS baseline:
# Chalk stores per-feature statistics for every dataset it writes.
baseline = client.offline_query(
input=reference_applications, # e.g. last quarter's loans
output=[
"loan_application.debt_to_income",
"loan_application.amount",
"applicant.credit_utilization",
],
dataset_name="loan-model-baseline",
recompute_features=True,
)
print(baseline.dataset_id)Monitor feature drift on Chalk
Talk to an engineer about comparing numeric model inputs with a reference dataset and configuring drift alerts for your workflow.