Redact sensitive log data with models on Chalk CPUs
This demo shows how to run a fine-tuned DistilBERT on Chalk CPUs to tag and redact eight types of PII in log lines. Watch the synthetic logs or scan a sample, then explore the code to see how it works.
See what stays in the log and what gets removed
Compare each incoming line with its redacted output. Names, emails, and other detected values become labeled placeholders; the rest of the message stays readable. Pause the live updates to inspect a pair, or scan one sample to compare regex, model, and combined detection.
Every name, email, phone, address, card, SSN, IP and key here is fictional: phones in the 555-01xx range, example and .test email domains, published test card numbers, the 987-65-432x advertising SSNs and documentation IP ranges. The tail, the scan box and the SQL queries run live on a Chalk ScalingGroup and function each time; pasted lines are never stored or logged.
Results on synthetic logs
The model catches names that regex misses. Validators reject false matches, such as version numbers mistaken for IP addresses. These results use synthetic logs; test on your own data.
Evaluation details and live latency
2,000 held-out log lines with 3,829 labelled entities, written from templates and names the model never trained on. A prediction counts when its label matches and it overlaps the true span by at least half.
The test lines use unseen templates and names but the same kinds of values. Expect lower scores on your own logs until you add representative examples to the training set.
| Entity · precision / recall | Regex only | Model only | Model + validators |
|---|---|---|---|
| NAME | – / 0.0% | 99.8% / 100.0% | 99.8% / 100.0% |
| 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% | |
| PHONE | 59.4% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% |
| ADDRESS | 100.0% / 100.0% | 99.3% / 100.0% | 100.0% / 100.0% |
| CARD | 51.6% / 100.0% | 90.6% / 98.6% | 100.0% / 100.0% |
| SSN | 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% |
| IP | 96.5% / 100.0% | 98.9% / 100.0% | 100.0% / 100.0% |
| API KEY | 100.0% / 78.6% | 50.0% / 100.0% | 100.0% / 99.4% |
| All entities | 81.8% / 71.9% | 94.4% / 99.9% | 99.9% / 99.9% |
Where the model beats regex
Where validators save the model
Live latency on this page
From the live tail: one ScalingGroup replica with 2 CPU cores, model inference only.
Built on Chalk
Chalk serves the CPU model and makes it callable from SQL. The example supplies the synthetic training data, token classifier, regex detectors and validators that decide which spans to redact.
Serve the tagger on CPUs
Chalk runs the fine-tuned DistilBERT in a function with two CPU cores. The demo’s live tail runs the same detection code in a CPU ScalingGroup.
Models inside queries
Call a deployed model from ChalkSQL with catalog_call. Combine its output with filters and aggregations in the same query.
Package code and weights
Define the Python dependencies, detection module and trained weights in the function’s image configuration. Chalk deploys them together for inference.
Set the replica bounds
The example configures the function with one to two replicas and a maximum batch size of 64. Tune those settings for the volume of logs your queries scan.
Run it on your own
You will need uv, the Chalk CLI and a Chalk environment. Clone the example, log in, generate the synthetic training set, train the tagger and evaluate it before deploying the Chalk function.
The quickstart deploys the SQL-callable function. The repository also includes the live-tail API and sample log-table setup. Add representative logs and entity types to the training data, then measure detection quality on your own held-out examples.
git clone https://github.com/chalk-ai/log-pii-detection
cd log-pii-detection
chalk login
# Synthetic logs with labelled PII, then fine-tune DistilBERT on them
./synth.py --out data
./train.py --data data --out model
# Precision and recall on held-out templates and names
./evaluate.py --data data --model model
# Serve the tagger as a Chalk function
./deploy_function.py --model-dir modelBuild log redaction on Chalk
Logs, support tickets, chat transcripts or prompts: talk to an engineer about running your own detection models next to your data.