Find the PII in your logs.
Before anyone else does.
A fine-tuned DistilBERT tags names, emails, phones, addresses, cards, SSNs, IPs and API keys in raw log lines, and cheap validators throw out the look-alikes. It runs as a Chalk function you can call from SQL. On held-out synthetic logs it reaches 99.9% precision and 99.9% recall, against 71.9% recall for regex alone.
Leaky logs, caught live
The tail below is a stream of synthetic log lines from a fictional shop, scanned in small batches as they arrive. Flip to redacted to see what would reach your log store. Then paste a line of your own, or redact a whole table with one ChalkSQL query.
Every name, email, phone, address, card, SSN, IP and key here is fictional: phones in the 555-01xx range, example and .test email domains, published test card numbers, the 987-65-432x advertising SSNs and documentation IP ranges. The tail, the scan box and the SQL queries run live on a Chalk ScalingGroup and function each time; pasted lines are never stored or logged.
Measured, not claimed
2,000 held-out log lines with 3,829 labelled entities, written from templates and names the model never trained on. A prediction counts when its label matches and it overlaps the true span by at least half. Regex alone gets 81.8% precision and 71.9% recall; the model alone 94.4% and 99.9%; together with validators, 99.9% and 99.9%.
These are synthetic logs. The test lines use unseen templates and names but the same kinds of values, so expect lower scores on your own logs until you add examples of them to the training set. The comparison between the three detectors is the part that carries over.
| Entity · precision / recall | Regex only | Model only | Model + validators |
|---|---|---|---|
| NAME | – / 0.0% | 99.8% / 100.0% | 99.8% / 100.0% |
| 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% | |
| PHONE | 59.4% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% |
| ADDRESS | 100.0% / 100.0% | 99.3% / 100.0% | 100.0% / 100.0% |
| CARD | 51.6% / 100.0% | 90.6% / 98.6% | 100.0% / 100.0% |
| SSN | 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% |
| IP | 96.5% / 100.0% | 98.9% / 100.0% | 100.0% / 100.0% |
| API KEY | 100.0% / 78.6% | 50.0% / 100.0% | 100.0% / 99.4% |
| All entities | 81.8% / 71.9% | 94.4% / 99.9% | 99.9% / 99.9% |
Where the model beats regex
Where validators save the model
Live latency on this page
From the live tail: one ScalingGroup replica with 2 CPU cores, model inference only.
How it works
Regex is great at shapes and blind to meaning. A small model reads context. Using each for what it is good at beats either one alone.
Synthetic, labelled logs
A generator writes log lines in text, JSON and access-log formats with PII planted at known offsets, plus look-alikes: order IDs shaped like phones, tracking numbers, version strings.
A small BERT, fine-tuned
DistilBERT learns to tag eight entity types token by token. The test split uses templates and names it never saw, so its scores measure generalization.
Validators keep it honest
Structured values must pass a cheap check (Luhn, NANP phone shape, valid IP, SSN groups) before they count, which removes false positives from both the model and regex.
Called from Chalk
The tagger is a Chalk function. ChalkSQL calls it on every row with catalog_call, and the same code runs this page's live tail on a ScalingGroup.
Redact your own logs
Generate the synthetic training set, fine-tune the tagger on CPU (about half an hour on 48 cores), check its scores, and deploy it as a Chalk function. Then call it from ChalkSQL over any log table your environment can read.
The model is fine-tuned from DistilBERT (Apache-2.0) on synthetic data only, so you can retrain it on your own entity types and log formats without licensing questions.
chalk-ai/log-pii-detectiongit clone https://github.com/chalk-ai/log-pii-detection
cd log-pii-detection
chalk login
# Synthetic logs with labelled PII, then fine-tune DistilBERT on them
./synth.py --out data
./train.py --data data --out model
# Precision and recall on held-out templates and names
./evaluate.py --data data --model model
# Serve the tagger as a Chalk function
./deploy_function.py --model-dir modelWhat Chalk handles
Serving the model, batching the calls and putting it inside your queries.
Runs on CPU
A 66M-parameter DistilBERT tags a batch of log lines in tens of milliseconds on two CPU cores. No GPU needed for this one.
Models inside SQL
catalog_call puts the model in the middle of a query, next to joins, filters and aggregations over every connected source.
Your logs stay in your cloud
The model runs in your Chalk environment, so log lines never pass through a third-party API on their way to being redacted.
Batching built in
max_batching_size lets one function call carry many rows, so scanning a table is not one network round trip per line.
Scales with the queue
Functions and ScalingGroups scale between replica bounds you set, from a single demo replica to a fleet behind your log pipeline.
Bring any model
Swap in a larger tagger, an ONNX export or your own entity types. The function signature and the SQL do not change.
Keep PII out of your log store
Logs, support tickets, chat transcripts or prompts: talk to an engineer about running your own detection models next to your data.