Verisoul Demo Recap

We just wrapped up an inspiring live demo with Verisoul where Neil, their co-founder, walked us through how they catch fraud in real-time. Starting with 1,000 labeled domains, Neil built three different detection systems on Chalk, each more sophisticated than the last.

This is exactly the type of challenge Verisoul tackles daily. They recently helped a major AI code editor discover they were bleeding $5M through free trial abuse — fraudsters were running 2,000+ free trials simultaneously from data center IPs! It’s the kind of brazen fraud that basic rules can’t fully catch.

How Verisoul thinks about fraud

Neil explained how Verisoul organizes their fraud detection into distinct modules — network intelligence, device fingerprinting, behavioral analysis. Their network module alone packs over 800 features, and that's just one piece of the system.

These modules pull fraud signals from everywhere with database lookups, API calls, and even accelerometer readings. Before Chalk, stitching together those data sources meant weeks of engineering work. Now they can just define a feature, connect it to a data source with a few lines of code, and immediately use it across training and production.

During the demo, Neil iterated through three different fraud detection approaches using real labeled data:

1. Basic domain intelligence

Check if domains have valid email servers and how long they've been registered. Domains without MX records were marked suspicious, those under a month old were marked risky.

The result: 42% accuracy.

2. Search result analysis

Use an external search API (serper.dev) to fetch Google results for each domain, then scan for keywords like "disposable" and "temporary." This is because throwaway domains mainly appear in blocklists and security forums. Neil showed how simple this integration was in Chalk: just a Python resolver making a standard requests.post() call.*

try:
    r = requests.post('https://google.serper.dev/search', json=payload, headers=headers)
except requests.RequestException as exc:
    chalk_logger.error(f'Error fetching SERP results: {exc=}')

The result: 54% accuracy.

*For production workloads, you can replace Python's requests with Chalk's native F.http_get expressions for faster HTTP calls!

3. LLM-powered analysis

Here’s where things get interesting: use an LLM to generate a research report for each domain, and then another LLM to distill the report into a trust score.

The result leaped to 96% accuracy. Seems like sometimes, the best feature engineering is letting an LLM do the pattern recognition for you!

How Chalk makes ML + AI iteration fast

The jump from 42% to 96% accuracy was impressive, but even crazier was how quickly Neil got there by testing three completely different approaches in under an hour. Before Chalk — what Neil called 'the dark days' — each iteration would have taken weeks.

So how did Neil iterate this quickly? A few things made it possible:

1. The code stayed simple. Every feature was just a Python function that would work in both development and production.
2. He tested everything on a branch server, experimenting with production data without any risk to the live system. When he applied changes with chalk apply --branch, he got a completely isolated environment.
3. He could backtest immediately against his labeled dataset without waiting for ETL jobs or copying data around. Just client.offline_query() and see results in seconds.
4. Traditional ML and AI features worked seamlessly together. When Neil's final approach needed to combine Spanner queries, web scraping APIs, and LLM calls, they all became features in the same pipeline.

This combination — safe experimentation, instant feedback, and zero infrastructure overhead — is what let Neil try ‘wacky’ ideas like multimodal LLM analysis. When ML and AI features combine this naturally, the best solution no longer feels too complex to build!

Next steps and resources

If you want to see Neil's complete live coding session, including the Q&A where he dives deeper into Verisoul's architecture and fraud patterns, the full recording is available here.