Hours of footage.
Find the ten seconds you need.
Robotics and video teams collect far more first-person footage than anyone can watch. Describe a scenario in words and get back the clips that match. A vision-language model on a GPU embeds every clip once; each search is a single ChalkSQL query that embeds your text, ranks 2,492 clips and filters on metadata.
Search a smartwatch assembly line
Eight workers wore head cameras through their shifts at an electronics factory. Pick a scenario or describe your own, narrow it by station, activity or camera motion, and play the matches. Every search runs live; open the query to see exactly what ran.
Footage: a 416-clip subset (factory 51) of Egocentric-10K by Build AI, via Voxel51's subset, Apache-2.0, re-encoded to 640 px. Model: SigLIP 2 so400m (Apache-2.0). Searches run live on a Chalk ScalingGroup, function and GPU each time; identical searches are cached for ten minutes and free-text searches share a per-minute budget. Activity tags are zero-shot and approximate. Typed searches are never stored.
How it works
Embed every clip once, offline. At query time, embed only the words and let the query engine do the ranking and filtering.
Clips in a Volume
416 three-minute first-person recordings from an electronics assembly line, cut into 2,492 ten-second segments and stored in a Chalk Volume with thumbnails and four frames each.
A vision model on a GPU
SigLIP 2 (so400m, 384 px) runs on a GPU ScalingGroup. A segment's vector is the mean of its four frame embeddings; a query's vector comes from the same model's text tower.
Embedded once, in a batch
One ChalkSQL query sends every segment through a Chalk function to the GPU: 2,492 segments in about 200 seconds on one L4. A second query adds a zero-shot activity tag; a camera-motion score comes from the clip itself.
Searched with one query
Each search embeds the text with catalog_call, scores every stored vector with cosine similarity, filters on metadata and returns the top 12. The same plan works as a DataFrame.
Search your own footage
Cut your videos into segments, upload them to a Volume, deploy the model and the two functions, and run the batch query. The search is the same query in SQL or as a DataFrame; both run in your Chalk environment.
The embeddings are plain parquet, so you can filter them by anything you know about a clip: robot, site, task, operator, date or a label from your own model.
chalk-ai/scenario-searchWITH q AS (
SELECT catalog_call('function.scenario-embed-text',
'walking down the aisle between workbenches') AS v
)
SELECT s.segment_id, s.worker_id, s.clip_index, s.start_s, s.activity,
cosine_similarity_vector(s.embedding, q.v) AS score
FROM read_parquet('https://<app>/data/segments.parquet') AS s
CROSS JOIN q
WHERE s.motion >= 38 -- camera on the move
AND s.activity = 'walking' -- zero-shot tag from the batch job
ORDER BY score DESC
LIMIT 12What Chalk handles
Storing the media, serving the model on GPUs and putting it inside your queries.
Models inside queries
catalog_call puts a deployed model in the middle of a SQL query or DataFrame, next to joins, filters, window functions and aggregations.
GPUs that scale
ScalingGroups run any container on a GPU between replica bounds you set: one L4 for this demo, a fleet for a backfill over millions of clips.
Batching built in
max_batching_size lets one function call carry many rows, so embedding a whole table is not one network round trip per row.
Volumes for media
Clips, frames and embeddings live in a versioned Chalk Volume mounted into the app, so the data, the model and the queries share one environment.
Metadata and vectors together
Similarity is just another column. Filter by station, activity or motion in the same WHERE clause, before or after ranking.
Bring any model
Swap SigLIP 2 for a video model, a domain-tuned CLIP or your own encoder. The function name and the queries stay the same.
Make your video data searchable
Robot teleoperation logs, dashcam fleets or factory footage: talk to an engineer about running your own embedding models next to your data.