Serve open video search models on Chalk GPUs
This demo shows how to run SigLIP 2 on Chalk GPUs to find video segments from a text description. Pick a scenario or describe one to search the footage, then explore the code to see how it works.
Check whether a video match shows the requested scenario
Search ranks short factory-video segments; playing a match is the evidence check.
Try this experiment · expected results
1Start with a preset
Choose Screen lit up for testing and play a returned segment.
Expected: Compare the visible action with the query rather than treating a high score as a confirmed label.
2Narrow the same search
Change one activity, worker, or camera-motion filter, then inspect the query.
Expected: The results show the combination of semantic ranking and structured filtering.
Footage: a 416-clip subset (factory 51) of Egocentric-10K by Build AI, via Voxel51's subset, Apache-2.0, re-encoded to 640 px. Model: SigLIP 2 so400m (Apache-2.0). Uncached searches run through a Chalk ScalingGroup, function and GPU; identical searches are cached for ten minutes and free-text searches share a per-minute budget. Activity tags are zero-shot and approximate. The app keeps search text and results in its in-memory cache.
Built on Chalk
Chalk hosts the embedding model, stores the media and runs the search query. The example averages four frame embeddings per segment, adds approximate activity tags and ranks matches by cosine similarity.
A GPU endpoint for embeddings
Chalk runs SigLIP 2 in an NVIDIA L4 ScalingGroup. The example’s text and segment endpoints use the same model to create comparable vectors.
Models inside queries
Call a deployed model from ChalkSQL with catalog_call. Combine its output with filters and aggregations in the same query.
Media in a Chalk Volume
Clips, frames and embedding files live in a Chalk Volume mounted into the CPU web app. The GPU service fetches frames from that app over HTTPS.
Rank and filter together
ChalkSQL computes cosine similarity over the stored embeddings, applies metadata filters and returns the top matches. The example also includes the same search as a Chalk DataFrame.
Run it on your own
You will need uv, ffmpeg, Chalk credentials and an environment with NVIDIA L4 capacity. Set EMBED_TOKEN, prepare the video segments, upload the media and deploy the app, embedding service and two functions.
After the first function deployment, deploy or reactivate the environment so ChalkSQL sees the function names. Run the batch queries to embed and tag the segments. This demo scans all stored vectors per search; use a vector index for a much larger collection.
WITH q AS (
SELECT catalog_call('function.scenario-embed-text',
'walking down the aisle between workbenches') AS v
)
SELECT s.segment_id, s.worker_id, s.clip_index, s.start_s, s.activity,
cosine_similarity_vector(s.embedding, q.v) AS score
FROM read_parquet('https://<app>/data/segments.parquet') AS s
CROSS JOIN q
WHERE s.motion >= 38 -- camera on the move
AND s.activity = 'walking' -- zero-shot tag from the batch job
ORDER BY score DESC
LIMIT 12Make your video data searchable
Robot teleoperation logs, dashcam fleets or factory footage: talk to an engineer about running your own embedding models next to your data.