Five ways to write a feature.
One query to run them.
A fictional lender decides loan applications with features written in SQL, Python, underscore expressions, a remote function and a registered model. Pick an applicant, run the decision live, and watch Chalk plan the whole mixed graph as one query and execute it from the real trace.
Pick an applicant, then run the decision
Each applicant opens on a recorded run. Run live query sends a real online query for applicant.decision to a Chalk branch: BigQuery tables, a bureau API, a Chalk compute function and a scikit-learn model. The timeline comes from that query's trace. Saltmarsh Credit, its applicants and its bureau are made up.
One graph, five ways to write a feature
Click a node to see its source. Edges point from a feature to the features computed from it.
decision: str = F.if_then_else(
_.name_match_score < 0.6,
"review_identity",
F.if_then_else(
(_.default_probability > 0.45) | (_.debt_to_income > 0.6),
"decline",
F.if_then_else(_.default_probability > 0.25, "review", "approve"),
),
)- decision
- approve
Timing: derived: runs inside a native operator
Underscore expression, planned natively · computed for this query
What ran, when, from the query's trace
Chalk planned every feature above as one query, then ran independent branches at the same time: applicant.chalk.sql, payments.chalk.sql, bureau_report, name_match_score overlapped. Python, function and model bars are measured trace spans. BigQuery scans run as native operators with no span of their own, so their bars are inferred from the spans around them; expressions run inside native operators in well under a millisecond.
No planning span: Chalk reused the plan it compiled for an earlier query of this shape.
Cache: every value was computed for this query. The demo runs on a branch, which has no online store; in production, max_staleness serves features like the bureau report from the online store instead.
The plan Chalk built
Condensed from Chalk's explain output for this query. SQL, Python, expressions, the remote function and the model sit in one plan, and each BigQuery scan asks only for the columns the graph uses.
- ValuesPersister
- UnderscoreResolverOperator<['applicant.decision']>
- NaryMergeJoinOperator
- NaryMergeJoinOperator
- UnderscoreResolverOperator<['applicant.name_match_score']>
- NaryMergeJoinOperator
- GivensScan
- BigQueryNativeDataframeResolver<'applicant', ['BQ_CHALK_DEVELOP_TESTING']> → SELECT full_name, id, monthly_debt, requested_amount, stated_annual_income
- ScalarResolver<['lending.resolvers.bureau_report']>
- UnderscoreResolverOperator<['applicant.debt_to_income']>
- DataFrameResolver<'saltmarsh_default_risk__applicant_default_probability'>
- NaryMergeJoinOperator
- NaryMergeJoinOperator
- UnderscoreResolverOperator<['applicant.loan_to_income', 'applicant.late_payments__31536000__', 'applicant.payments_due__31536000__']>
- NaryMergeJoinOperator
- BigQueryNativeDataframeResolver<'payments', ['BQ_CHALK_DEVELOP_TESTING']> → SELECT applicant_id, days_late, due_at, id
- HasManyInputOperator
- BigQueryNativeDataframeResolver<'payments', ['BQ_CHALK_DEVELOP_TESTING']> → SELECT applicant_id, due_at, id
You write features. Chalk plans.
A real decision needs data from a warehouse, an API, custom code and a model. In Chalk they are all features of one entity, and one query computes whichever ones you ask for.
Write each feature in its best language
Warehouse data as SQL files, an API call in Python, aggregations and ratios as expressions, custom code as a function, a score from a registered model.
Chalk plans them as one query
A request for applicant.decision pulls in every feature beneath it. Chalk resolves the graph into a single plan, pushing filters and columns down into BigQuery.
Independent work runs in parallel
The warehouse scans and the bureau call start together. The function waits for both names, the model for all its inputs, and the decision for the model.
Each style, in a few lines
Every tab is taken from the project behind this demo. Put the SQL files and Python next to your feature classes, deploy the function and register the model, then chalk apply.
In the recorded runs on this page, Chalk executes the whole graph in two to five seconds, most of it the BigQuery scans; the function and the model each take tens of milliseconds.
-- applicant.chalk.sql: a SQL file resolver over the warehouse.
-- Chalk pushes the requested ids, and only the columns the query uses, into BigQuery.
-- source: BQ_CHALK_DEVELOP_TESTING
-- resolves: Applicant
SELECT id, full_name, stated_annual_income, monthly_debt, requested_amount
FROM demo_feature_graph.applicants
-- payments.chalk.sql: repayment history, joined to Applicant with has_many.
-- source: BQ_CHALK_DEVELOP_TESTING
-- resolves: Payment
SELECT id, applicant_id, due_at, amount, days_late
FROM demo_feature_graph.paymentsOne graph, no glue
The parts of a decision stop being separate services that you stitch together.
Mix styles freely
A SQL feature can feed an expression that feeds a model. No style is a separate system with its own pipeline, cache or deployment.
Pushdown into the warehouse
The SQL files have no WHERE clause. Chalk adds the requested ids and each window's filters to the BigQuery scan, and selects only the columns the graph uses.
Windows at decision time
Late payments in the last 90 and 365 days are counted over the has-many history when the decision is made, not in a nightly batch.
Functions for custom code
The name matcher is plain Python with a pip dependency, served by Chalk compute and called from an expression with catalog_call.
Models are features
The default-risk model is registered once and referenced by version. Chalk computes its seven inputs and the score in the same query.
Traced end to end
Every run records a trace. The demo reads it back to show which resolver ran when, which ran in parallel and where each value came from.
Bring every feature into one query
Talk to an engineer about building decisions from SQL, Python, functions and models in one feature graph.