Project structure
Every file a Spool project can hold, with an example of each, the keys it takes, and how a file's path becomes its name.
A Spool project is a directory, meant to be kept in Git. It declares where records come from, what kinds of thing they describe and how those are matched. spool dev reads it as you save, and spool build compiles it into the one file a deployed server reads.
This page is a reference: every file a project can hold, with an example of each and a link to the page that explains it. The examples are files spool init writes, and one that it does not.
spool init companies
cd companiesThe directory
companies/
├── sources/
│ ├── crm.yaml the source `crm`: its key and its columns
│ └── notes.yaml the source `notes`: what a record from a note may say
├── transforms/
│ └── name_key.sql the transform `name_key`: one SQL expression
├── entities/
│ └── company/ the entity type `company`
│ ├── entity.yaml its features, and how each source maps onto them
│ ├── rules/
│ │ ├── block/ which pairs are compared
│ │ │ └── name_prefix.sql
│ │ ├── match/ which pairs are the same thing
│ │ │ └── similar_name.sql
│ │ └── no_match/ which pairs are not
│ │ └── different_country.sql
│ ├── strategies/ optional: matching written in Python or TypeScript
│ └── extract.md optional: what an extractor is told about the type
├── extractors/
│ └── companies.py the extractor `companies`: a Python file that asks a small model
├── samples/ sample rows and a sample note. Not read by the server
├── .gitignore
└── .spool/ the development store and the compiled manifest. Not committedThe path is the name
Nothing is registered by hand. entities/company/ declares the entity type company. entities/company/rules/match/similar_name.sql is its match rule similar_name. sources/crm.yaml is the source crm. Renaming a file renames the thing.
Use lowercase letters, digits and underscores, starting with a letter: that is accepted for every kind of name. An entity type's name may have 48 characters at most.
Source files
sources/<source>.yaml. Explained in Records and sources.
# A source of rows. `key` is the column that tells one row from another, and
# `columns` is every column a row may have.
key: id
columns: [id, name, country]| Key | Meaning |
|---|---|
columns | Every column a row may have. A row with another column is refused. Required |
key | The column that tells one row from another. Required for a source of rows. A source of documents has none: its records are keyed by where they were found |
evidence | A column of JSON text that says where a row was found. It is kept as the record's evidence and is not part of what is compared. Optional |
There are no other keys, and a file with one is refused.
# A source of documents: here, notes of calls. An extractor finds records in
# them, and `columns` is everything such a record may say.
columns: [name]Transform files
transforms/<transform>.sql. The word lambda, the names of the arguments, and one DuckDB expression that turns column values into a feature. Lines that start with -- are comments. A transform with no file is the DuckDB function of that name, such as upper. Explained in Entity types and features.
-- A transform is `lambda`, the names of its arguments, and a DuckDB
-- expression over them.
lambda name: lower(trim(name))Entity type files
entities/<type>/entity.yaml. Explained in Entity types and features.
# `features` is what records are compared by, and `label` is the feature a
# record is called by. `mapping` says, for each source, which column each
# feature comes from, and the transform it goes through if it needs one. A
# record found in a note has no country.
label: name
features: [name, name_key, country]
mapping:
crm:
name: name
name_key: { from: name, transform: name_key }
country: country
notes:
name: name
name_key: { from: name, transform: name_key }| Key | Meaning |
|---|---|
features | The columns of the type's feature table, which is what rules read |
label | The feature a record is called by when it is shown |
mapping | For each source, where each feature comes from: a column, or { from: <column>, transform: <transform> }. from may be a list of columns for a transform with several arguments. A feature a source does not map is empty for its records |
There are no other keys, and a file with one is refused.
Rule files
entities/<type>/rules/<kind>/<rule>.sql. One SQL condition over two rows of the type's feature table, l and r. The directory is the kind of rule. Explained in Matching.
| Directory | A rule there says |
|---|---|
rules/block/ | This pair is worth comparing |
rules/match/ | This pair is the same thing |
rules/no_match/ | This pair is not the same thing. It wins over every match rule |
-- A block rule says which pairs are worth comparing: here, two names that
-- begin with the same four characters. `l` and `r` are two rows of the
-- feature table.
left(l.name_key, 4) = left(r.name_key, 4)-- A match rule says a pair is the same thing: here, two names that are
-- close. The similarity is 1 for equal names and 0 for names with nothing
-- in common.
jaro_winkler_similarity(l.name_key, r.name_key) >= 0.92-- A no_match rule says a pair is not the same thing, and wins over any
-- match rule. This one holds only when both rows have a country.
l.country <> r.countryExtractors
extractors/<name>.py or .ts. An extractor finds records in documents. Explained in Extraction, which shows the one spool init writes, extractors/companies.py, and writes one in each language.
A .py file is run with uv and a .ts file with Bun, as a child process of the server.
entities/<type>/extract.md is prose: what a thing of this type is and how to quote it. An extractor written in code can read it, to tell a language model what to look for. Extraction has one.
Strategies
entities/<type>/strategies/<name>.py or .ts. A strategy is matching written in code, for what SQL conditions cannot say: a trained model, a library, a call to a service. It is given the features of the records, new and known, and answers with judgements. Spool turns those into identities by the same fixed rules as for its own. A directory of the same name beside the file holds whatever else it needs.
"""Judges two records a match when their name keys are equal."""
from spool import strategy
@strategy
def judge(features):
rows = features.to_pylist()
judged = []
for new in rows:
if not new["is_new"]:
continue
for other in rows:
# Each pair once: a new record with every known one, and with every new one before it.
if other["record_id"] == new["record_id"] or (other["is_new"] and other["record_id"] > new["record_id"]):
continue
if other["name_key"] == new["name_key"]:
judged.append({"left_id": new["record_id"], "right_id": other["record_id"], "verdict": "match", "score": 1.0})
return judgedfeatures is an Arrow table with a row for each record of the type: its features, record_id, is_new, and identity_id for a record that has one. A judgement names two records and gives a verdict, match, no_match or unsure, with an optional score and detail.
A match uses a strategy when it is started with its name:
spool devspool load samples/crm-1.csv
curl -s -X POST http://127.0.0.1:7765/v1/match -d '{"entity_type": "company", "strategy": {"name": "same_key"}, "wait_ms": 60000}'spool show recordssource and key name identity
crm:c-1 Acorn Tools 01a1efc6-41cd-775d-8758-e7300637659a
crm:c-2 ACORN TOOLS 01a1efc6-41cd-775d-8758-e7300637659a
crm:c-3 Acorn Tools Ltd 01a1f0a3-638e-74a5-a177-9bffb14d9c38
crm:c-4 Larkfield Logistics 01a1efc6-41cd-775d-8758-e733aa086f7d
crm:c-5 Halden Group 01a1efc6-41cd-775d-8758-e7329447fd9a
crm:c-6 Halden Group 01a1efc6-41cd-775d-8758-e7329447fd9aEqual keys are a weaker rule than close names, so "Acorn Tools Ltd" is alone here. A strategy replaces the rules, the country rule among them, so the two Halden Groups are one. spool info lists what the project declares:
spool infoproject .
manifest 43818eb1654f89696c341d3bea519c22dd3bfb68f67e8f01edfdb38e59781926
entity type `company`
features record_id UUID, name VARCHAR, name_key VARCHAR, country VARCHAR
label name
sources crm, notes
block name_prefix
no_match different_country
match similar_name
strategy same_key (python, entities/company/strategies/same_key.py)
extractor `companies` (python, extractors/companies.py)The repository has a strategy that runs Splink, in benchmarks/resolution/people.
What Spool writes
.spool/ holds what is made from the project and is not committed. spool init puts it in .gitignore. spool dev keeps its store there, as dev.duckdb, with the accounts file beside it, and spool build writes the manifest there. Deploying covers spool build and spool apply.
Every run records the hash of the manifest it ran with, so a judgement can be traced to the version of the project that produced it. spool info and spool dev print that hash.
What is ignored
A file in the project's directories that this version does not use is listed by spool info under not used by this version, and has no effect. Files at the root of the project, such as a README, and the samples/ directory are not read at all.
The command line lists the commands that read a project.