Project structure

Every file a Spool project can hold, with an example of each, the keys it takes, and how a file's path becomes its name.

A Spool project is a directory, meant to be kept in Git. It declares where records come from, what kinds of thing they describe and how those are matched. spool dev reads it as you save, and spool build compiles it into the one file a deployed server reads.

This page is a reference: every file a project can hold, with an example of each and a link to the page that explains it. The examples are files spool init writes, and one that it does not.

spool init companies
cd companies

The directory

companies/
├── sources/
│   ├── crm.yaml                    the source `crm`: its key and its columns
│   └── notes.yaml                  the source `notes`: what a record from a note may say
├── transforms/
│   └── name_key.sql                the transform `name_key`: one SQL expression
├── entities/
│   └── company/                    the entity type `company`
│       ├── entity.yaml             its features, and how each source maps onto them
│       ├── rules/
│       │   ├── block/              which pairs are compared
│       │   │   └── name_prefix.sql
│       │   ├── match/              which pairs are the same thing
│       │   │   └── similar_name.sql
│       │   └── no_match/           which pairs are not
│       │       └── different_country.sql
│       ├── strategies/             optional: matching written in Python or TypeScript
│       └── extract.md              optional: what an extractor is told about the type
├── extractors/
│   └── companies.py                the extractor `companies`: a Python file that asks a small model
├── samples/                        sample rows and a sample note. Not read by the server
├── .gitignore
└── .spool/                         the development store and the compiled manifest. Not committed

The path is the name

Nothing is registered by hand. entities/company/ declares the entity type company. entities/company/rules/match/similar_name.sql is its match rule similar_name. sources/crm.yaml is the source crm. Renaming a file renames the thing.

Use lowercase letters, digits and underscores, starting with a letter: that is accepted for every kind of name. An entity type's name may have 48 characters at most.

Source files

sources/<source>.yaml. Explained in Records and sources.

sources/crm.yaml
# A source of rows. `key` is the column that tells one row from another, and
# `columns` is every column a row may have.
key: id
columns: [id, name, country]
KeyMeaning
columnsEvery column a row may have. A row with another column is refused. Required
keyThe column that tells one row from another. Required for a source of rows. A source of documents has none: its records are keyed by where they were found
evidenceA column of JSON text that says where a row was found. It is kept as the record's evidence and is not part of what is compared. Optional

There are no other keys, and a file with one is refused.

sources/notes.yaml
# A source of documents: here, notes of calls. An extractor finds records in
# them, and `columns` is everything such a record may say.
columns: [name]

Transform files

transforms/<transform>.sql. The word lambda, the names of the arguments, and one DuckDB expression that turns column values into a feature. Lines that start with -- are comments. A transform with no file is the DuckDB function of that name, such as upper. Explained in Entity types and features.

transforms/name_key.sql
-- A transform is `lambda`, the names of its arguments, and a DuckDB
-- expression over them.
lambda name: lower(trim(name))

Entity type files

entities/<type>/entity.yaml. Explained in Entity types and features.

entities/company/entity.yaml
# `features` is what records are compared by, and `label` is the feature a
# record is called by. `mapping` says, for each source, which column each
# feature comes from, and the transform it goes through if it needs one. A
# record found in a note has no country.
label: name
features: [name, name_key, country]

mapping:
  crm:
    name: name
    name_key: { from: name, transform: name_key }
    country: country
  notes:
    name: name
    name_key: { from: name, transform: name_key }
KeyMeaning
featuresThe columns of the type's feature table, which is what rules read
labelThe feature a record is called by when it is shown
mappingFor each source, where each feature comes from: a column, or { from: <column>, transform: <transform> }. from may be a list of columns for a transform with several arguments. A feature a source does not map is empty for its records

There are no other keys, and a file with one is refused.

Rule files

entities/<type>/rules/<kind>/<rule>.sql. One SQL condition over two rows of the type's feature table, l and r. The directory is the kind of rule. Explained in Matching.

DirectoryA rule there says
rules/block/This pair is worth comparing
rules/match/This pair is the same thing
rules/no_match/This pair is not the same thing. It wins over every match rule
entities/company/rules/block/name_prefix.sql
-- A block rule says which pairs are worth comparing: here, two names that
-- begin with the same four characters. `l` and `r` are two rows of the
-- feature table.
left(l.name_key, 4) = left(r.name_key, 4)
entities/company/rules/match/similar_name.sql
-- A match rule says a pair is the same thing: here, two names that are
-- close. The similarity is 1 for equal names and 0 for names with nothing
-- in common.
jaro_winkler_similarity(l.name_key, r.name_key) >= 0.92
entities/company/rules/no_match/different_country.sql
-- A no_match rule says a pair is not the same thing, and wins over any
-- match rule. This one holds only when both rows have a country.
l.country <> r.country

Extractors

extractors/<name>.py or .ts. An extractor finds records in documents. Explained in Extraction, which shows the one spool init writes, extractors/companies.py, and writes one in each language.

A .py file is run with uv and a .ts file with Bun, as a child process of the server.

entities/<type>/extract.md is prose: what a thing of this type is and how to quote it. An extractor written in code can read it, to tell a language model what to look for. Extraction has one.

Strategies

entities/<type>/strategies/<name>.py or .ts. A strategy is matching written in code, for what SQL conditions cannot say: a trained model, a library, a call to a service. It is given the features of the records, new and known, and answers with judgements. Spool turns those into identities by the same fixed rules as for its own. A directory of the same name beside the file holds whatever else it needs.

entities/company/strategies/same_key.py
"""Judges two records a match when their name keys are equal."""

from spool import strategy


@strategy
def judge(features):
    rows = features.to_pylist()
    judged = []
    for new in rows:
        if not new["is_new"]:
            continue
        for other in rows:
            # Each pair once: a new record with every known one, and with every new one before it.
            if other["record_id"] == new["record_id"] or (other["is_new"] and other["record_id"] > new["record_id"]):
                continue
            if other["name_key"] == new["name_key"]:
                judged.append({"left_id": new["record_id"], "right_id": other["record_id"], "verdict": "match", "score": 1.0})
    return judged

features is an Arrow table with a row for each record of the type: its features, record_id, is_new, and identity_id for a record that has one. A judgement names two records and gives a verdict, match, no_match or unsure, with an optional score and detail.

A match uses a strategy when it is started with its name:

spool dev
spool load samples/crm-1.csv
curl -s -X POST http://127.0.0.1:7765/v1/match -d '{"entity_type": "company", "strategy": {"name": "same_key"}, "wait_ms": 60000}'
spool show records
Output
source and key  name                 identity
crm:c-1         Acorn Tools          01a1efc6-41cd-775d-8758-e7300637659a
crm:c-2         ACORN TOOLS          01a1efc6-41cd-775d-8758-e7300637659a
crm:c-3         Acorn Tools Ltd      01a1f0a3-638e-74a5-a177-9bffb14d9c38
crm:c-4         Larkfield Logistics  01a1efc6-41cd-775d-8758-e733aa086f7d
crm:c-5         Halden Group         01a1efc6-41cd-775d-8758-e7329447fd9a
crm:c-6         Halden Group         01a1efc6-41cd-775d-8758-e7329447fd9a

Equal keys are a weaker rule than close names, so "Acorn Tools Ltd" is alone here. A strategy replaces the rules, the country rule among them, so the two Halden Groups are one. spool info lists what the project declares:

spool info
Output
project   .
manifest  43818eb1654f89696c341d3bea519c22dd3bfb68f67e8f01edfdb38e59781926

entity type `company`
  features  record_id UUID, name VARCHAR, name_key VARCHAR, country VARCHAR
  label     name
  sources   crm, notes
  block     name_prefix
  no_match  different_country
  match     similar_name
  strategy  same_key (python, entities/company/strategies/same_key.py)

extractor `companies` (python, extractors/companies.py)

The repository has a strategy that runs Splink, in benchmarks/resolution/people.

What Spool writes

.spool/ holds what is made from the project and is not committed. spool init puts it in .gitignore. spool dev keeps its store there, as dev.duckdb, with the accounts file beside it, and spool build writes the manifest there. Deploying covers spool build and spool apply.

Every run records the hash of the manifest it ran with, so a judgement can be traced to the version of the project that produced it. spool info and spool dev print that hash.

What is ignored

A file in the project's directories that this version does not use is listed by spool info under not used by this version, and has no effect. Files at the root of the project, such as a README, and the samples/ directory are not read at all.

The command line lists the commands that read a project.