Limits
What Spool is not meant for, what it does not do yet, what stable IDs cost, and how large it has been run.
Read this page before you decide to use Spool. Its first commit was on 14 November 2026.
Not meant for
These follow from the design and are not planned to change.
- Deduplicating one file once. Spool exists to keep IDs the same as data arrives over time. If you have one file, will never get another, and nobody stores the IDs, use Splink or dedupe.
- A decision per record, in the request. Rows are loaded in batches and matched by runs. Loading a row does not answer whether it is a duplicate. A match of a few rows takes well under a second, and it is still a second call. Senzing resolves each record as it arrives.
- Streaming. There is no connection to Kafka or to a change feed. Something else has to collect rows and send them.
- Many writers or many machines. One server process owns one file of DuckDB, a database that runs inside the process. It uses the cores and memory of one machine. There is no replication and no clustering.
- Many tenants in one server. A store holds one project. Separate customers need separate servers.
- Embedding. There are no Python or Node bindings. Your code calls the server over HTTP.
- Golden records. Spool says which records are one thing. It does not produce one merged record with the best name and address, and it does not write anything back to a source.
- Matching with no rules. The built-in matcher runs SQL conditions that you write. There is no trained model inside. Splink, dedupe and Zingg each learn one. A model can be plugged in as a strategy, which is your own code in Python or TypeScript.
- Reading PDFs, HTML or images. Spool takes records from text through an extractor. Documents are text: turn anything else into text first.
Compared with other tools says what each of those tools does with IDs and when to pick it.
What stable IDs cost
An ID that never changes by itself has a price, and this is it.
- The result depends on the order rows arrive in. Two records that arrive together and match become one identity. The same two, each placed alone before a rule matched them, stay apart until someone merges them.
- A better rule suggests, and changes nothing. A rematch under new rules lists the merges the rules now imply. A person or an agent makes them. The suggestions are in the run's report and nowhere else.
- A match never splits an identity. A wrong match that was committed stays until someone splits it. When a row that held an identity together is taken back, a review item asks whether to split, and nothing is split by itself.
- A row that changes from A to B and back to A stays at B. The second A is content the server has seen under that key, so it is ignored. This is a known fault.
- An earlier version of a row still counts. It stays in its identity and still matches, so a row that was corrected can raise a review item through the version it replaced.
Matching shows the first two with rows.
Not yet
Getting it
- Releases are an archive for Linux on x86-64 and arm64 and for macOS on Apple silicon, and a container image. There is no Homebrew, apt or Cargo package and no Windows build.
- Spool is two files, the binary and the DuckDB library in
libbeside it. The archive and the image hold both. There is no single static binary.
Loading and matching
spool loadreads CSV files. Nothing reads a database, a warehouse or a file store.- A row that was loaded by mistake cannot be erased. It can be taken back with
spool retract: it leaves its identity and no later match looks at it, and it stays in the store. So there is no way to remove a person's data on request. - The rules by which a match places a record, starts an identity or leaves a record for review are fixed. There is no setting for them.
- A rule sees the features of two records and not their sources. It cannot say that one source never holds the same thing twice.
- No rule can say
unsure. A pair is matched, kept apart, or not judged. A strategy can. - Rules compare records of one entity type. Nothing matches a company against a person.
- An entity type is declared in the project's files. It cannot be added through the API.
- Features are computed by SQL expressions only.
Extraction
- A pattern extractor finds what a regular expression finds.
- An extractor that asks a model is a Python or TypeScript file and needs
uvorbunwhere the server runs. No model is included, and none has been run in the repository's tests. - Documents are text: Spool does not read PDF or HTML.
- An extractor that changed does not read the documents again until it is run with
--again.
Review
- A
possible_splititem is raised when a row is taken back or sent again. It is not raised when a row's content changes and the new version no longer matches, and a rematch reports no splits. - A retraction that was cut short and sent again raises no item for the rows the first attempt took back.
- Nothing notifies anyone that review items are waiting, and review items cannot be assigned to a person.
Agents
- Over MCP an agent can load rows and documents, run extractors and matches, look things up and change identities. It cannot approve a draft, add a person or make a token.
- A token has full access, so an agent that may read may also load and change. There are no read-only tokens and no tokens limited to one entity type.
- A request to
/mcpmay be 4 MiB at most, and a tool's answer 256 KiB. - ChatGPT on the web and Claude on the web cannot connect to a server that asks for a token, because Spool is not an OAuth server.
- How well an agent reviews has not been measured.
Access
- Tokens do not expire. They can be revoked.
- People sign in with a password or with Google. There is no other identity provider.
- Anyone who can sign in can read everything and change identities. Only adding people and making tokens is kept to admins.
Clients
- There is no published client library.
spool schemaprints the JSON Schema of the API. - The command line loads, matches, takes rows back, shows and explains. It cannot merge, split, undo, decide a review item or approve a draft. Those are HTTP calls, MCP tools, or buttons in the studio.
- Nothing is pushed to other systems when an identity changes. A consumer reads the log from the position it last saw.
- A changeset names records by their long IDs.
source:keyworks on the command line and for explaining a pair.
Operating it
- There is no backup command. A backup is a copy of the store file, taken while the server is stopped.
- Until 1.0, nothing is promised about a store written by one version opening in the next.
- There are no metrics and no tracing.
GET /v1/healthzsays the server is up.
How large
The repository's benchmarks ran the built-in rules on public data, on four cores and 16 GB of memory:
| Data | Records | Seconds | Peak memory | Pair precision | Pair recall |
|---|---|---|---|---|---|
| North Carolina voters, five sources | 500,323 | 15 | 1.4 GB | 0.988 | 0.881 |
| North Carolina voters, five sources | 5,000,000 | 210 | 13.4 GB | 0.880 | 0.879 |
| MusicBrainz songs, five sources | 193,750 | 8 | 0.7 GB | 0.997 | 0.862 |
| MusicBrainz songs, five sources | 1,937,500 | 67 | 4.3 GB | 0.959 | 0.863 |
Each row is a whole data set loaded and matched once, from nothing. The same rules lose precision as the set grows: twice the people in the same towns is four times the pairs of different people who look alike. The voter rules left 13,755 of the 5,000,000 records for review.
Memory grows with the number of pairs that are compared, so the block rules decide how far one machine goes. Nothing larger than 5,000,000 records has been tried.
Data that changes was measured once: five sources of about 100,000 voters each, 100,000 rows added, 50,000 taken back, everything matched again, and 10,000 sent again, in 216 seconds and 2.2 GB. Every record that was not taken back kept its identity. Taking the rows back left 70 possible_split review items, which covered every pair of records that a store built from nothing would have kept apart.
What it costs to add a day's rows to a store that already holds millions has not been measured.
The scenarios, the rules and the method are in the repository's benchmarks/ directory, and spool bench runs them.
A request to the API may be 256 MiB at most.
Awkward today
Each of these works and costs you something.
- The API answers in JSON on one line.
spool showis the readable way to see what a call did. - An identity is addressed only by its ID, which differs in every store, so a correction starts with a lookup.
spool show recordsputsc-10beforec-2, because keys are text.spool devsays nothing when a broken file is fixed by removing it.- A call that is refused still uses up a run number.
- Strategies and extractors need
uvorbunon the machine, and nothing says so until one is run.