Extraction
Records out of text. A document is stored, an extractor finds where it names a thing, and each place becomes a record. With the sample extractor, which asks a small model, and with your own code in Python or TypeScript.
Rows are one way records get into Spool. The other is text.
- A document is a text, stored as it arrived: a call note, an article, an email.
- An extractor reads documents and reports each place where one names a thing of an entity type. A place is a mention: the document, where it starts and ends, and the attributes the extractor read off it, such as a name.
- Each mention becomes a record. From there it is matched, placed in an identity, explained and corrected like a record that came from a row.
An extractor is a Python file or a TypeScript file: code that the server runs as a child process. The one spool init writes asks a small model that runs on your machine and is fetched on the extractor's first run. An extractor that asks a hosted model is your code, with your key.
Before you start
The examples use the sample project with its first file of rows loaded and matched:
spool init companies
cd companiesspool devLeave that running. In a second terminal, in the companies directory:
spool load samples/crm-1.csv
spool matchWhat the project needs
A source for the documents. Its columns are the attributes an extractor may give a mention:
# A source of documents: here, notes of calls. An extractor finds records in
# them, and `columns` is everything such a record may say.
columns: [name]A mapping from those attributes to the entity type's features, beside the mapping of the rows:
# `features` is what records are compared by, and `label` is the feature a
# record is called by. `mapping` says, for each source, which column each
# feature comes from, and the transform it goes through if it needs one. A
# record found in a note has no country.
label: name
features: [name, name_key, country]
mapping:
crm:
name: name
name_key: { from: name, transform: name_key }
country: country
notes:
name: name
name_key: { from: name, transform: name_key }A note says nothing about a country, so a record from one has none. The rule that keeps different countries apart does not hold against it.
And an extractor, under extractors/. spool init writes all three.
The sample extractor
# /// script
# requires-python = ">=3.10,<3.13"
# dependencies = ["fast-gliner>=0.4.1,<0.5"]
# ///
"""Finds the companies a document names, with a small model that runs here.
An extractor reads documents and says where they name something, and each
thing it finds becomes a record. Spool runs this file with `uv`, which
installs what the lines above list. The first run fetches about 330 MB, most
of it the model, and later runs fetch nothing. No document leaves this
machine and there is no key to set.
"""
import sys
from fast_gliner import FastGLiNER
from spool import extractor
# What to look for. On the left is the word the model is given, and the word
# decides what is found: "company" misses most universities, parties and
# charities, and "organization" finds them. Any word will do, "person" or
# "medicine" as well. On the right is the entity type of this project that
# such a find becomes a record of.
LOOK_FOR = {"company": "company"}
# GLiNER small: 245 MB, Apache-2.0, at the version it was tried with.
MODEL = "onnx-community/gliner_small-v2.1"
WEIGHTS = "onnx/model_q4f16.onnx"
VERSION = "8142fb00740ccea973e64b1272949ff48653df5e"
def load():
"""The model: read from this machine if it is here, and fetched if not."""
files = {"onnx_path": WEIGHTS, "allow_patterns": ["*.json", WEIGHTS], "revision": VERSION}
try:
return FastGLiNER.from_pretrained(MODEL, local_files_only=True, **files)
except FileNotFoundError:
print(f"Fetching the model {MODEL} (245 MB), once.", file=sys.stderr)
return FastGLiNER.from_pretrained(MODEL, **files)
# Loaded as the file starts, so that `spool build --install` fetches it too.
model = load()
@extractor
def find(document):
# The model reads 384 words at a time and ignores the rest, so a long
# document is read in pieces.
for piece in document.pieces(300):
for found in model.predict_entities(piece.text, list(LOOK_FOR)):
# The model counts in bytes, and Spool in characters.
start = len(piece.text.encode()[: found["start"]].decode())
end = start + len(found["text"])
# `name` is a column of sources/notes.yaml.
yield piece.mention(LOOK_FOR[found["label"]], start, end, name=found["text"])The server runs a .py file with uv, which has to be installed where the server runs. uv installs what the first lines of the file list. The first run fetches about 330 MB, most of it the model, and later runs fetch nothing.
@extractormarks the function the server calls, once for each document.document.pieces(300)gives the text in pieces of at most 300 words, because the model reads only so much at once.LOOK_FORholds the word the model is given, "company", and the entity type a find becomes a record of.piece.mention(entity_type, start, end, **attributes)is one mention, withstartandendcounted within the piece. An attribute the source does not list is refused.
Call with Dana Reyes on 12 November. She has moved to Acorn Tools Ltd and runs
purchasing there. They now ship through Larkfield Logistics Ltd, and she asked
whether we work with Brightwater Foods Ltd, their new supplier.spool extract stores a file as a document and runs the extractor over the documents it has not read:
spool extract samples/notes-1.txtsource notes
read 1 document from samples/notes-1.txt
loaded 1 new, 0 already held, 0 new versions
extractor companies
found 3 records in 1 document: 3 new, 0 already held
company Acorn Tools Ltd notes:notes-1.txt[54-69]
company Larkfield Logistics Ltd notes:notes-1.txt[119-142]
company Brightwater Foods Ltd notes:notes-1.txt[179-200]
next `spool match` places them in identitiesThe document's key is its file's name. A file of the same name with other text is the next version of the document, and the same text again adds nothing.
spool matchspool show records --source notessource and key name identity
notes:notes-1.txt[54-69] Acorn Tools Ltd 01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-1.txt[119-142] Larkfield Logistics Ltd 01a1efc6-41cd-775d-8758-e733aa086f7d
notes:notes-1.txt[179-200] Brightwater Foods Ltd 01a1efc6-43a9-7403-b9f6-1f8ee76184edTwo of the three joined identities the CRM rows had made, and Brightwater Foods is new.
What a record from text keeps
MENTION=$(spool show records --source notes | awk 'NR==2 {print $1}')
spool show records $MENTIONrecord 00308454-974b-52ac-112f-80f483e081fd
type company
source notes
key fb9a1372-e552-f564-d2b5-c2a510bd408a#company:54-69
loaded 2026-11-18T21:54:51Z by run 6
identity 01a1efc6-41cd-775d-8758-e7300637659a
payload
name Acorn Tools Ltd
evidence
document fb9a1372-e552-f564-d2b5-c2a510bd408a
end 69
extractor companies
key notes-1.txt
start 54
text Acorn Tools Ltd
version eacb9397cacc366a1cd065d8f328039484a5e5d4fbf49b8ca1f2a3cbd34f8c92
in document notes:notes-1.txt, characters 54 to 69
Call with Dana Reyes on 12 November. She has moved to [Acorn Tools Ltd] and runs purchasing there. They now ship through Larkfield Logistics Ltd, and s…The evidence names the document, the characters, the extractor and the version of the extractor, which is a hash of its file. The last lines show the place in the text. The studio's Documents screen shows the whole document with every mention marked.
spool show documentsdocument source and key characters read by
fb9a1372-e552-f564-d2b5-c2a510bd408a notes:notes-1.txt 222 companies in run 5 (3 mentions)Your own code, in Python
An extractor of your own is another file in extractors/. This one reads a second note:
Dana called again. Acorn Tools are moving their account to Kestrel Analytics
next quarter, and Larkfield will keep the deliveries.This file has the shape of an extractor that asks a hosted language model. ask stands where the model's answer goes: here it gives two names from a fixed list, so that the example runs with no key.
"""Finds the companies a note names, by asking for them and locating each answer in the text."""
from spool import extractor
KNOWN = ["Acorn Tools", "Kestrel Analytics"]
def ask(text):
"""The names in `text`. A real extractor sends the text to a model here."""
return [name for name in KNOWN if name in text]
@extractor
def find(document):
for quote in ask(document.text):
for start, end in document.find(quote):
yield document.mention("company", start, end, name=quote)document.textis the text, anddocument.keyits key.document.find(quote)gives each place the quote is in the text. Ask a model for quotes and not for positions: models count characters badly.document.mention(entity_type, start, end, **attributes)is one mention, withstartandendcounted within the whole text.
The project now has two extractors, so the command says which:
spool extract notes-2.txt --extractor namessource notes
read 1 document from notes-2.txt
loaded 1 new, 0 already held, 0 new versions
extractor names
found 3 records in 2 documents: 3 new, 0 already held
company Acorn Tools notes:notes-2.txt[19-30]
company Kestrel Analytics notes:notes-2.txt[59-76]
and more: `spool show records --source notes` lists every record of the source
next `spool match` places them in identitiesIt read both notes, because an extractor reads every document it has not read. In the first note it found "Acorn Tools" inside "Acorn Tools Ltd", which the sample extractor had found too. Those are two records for nearly the same place, from two extractors, and a match puts both in Acorn's identity:
spool matchspool show records --source notessource and key name identity
notes:notes-1.txt[54-65] Acorn Tools 01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-1.txt[54-69] Acorn Tools Ltd 01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-1.txt[119-142] Larkfield Logistics Ltd 01a1efc6-41cd-775d-8758-e733aa086f7d
notes:notes-1.txt[179-200] Brightwater Foods Ltd 01a1efc6-43a9-7403-b9f6-1f8ee76184ed
notes:notes-2.txt[19-30] Acorn Tools 01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-2.txt[59-76] Kestrel Analytics 01a1f09f-0c50-7164-ba08-984961afefc8Kestrel Analytics is new. "Larkfield" alone was not in the list, and a model might or might not have given it.
With a model
Replace the body of ask with a call to a model's API and return the names it gives. The repository has a whole one, llm.py, which uses only Python's standard library, speaks the chat-completions API, and reads its key and model from the environment. It tells the model what each entity type is from a file of the project, entities/<type>/extract.md:
A company is a business the text names: a legal entity such as a corporation,
a limited company or a partnership.
Quote the name exactly as the text writes it, with its legal form if the text
gives one. A product, a person, a place or a public body is not a company.
Attributes:
- `name`: the name as quoted.An extractor that takes (documents, task) in place of (document) gets every document at once and reads those instructions with task.instructions("company").
No test in the repository runs llm.py against a real model, because a test that needs a key and answers differently each time is not a test. An extractor that calls a model costs what the model costs, for every document, and the server stops one that runs longer than --task-timeout, which is 600 seconds unless set.
In TypeScript
The same extractor as a .ts file, which the server runs with Bun:
import { extractor } from "spool-strategy";
const KNOWN = ["Acorn Tools", "Kestrel Analytics"];
await extractor(async (task) => {
const documents = await task.documents();
return documents.flatMap((document) =>
KNOWN.flatMap((name) =>
document.find(name).map(([start, end]) => document.mention("company", start, end, { name })),
),
);
});spool dev picks up a new extractor file as it is saved, and spool info lists what the project has:
spool infoproject .
manifest 9e083661f159998deacca01381d4a683c2eb295b258f0ad623ffad75159af376
entity type `company`
features record_id UUID, name VARCHAR, name_key VARCHAR, country VARCHAR
label name
sources crm, notes
block name_prefix
no_match different_country
match similar_name
extract extract.md says what an extractor looks for
extractor `companies` (python, extractors/companies.py)
extractor `names` (python, extractors/names.py)
extractor `names_ts` (typescript, extractors/names_ts.ts)Over HTTP and MCP
| Store a document | Run an extractor | |
|---|---|---|
| Command | spool extract <file> does both | spool extract with no file |
| HTTP | POST /v1/documents | POST /v1/extractions |
| MCP | load_documents | run_extraction |
Getting started has a call of each. POST /v1/extractions takes extractor, source to keep to the documents of one source, again to read every document again, and limit.
How good is it
An extractor is scored apart from matching. spool bench runs a scenario with labelled mentions and reports what the extractor found, missed and made up. The repository's benchmarks/extraction/company-news has ten news items, a rule-based extractor in both languages, and its scores.
It also reports the two together, end to end: the identities against the true entities over every labelled mention, where a mention the extractor missed is with nothing. benchmarks/extraction/linked-docred measures all three on 5,051 Wikipedia articles.
What extraction does not do
- Documents are text. Spool does not read PDF or HTML. Turn them into text first.
- The sample model finds what its word asks for. Given "company", it misses most universities, parties and charities.
- No model is in the binary. The sample extractor fetches its model on its first run. Every extractor is a Python or TypeScript file, and needs
uvorbunwhere the server runs. - A mention is a place. Two mentions of one company in one document are two records, in one identity.
- A changed extractor does not read again by itself.
--againhas it read every document again. - A document is one request. A file larger than the API's limit of 256 MiB cannot be stored.