---
title: "Extraction"
description: "Records out of text. A document is stored, an extractor finds where it names a thing, and each place becomes a record. With the sample extractor, which asks a small model, and with your own code in Python or TypeScript."
url: https://spool.alexpatow.com/docs/concepts/extraction
docs_index: https://spool.alexpatow.com/llms.txt
---

# Extraction

Records out of text. A document is stored, an extractor finds where it names a thing, and each place becomes a record. With the sample extractor, which asks a small model, and with your own code in Python or TypeScript.

Rows are one way records get into Spool. The other is text.

- A **document** is a text, stored as it arrived: a call note, an article, an email.
- An **extractor** reads documents and reports each place where one names a thing of an entity type. A place is a **mention**: the document, where it starts and ends, and the attributes the extractor read off it, such as a name.
- Each mention becomes a **record**. From there it is matched, placed in an identity, explained and corrected like a record that came from a row.

An extractor is a Python file or a TypeScript file: code that the server runs as a child process. The one `spool init` writes asks a small model that runs on your machine and is fetched on the extractor's first run. An extractor that asks a hosted model is your code, with your key.

## Before you start

The examples use the sample project with its first file of rows loaded and matched:

```sh
spool init companies
cd companies
```

```sh
spool dev
```

Leave that running. In a second terminal, in the `companies` directory:

```sh
spool load samples/crm-1.csv
spool match
```

## What the project needs

A source for the documents. Its columns are the attributes an extractor may give a mention:

```yaml title="sources/notes.yaml"
# A source of documents: here, notes of calls. An extractor finds records in
# them, and `columns` is everything such a record may say.
columns: [name]
```

A mapping from those attributes to the entity type's features, beside the mapping of the rows:

```yaml title="entities/company/entity.yaml"
# `features` is what records are compared by, and `label` is the feature a
# record is called by. `mapping` says, for each source, which column each
# feature comes from, and the transform it goes through if it needs one. A
# record found in a note has no country.
label: name
features: [name, name_key, country]

mapping:
  crm:
    name: name
    name_key: { from: name, transform: name_key }
    country: country
  notes:
    name: name
    name_key: { from: name, transform: name_key }
```

A note says nothing about a country, so a record from one has none. The rule that keeps different countries apart does not hold against it.

And an extractor, under `extractors/`. `spool init` writes all three.

## The sample extractor

```python title="extractors/companies.py"
# /// script
# requires-python = ">=3.10,<3.13"
# dependencies = ["fast-gliner>=0.4.1,<0.5"]
# ///
"""Finds the companies a document names, with a small model that runs here.

An extractor reads documents and says where they name something, and each
thing it finds becomes a record. Spool runs this file with `uv`, which
installs what the lines above list. The first run fetches about 330 MB, most
of it the model, and later runs fetch nothing. No document leaves this
machine and there is no key to set.
"""

import sys

from fast_gliner import FastGLiNER
from spool import extractor

# What to look for. On the left is the word the model is given, and the word
# decides what is found: "company" misses most universities, parties and
# charities, and "organization" finds them. Any word will do, "person" or
# "medicine" as well. On the right is the entity type of this project that
# such a find becomes a record of.
LOOK_FOR = {"company": "company"}

# GLiNER small: 245 MB, Apache-2.0, at the version it was tried with.
MODEL = "onnx-community/gliner_small-v2.1"
WEIGHTS = "onnx/model_q4f16.onnx"
VERSION = "8142fb00740ccea973e64b1272949ff48653df5e"

def load():
    """The model: read from this machine if it is here, and fetched if not."""
    files = {"onnx_path": WEIGHTS, "allow_patterns": ["*.json", WEIGHTS], "revision": VERSION}
    try:
        return FastGLiNER.from_pretrained(MODEL, local_files_only=True, **files)
    except FileNotFoundError:
        print(f"Fetching the model {MODEL} (245 MB), once.", file=sys.stderr)
        return FastGLiNER.from_pretrained(MODEL, **files)

# Loaded as the file starts, so that `spool build --install` fetches it too.
model = load()

@extractor
def find(document):
    # The model reads 384 words at a time and ignores the rest, so a long
    # document is read in pieces.
    for piece in document.pieces(300):
        for found in model.predict_entities(piece.text, list(LOOK_FOR)):
            # The model counts in bytes, and Spool in characters.
            start = len(piece.text.encode()[: found["start"]].decode())
            end = start + len(found["text"])
            # `name` is a column of sources/notes.yaml.
            yield piece.mention(LOOK_FOR[found["label"]], start, end, name=found["text"])
```

The server runs a `.py` file with [uv](https://docs.astral.sh/uv/), which has to be installed where the server runs. `uv` installs what the first lines of the file list. The first run fetches about 330 MB, most of it the model, and later runs fetch nothing.

- `@extractor` marks the function the server calls, once for each document.
- `document.pieces(300)` gives the text in pieces of at most 300 words, because the model reads only so much at once.
- `LOOK_FOR` holds the word the model is given, "company", and the entity type a find becomes a record of.
- `piece.mention(entity_type, start, end, **attributes)` is one mention, with `start` and `end` counted within the piece. An attribute the source does not list is refused.

```text title="samples/notes-1.txt"
Call with Dana Reyes on 12 November. She has moved to Acorn Tools Ltd and runs
purchasing there. They now ship through Larkfield Logistics Ltd, and she asked
whether we work with Brightwater Foods Ltd, their new supplier.
```

`spool extract` stores a file as a document and runs the extractor over the documents it has not read:

```sh
spool extract samples/notes-1.txt
```

```text title="Output"
source     notes
read       1 document from samples/notes-1.txt
loaded     1 new, 0 already held, 0 new versions
extractor  companies
found      3 records in 1 document: 3 new, 0 already held

  company  Acorn Tools Ltd          notes:notes-1.txt[54-69]
  company  Larkfield Logistics Ltd  notes:notes-1.txt[119-142]
  company  Brightwater Foods Ltd    notes:notes-1.txt[179-200]

next       `spool match` places them in identities
```

The document's key is its file's name. A file of the same name with other text is the next version of the document, and the same text again adds nothing.

```sh
spool match
```

```sh
spool show records --source notes
```

```text title="Output"
source and key              name                     identity
notes:notes-1.txt[54-69]    Acorn Tools Ltd          01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-1.txt[119-142]  Larkfield Logistics Ltd  01a1efc6-41cd-775d-8758-e733aa086f7d
notes:notes-1.txt[179-200]  Brightwater Foods Ltd    01a1efc6-43a9-7403-b9f6-1f8ee76184ed
```

Two of the three joined identities the CRM rows had made, and Brightwater Foods is new.

## What a record from text keeps

```sh
MENTION=$(spool show records --source notes | awk 'NR==2 {print $1}')
spool show records $MENTION
```

```text title="Output"
record    00308454-974b-52ac-112f-80f483e081fd
type      company
source    notes
key       fb9a1372-e552-f564-d2b5-c2a510bd408a#company:54-69
loaded    2026-11-18T21:54:51Z by run 6
identity  01a1efc6-41cd-775d-8758-e7300637659a

payload
  name  Acorn Tools Ltd

evidence
  document   fb9a1372-e552-f564-d2b5-c2a510bd408a
  end        69
  extractor  companies
  key        notes-1.txt
  start      54
  text       Acorn Tools Ltd
  version    eacb9397cacc366a1cd065d8f328039484a5e5d4fbf49b8ca1f2a3cbd34f8c92

in document notes:notes-1.txt, characters 54 to 69
  Call with Dana Reyes on 12 November. She has moved to [Acorn Tools Ltd] and runs purchasing there. They now ship through Larkfield Logistics Ltd, and s…
```

The evidence names the document, the characters, the extractor and the version of the extractor, which is a hash of its file. The last lines show the place in the text. The studio's Documents screen shows the whole document with every mention marked.

```sh
spool show documents
```

```text title="Output"
document                              source and key     characters  read by
fb9a1372-e552-f564-d2b5-c2a510bd408a  notes:notes-1.txt  222         companies in run 5 (3 mentions)
```

## Your own code, in Python

An extractor of your own is another file in `extractors/`. This one reads a second note:

```text title="notes-2.txt"
Dana called again. Acorn Tools are moving their account to Kestrel Analytics
next quarter, and Larkfield will keep the deliveries.
```

This file has the shape of an extractor that asks a hosted language model. `ask` stands where the model's answer goes: here it gives two names from a fixed list, so that the example runs with no key.

```py title="extractors/names.py"
"""Finds the companies a note names, by asking for them and locating each answer in the text."""

from spool import extractor

KNOWN = ["Acorn Tools", "Kestrel Analytics"]

def ask(text):
    """The names in `text`. A real extractor sends the text to a model here."""
    return [name for name in KNOWN if name in text]

@extractor
def find(document):
    for quote in ask(document.text):
        for start, end in document.find(quote):
            yield document.mention("company", start, end, name=quote)
```

- `document.text` is the text, and `document.key` its key.
- `document.find(quote)` gives each place the quote is in the text. Ask a model for quotes and not for positions: models count characters badly.
- `document.mention(entity_type, start, end, **attributes)` is one mention, with `start` and `end` counted within the whole text.

The project now has two extractors, so the command says which:

```sh
spool extract notes-2.txt --extractor names
```

```text title="Output"
source     notes
read       1 document from notes-2.txt
loaded     1 new, 0 already held, 0 new versions
extractor  names
found      3 records in 2 documents: 3 new, 0 already held

  company  Acorn Tools        notes:notes-2.txt[19-30]
  company  Kestrel Analytics  notes:notes-2.txt[59-76]
  and more: `spool show records --source notes` lists every record of the source

next       `spool match` places them in identities
```

It read both notes, because an extractor reads every document it has not read. In the first note it found "Acorn Tools" inside "Acorn Tools Ltd", which the sample extractor had found too. Those are two records for nearly the same place, from two extractors, and a match puts both in Acorn's identity:

```sh
spool match
```

```sh
spool show records --source notes
```

```text title="Output"
source and key              name                     identity
notes:notes-1.txt[54-65]    Acorn Tools              01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-1.txt[54-69]    Acorn Tools Ltd          01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-1.txt[119-142]  Larkfield Logistics Ltd  01a1efc6-41cd-775d-8758-e733aa086f7d
notes:notes-1.txt[179-200]  Brightwater Foods Ltd    01a1efc6-43a9-7403-b9f6-1f8ee76184ed
notes:notes-2.txt[19-30]    Acorn Tools              01a1efc6-41cd-775d-8758-e7300637659a
notes:notes-2.txt[59-76]    Kestrel Analytics        01a1f09f-0c50-7164-ba08-984961afefc8
```

Kestrel Analytics is new. "Larkfield" alone was not in the list, and a model might or might not have given it.

### With a model

Replace the body of `ask` with a call to a model's API and return the names it gives. The repository has a whole one, [`llm.py`](https://github.com/alexpatow/spool/blob/main/benchmarks/extraction/company-news/extractors/llm.py), which uses only Python's standard library, speaks the chat-completions API, and reads its key and model from the environment. It tells the model what each entity type is from a file of the project, `entities/<type>/extract.md`:

```md title="entities/company/extract.md"
A company is a business the text names: a legal entity such as a corporation,
a limited company or a partnership.

Quote the name exactly as the text writes it, with its legal form if the text
gives one. A product, a person, a place or a public body is not a company.

Attributes:

- `name`: the name as quoted.
```

An extractor that takes `(documents, task)` in place of `(document)` gets every document at once and reads those instructions with `task.instructions("company")`.

No test in the repository runs `llm.py` against a real model, because a test that needs a key and answers differently each time is not a test. An extractor that calls a model costs what the model costs, for every document, and the server stops one that runs longer than `--task-timeout`, which is 600 seconds unless set.

## In TypeScript

The same extractor as a `.ts` file, which the server runs with [Bun](https://bun.sh):

```ts title="extractors/names_ts.ts"
import { extractor } from "spool-strategy";

const KNOWN = ["Acorn Tools", "Kestrel Analytics"];

await extractor(async (task) => {
  const documents = await task.documents();
  return documents.flatMap((document) =>
    KNOWN.flatMap((name) =>
      document.find(name).map(([start, end]) => document.mention("company", start, end, { name })),
    ),
  );
});
```

`spool dev` picks up a new extractor file as it is saved, and `spool info` lists what the project has:

```sh
spool info
```

```text title="Output"
project   .
manifest  9e083661f159998deacca01381d4a683c2eb295b258f0ad623ffad75159af376

entity type `company`
  features  record_id UUID, name VARCHAR, name_key VARCHAR, country VARCHAR
  label     name
  sources   crm, notes
  block     name_prefix
  no_match  different_country
  match     similar_name
  extract   extract.md says what an extractor looks for

extractor `companies` (python, extractors/companies.py)
extractor `names` (python, extractors/names.py)
extractor `names_ts` (typescript, extractors/names_ts.ts)
```

## Over HTTP and MCP

| | Store a document | Run an extractor |
| --- | --- | --- |
| Command | `spool extract <file>` does both | `spool extract` with no file |
| HTTP | `POST /v1/documents` | `POST /v1/extractions` |
| MCP | `load_documents` | `run_extraction` |

[Getting started](https://spool.alexpatow.com/docs/getting-started.md#extract-records-from-text) has a call of each. `POST /v1/extractions` takes `extractor`, `source` to keep to the documents of one source, `again` to read every document again, and `limit`.

## How good is it

An extractor is scored apart from matching. `spool bench` runs a scenario with labelled mentions and reports what the extractor found, missed and made up. The repository's [`benchmarks/extraction/company-news`](https://github.com/alexpatow/spool/tree/main/benchmarks/extraction/company-news) has ten news items, a rule-based extractor in both languages, and its scores.

It also reports the two together, end to end: the identities against the true entities over every labelled mention, where a mention the extractor missed is with nothing. [`benchmarks/extraction/linked-docred`](https://github.com/alexpatow/spool/tree/main/benchmarks#from-text-to-identities) measures all three on 5,051 Wikipedia articles.

## What extraction does not do

- **Documents are text.** Spool does not read PDF or HTML. Turn them into text first.
- **The sample model finds what its word asks for.** Given "company", it misses most universities, parties and charities.
- **No model is in the binary.** The sample extractor fetches its model on its first run. Every extractor is a Python or TypeScript file, and needs `uv` or `bun` where the server runs.
- **A mention is a place.** Two mentions of one company in one document are two records, in one identity.
- **A changed extractor does not read again by itself.** `--again` has it read every document again.
- **A document is one request.** A file larger than the API's limit of 256 MiB cannot be stored.
