Extract, resolve and track entities.

Spool pulls records out of text and rows, decides which are the same real-world thing, and gives each thing an ID that stays the same as more arrive.

Rowssamples/crm-1.csv, crm-2.csv
c-1,Acorn Tools,GB
c-2,ACORN TOOLS,GB
c-3,Acorn Tools Ltd,GB
c-4,Larkfield Logistics,GB
c-5,Halden Group,GB
c-6,Halden Group,US
c-7,Larkfeild Logistics,GB
c-8,Kestrel Analytics,US
c-9,Acorn Tool,GB
Textsamples/notes-1.txt

Call with Dana Reyes on 12 November. She has moved to Acorn Tools Ltd and runs purchasing there. They now ship through Larkfield Logistics Ltd, and she asked whether we work with Brightwater Foods Ltd, their new supplier.

Recordskept as they arrived12
Source and keyName
crm:c-1Acorn Tools
crm:c-2ACORN TOOLS
crm:c-3Acorn Tools Ltd
crm:c-4Larkfield Logistics
crm:c-5Halden Group
crm:c-6Halden Group
crm:c-7Larkfeild Logistics
crm:c-8Kestrel Analytics
crm:c-9Acorn Tool
notes:notes-1.txt[54-69]Acorn Tools Ltd
notes:notes-1.txt[119-142]Larkfield Logistics Ltd
notes:notes-1.txt[179-200]Brightwater Foods Ltd

9 from rows, 3 from the note. A record from text is keyed by where it was found.

Identitiesone ID each6
Changeset logappend-only
  1. match after crm-1.csv4 identities created, 6 records placed
  2. match after crm-2.csv1 identity created, 3 records placed
  3. match after notes-1.txt1 identity created, 3 records placed
  4. next

The sample project at the end of Getting started: nine rows from a CRM and one call note. "Acorn Tools Ltd" in the note joined the identity that four CRM rows already had, and its ID did not change.

Run it

Install it, make the sample project and start the server on your machine.

curl -fsSL https://github.com/alexpatow/spool/releases/latest/download/install.sh | sh
spool init companies
cd companies
spool dev

Linux or macOS on Apple silicon. Version 0.1.0, MIT licence.

Then, in a second terminal

Load the rows, match them, and get the companies out of the note. These seven commands give the picture above.

spool load samples/crm-1.csv
spool match
spool load samples/crm-2.csv
spool match
spool extract samples/notes-1.txt
spool match
spool show identities

Getting started has what each one prints, and the same steps over HTTP and MCP.

What it does

  • Takes rows from several sources and keeps each one as it arrived.
  • Finds records in text with a pattern, or with code of yours in Python or TypeScript, and keeps where in the text each one was found.
  • Matches records with rules you write in SQL, kept as files in Git.
  • Gives each real-world thing an ID that later runs and better rules do not change.
  • Records every merge, split and correction in a log, with an author, a reason and an undo.
  • Says why two records were matched, and leaves what it cannot decide for a person or an agent.

What it is not meant for

  • Deduplicating one file once. A script or a library is less to run.
  • Answering per record, in the request, whether a row is a duplicate. Matching runs in batches.
  • Matching with no rules of your own. There is no trained model inside.
  • Building a merged golden record, or writing corrections back to a source.
  • Many tenants, many writers or many machines. One server owns one store.

Limits has the full list, with what it does not do yet.