Explained
What the hell is Entity Extraction and Resolution?
Your data names the same companies in different ways. Spool finds them in text, works out which records are the same company, and gives each one an ID that stays. Scroll, and watch it happen to the sample data in Getting started.
01Messy input
Start with what you have. A CRM exports rows, and each row names a company. Spool keeps every row exactly as it arrived, and calls it a record.
A second export arrives. Somebody typed “Acorn Tool”. Somebody else typed “Larkfeild”.
Some of what you know is not in rows at all. It is a note from a call, in sentences.
Acorn Tools turns up five times, spelt four ways, in two kinds of data. Is that one company or five? And is Halden Group in GB the same as Halden Group in the US?
02Extraction
Text names things too, just not in columns. Here is that note.
An extractor reads it and marks each place that names a company. Each place is a mention: where it starts, where it ends, and the name.
Each mention becomes a record, like a row. Its key is where it was found: characters 54 to 69 of notes-1.txt.
Dana Reyes is a person, and this project only knows companies, so she stays in the text. An extractor is code of yours in Python or TypeScript. The one a new project has asks a small model on your machine.
03Resolution
Now there are twelve records: nine rows and three mentions. Which of them are the same company? Answering that is entity resolution.
Spool compares pairs of records with rules you write in SQL. Two names that are close enough are a match. Each answer is kept as a judgement.
The two Halden Groups have the same name, so the match rule holds. But a no_match rule holds too, because their countries differ, and no_match always wins. A wrong match, turned down.
The judgements become identities: one for each real company, each with an ID, and every record placed in one. Twelve records, six companies.
04Persistent identity
Other systems store that ID, so it has to stay put. After the first file, Acorn Tools is three records and one ID.
A new file brings “Acorn Tool”. It joins. The ID does not change.
The mention from the note joins too. Still the same ID. A match places new records, and never moves the ones it placed before.
Then another file brings “Acorn Tooling”. The name is close enough for the rule, so it joins. But it is a different company.
A person corrects it with a split. That is a changeset, kept in a log with an author and a reason, and it can be undone. Acorn Tooling gets an ID of its own. Acorn Tools keeps the one it had all along.
Now run it
Getting started does all of this on your machine, with the same files, in about five minutes.
- Record
- One thing a source said: a row, or a mention found in text. Kept as it arrived.
- Mention
- A place in a document that names a thing, found by an extractor.
- Judgement
- The answer for one pair of records: match or no_match.
- Identity
- One real-world thing: an ID, and the records placed in it.
- Changeset
- Every change to identities, in a log, with an author, a reason and an undo.