Entity Resolution Lab

Executive summary

Entity resolution is the problem of deciding when records from different sources refer to the same real-world entity. This lab documents a genericized approach to blocking, scoring, and graph clustering — the kinds of patterns used on knowledge-graph data platforms — without exposing proprietary data, code, or employer-specific outcomes.

TODO: Replace architecture diagrams and any quantitative notes with public-safe materials you own. Do not publish confidential pipeline metrics.

Problem

Multiple systems emit partially overlapping identifiers, attributes, and relationships. Naive joins either miss true matches or over-merge distinct entities. The goal is a repeatable pipeline that:

Reduces pairwise comparison cost via blocking
Scores candidate pairs with interpretable features
Clusters linked pairs into entity components
Writes a durable identity graph for downstream consumers

Context

Knowledge-graph platforms often sit between operational data stores and analytical consumers. Latency, auditability, and lineage matter as much as raw match rate. This write-up stays at the level of patterns and tradeoffs, not a specific deployment.

Role

Design and implementation of resolution stages, evaluation harness ideas, and graph modeling choices — framed as personal technical notes.

Constraints

Scale: pairwise all-to-all is infeasible at interesting volumes
Noise: missing fields, inconsistent formatting, temporal drift
Operability: pipelines must be replayable and observable
Privacy: no real PII or confidential firm data in this public case study

Approach

Normalize inputs into a common record model
Block on high-recall keys (e.g., normalized name tokens + geography proxies)
Score candidates with a small set of features (string similarity, shared identifiers, structural hints)
Cluster above-threshold links into connected components
Materialize entities and provenance edges in a graph store

Sources → Normalize → Block → Score → Cluster → Graph (entities + evidence)
                ↓
           Lineage / run metadata

Architecture notes

Batch orientation: Spark-style distributed stages for heavy joins and scoring
Graph store: Neo4j-style entity and evidence relationships for traversal and investigation
Orchestration: Dagster-style assets for lineage between raw, blocked, scored, and resolved layers

Difficult problems

Threshold choice: Precision/recall tradeoffs are domain-specific; a single global cut rarely survives contact with reality
Transitivity: A–B and B–C links can force A–C merges that are wrong without additional constraints
Evaluation: Ground truth is expensive; weak labels and sampled audits are usually what you actually get

Tradeoffs

Option	Benefit	Cost
Aggressive blocking	Cheap candidates	Missed true pairs
Soft blocking / multi-pass	Higher recall	More compute
Pairwise ML classifier	Flexible scoring	Label demand, drift
Rules + similarities	Debuggable	Harder edge cases

Outcome

TODO: Document only results you can verify and publish. Example placeholder: “Internal experiments improved candidate reduction versus naïve comparison; public metrics withheld.”

This case study intentionally stops short of claiming production KPIs.

What I learned

Entity resolution is as much about evaluation design and operability as about similarity functions
Graph structure helps investigation even when the “best” match score is ambiguous
Genericizing production work for a portfolio requires discipline: patterns yes, secrets no

What I would change

Invest earlier in a reproducible evaluation set
Separate “investigation UI queries” from “batch resolution” contracts more cleanly
Explore incremental re-resolution strategies for high-churn sources