Web Directions Year round learning for product, design and engineering professionals

Why Most AI De-Identification Fails in Production, And How We Built One Lawyers Actually Trust

De-identifying text sounds simple when you're sitting in a demo. You've got a paragraph with personal information in it. You replace names with "[NAME]", phone numbers with "[PHONE]", dates with "[DATE]". The text is de-identified. Success. Show it to a lawyer and watch their reaction change. They want reversibility. They want stability. They want to know that if they edit the document, the de-identification doesn't break. They want guarantees about edge cases, about collision avoidance, about what happens when the same person appears multiple times with slightly different name variations.

Suddenly the problem is no longer simple. It's a production system operating under legal constraints where silent errors are unacceptable.

Why De-Identification Is Deceptively Hard

The gap between a demo and production is where most de-identification systems fail. In a demo, you can get away with aggressive masking. Replace everything that might be sensitive. The output is obviously de-identified. But in real legal workflows, that aggressive masking destroys the document's utility.

Consider a contract where you need to remove identifiable information but preserve enough context for downstream systems to understand the agreement. If you've masked all the dates, a language model can't figure out when something is effective. If you've masked all the amounts, it can't understand the financial terms. The text has been de-identified right into uselessness.

This is where the clever thinking starts. The problem isn't just masking sensitive data. It's preserving document semantics while removing individual identity. Those are in tension. One pulls toward removing everything. The other pulls toward keeping enough to be useful.

The reversibility requirement is another dimension. In legal work, you often need to go back from the de-identified version to the original to verify something or reconstruct the full context. So de-identification can't be a one-way hash. It needs to be reversible while still being de-identified. This is a hard cryptographic problem pretending to be a text processing problem.

Building SmartScrub

Moin Zaman and the team at Smartnote built SmartScrub as a response to these constraints. It's not a simple find-and-replace system. It's a reversible de-identification system designed specifically for legal workflows where accuracy and trust matter.

The architecture hinges on several key insights. First, placeholder token design matters enormously. If you just use "[NAME]", you lose all information about what kind of name it is (first, last, company, etc.). More importantly, you can't distinguish between different entities. If the same person appears multiple times, how do you track them? Simple masking creates ambiguity that downstream systems have to guess about.

SmartScrub uses tokens that preserve semantic information while removing identity. Each token contains structured information about what was masked—not in a way that's human-readable, but in a way that machines can work with. An LLM processing the de-identified document can understand that multiple references to "[ENTITY_PERSON_123]" refer to the same individual, even though it doesn't know who they are.

Collision avoidance is another critical problem. If you're generating tokens, you need to ensure that no two different people ever get the same token, and that the same person always gets the same token even if their name appears slightly differently (John vs Jon, Inc. vs Incorporated). This requires careful design and tracking.

The stability-across-edits problem is more subtle. A lawyer receives a de-identified document and wants to make edits. If the editing process breaks the de-identification (by creating new tokens, by regenerating token assignments, by removing the mapping information), the system fails. SmartScrub needs to maintain consistency across edits without requiring the editor to understand the underlying token system.

Building Under Legal Risk

What ties this all together is the context of legal risk and zero tolerance for errors. This isn't a system where approximate solutions are acceptable. A de-identification error in a legal document isn't a small mistake. It's potentially a breach of confidentiality, a violation of regulations, a failure of fiduciary duty.

That context shapes everything about the design. It shapes the decision to make the system reversible—because in legal work, being able to verify that you de-identified correctly is crucial. It shapes the decision to be conservative about what gets masked—better to preserve utility and accept minor risks than to break the document's usefulness in pursuit of perfect anonymity. It shapes the testing and validation process, which needs to cover not just happy paths but edge cases that lawyers will encounter in real contracts.

Moin Zaman brings experience at the intersection of product strategy, UX, and AI-enabled systems designed for high-trust environments. He's built systems that work because they're designed for the constraints of the real world, not an idealized version of it.

The session will be a deep technical case study: what it actually takes to ship de-identification that lawyers will trust, how to design for reversibility and stability, how to think about the tradeoffs between privacy and utility, and why the easy solutions fail in production.

Moin will be sharing this experience at AI Engineer Melbourne 2026, June 3-4.