Duplicate documents: Pipeline implementation and metadata

Defines duplicate detection and retrieval grouping while preserving each source document and version independently, preventing shared storage identities and invisible cross-project deletion dependencies.

Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.

Decision

Keep source records and content objects independent. Compute exact and near-duplicate signals into a versioned relationship graph, then choose a query-time representative only within the requester's authorized source set.

Scope

• Canonical text digest, similarity candidates, duplicate relationship, cluster version, representative policy, provenance, and reprocessing.
• Exact copies and high-similarity documents across independent sources.
• Retrieval grouping without physical object de-duplication.

Outside this block

• Deleting one stored copy because another looks equivalent.
• Declaring contradictory versions duplicates solely because their text overlaps.

Contract

• Each source document/version has its own storage key, lifecycle, permissions, and deletion record.
• Exact digest is computed over a documented canonicalization; the unmodified source digest is retained separately.
• Near-duplicate decisions store method, profile version, score, evidence spans, creation time, and review state.
• Clusters are scoped to an authorized comparison domain and cannot reveal inaccessible members or counts.
• Representative selection considers authority, freshness, completeness, and request visibility; it is not a permanent canonical source.
• A relationship change never mutates or deletes the underlying source record.

Implementation guidance

• Normalize transport artifacts such as repeated whitespace separately from semantic text.
• Use inexpensive fingerprints for candidate generation and a reviewed threshold or model for decisions.
• Store duplicate edges and derive clusters so merges and splits remain auditable.
• At retrieval, group authorized hits and retain alternate citations when their source identity matters.

Failure handling and safeguards

• If a cluster spans authorization boundaries, compute the visible representative from allowed members only.
• If a source is deleted, tombstone its edges and re-elect representatives without touching other objects.
• If detection profile changes materially, write a new relationship generation and compare before activation.

Verification and operations

• Measure exact/near-duplicate prevalence, reviewed precision/recall, cluster-size distribution, representative churn, and retrieval repetition.
• Sample cross-boundary clusters for leakage tests using synthetic tenants.
• Track reprocessing backlog and orphaned edges.

The executable-looking examples in this package are fixtures and acceptance contracts. Run python tools/validate.py from the collection root to check package structure and metadata; then implement and execute the scenarios in the target repository.

Adoption assumptions

• Names and numeric values in example.yaml are an adoption profile, not universal defaults.
• The adopting team must map actors, data classes, error vocabulary, and ownership to its system.

References

• No external normative source is required; this package defines a project decision.