Duplicate documents: Evaluation and operations

Defines labeled evaluation, threshold gates, drift checks, cluster health, and safe operational repair for duplicate-document grouping in a knowledge retrieval system.

Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.

Decision

Evaluate relationship and retrieval effects separately: pairwise labels test duplicate classification, while query judgments test whether grouping removes repetition without hiding useful provenance or conflicting evidence.

Scope

• Judgment sampling, exact and near-duplicate metrics, cluster metrics, retrieval impact, drift, review queues, and rollback.
• Evaluation by language, source type, document length, and authorization pattern.
• Operational repair of edges and cluster generations.

Outside this block

• The detection pipeline implementation.
• Using unlabeled similarity score distributions as proof of quality.

Contract

• The labeled set distinguishes exact copy, near duplicate, related, superseded, and conflicting source.
• Pair selection includes hard negatives with shared boilerplate and hard positives with formatting or extraction changes.
• Threshold gates are defined per risk segment where errors have different cost.
• Cluster evaluation catches transitive over-merges that pairwise precision can miss.
• Retrieval tests measure repeated passages, evidence coverage, source diversity, and unauthorized membership disclosure.
• A new detector or threshold activates as a reversible relationship generation.

Implementation guidance

• Sample from score bands, large clusters, cross-source edges, changed profiles, and user-reported repetitions.
• Keep adjudication instructions and disagreement; do not force ambiguous cases into duplicate/non-duplicate only.
• Use connected-component diagnostics for suspicious bridges and representative dominance.
• Replay fixed search fixtures before and after a generation change.

Failure handling and safeguards

• If precision falls below the protected threshold, stop automatic grouping and retain edges as suggestions for review.
• If a bad bridge over-merges a cluster, remove or supersede that edge and rebuild derived membership.
• If evaluation coverage is missing for a language or source, do not generalize a global threshold silently.

Verification and operations

• Track pairwise precision/recall, cluster purity, bridge rate, largest clusters, review backlog, representative churn, repetition@k, and evidence loss.
• Alert on sudden cluster growth or one source becoming representative disproportionately.
• Review sampled false merges because they can suppress distinct evidence.

The executable-looking examples in this package are fixtures and acceptance contracts. Run python tools/validate.py from the collection root to check package structure and metadata; then implement and execute the scenarios in the target repository.

Adoption assumptions

• Names and numeric values in example.yaml are an adoption profile, not universal defaults.
• The adopting team must map actors, data classes, error vocabulary, and ownership to its system.

References

• No external normative source is required; this package defines a project decision.