Chunking: Pipeline implementation and metadata

Defines structure-aware document chunks that retain heading-path metadata, overlap as a profile value, never drop ACL on split, and evaluate recall at k against a labeled retrieval set.

Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.

Decision

Ingest documents into structure-aware chunks: split on headings and semantic boundaries first, then enforce a token budget with a declared overlap. Every chunk stores the heading path from document root so retrieval and citations can name the section, not only a floating paragraph. Access-control labels copy from the parent document onto every fragment; a split must never produce an unlabeled or broadened chunk. Retrieval quality is accepted only when recall at k on a labeled question set meets the profile threshold.

Scope

• Document ingestion, heading-aware split, overlap, token budget, heading-path metadata, ACL propagation, and offline recall evaluation.
• Backend indexing pipeline for knowledge-base corpora.
• Chunk identity stability across re-ingest of unchanged bytes.

Outside this block

• How answers cite chunks after retrieval.
• Embedding vendor choice, hybrid lexical fusion, or production ranking UI.

Contract

• Each chunk records heading_path as an ordered list of section titles from root to the containing heading.
• overlap_tokens is taken from the adoption profile and is applied between adjacent chunks of the same document version.
• No chunk is written without the parent document ACL; child ACL is a subset, never a superset.
• Chunks exceeding token_budget are split further; chunks below min_tokens are merged with a sibling when headings allow.
• Re-ingesting identical document bytes and the same profile version yields the same chunk boundaries and identifiers.
• recall_at_k on the labeled evaluation set is computed and must meet min_recall_at_k before a corpus version is marked searchable.

Implementation

• Parse headings from the source format rather than sliding character windows through tables and code fences.
• Count tokens with the same tokenizer the embedder uses; freeze a labeled question set and gate ingest on recall@k.

Failure handling

• If heading parse fails, fall back to paragraph splits with heading_path [document_title] and raise a quality warning.
• If ACL is missing on the parent, abort ingest rather than indexing public-by-default chunks.

Verification

• Fixture a nested policy document with three heading levels and assert heading_path on each chunk.
• Compare recall@k before and after an overlap change using the same labeled set.

The executable-looking examples in this package are fixtures and acceptance contracts. Run python tools/validate.py from the collection root to check package structure and metadata; then implement and execute the scenarios in the target repository.

Adoption assumptions

• Names and numeric values in example.yaml are an adoption profile, not universal defaults.
• The adopting team must map actors, data classes, error vocabulary, and ownership to its system.

References

• No external normative source is required; this package defines a project decision for structure-aware ingestion.