Migrates between embedding model versions using parallel indexes, versioned vectors, controlled backfill, cutover, and rollback without mixed-version retrieval or silent quality regression.
Package status: AI-assisted reference package. Static package validation has passed; implementation scenarios remain not run and human domain review is required before stable adoption.
Decision
Embedding model upgrades run as a dual-index migration rather than an in-place overwrite. The old index continues serving until a new index is backfilled, evaluated, and cut over under explicit gates. Ingest during migration dual-writes or queues re-embedding jobs so new documents are not stranded on the retiring version. Production queries never blend vectors from two versions in one ranked result set. Rollback remains available while the previous index is retained. Quality gates compare offline retrieval metrics before cutover is permitted.
Scope
- Versioned embedding indexes, backfill jobs, and migration state machine.
- Dual-write or equivalent ingest behavior during migration.
- Cutover prerequisites, operator approval, and serving switch.
- Rollback triggers and retained-index policy.
- Observability for backfill progress and mixed-version detection.
Outside this block
- ACL enforcement on retrieval.
- Chunking policy changes concurrent with re-embedding.
- Choice of embedding model vendor.
Contract
- Every vector record carries an embedding version tag; queries in production target exactly one serving version at a time.
- During dual-write state, new ingests populate the target index while user-serving remains on the source index until cutover completes.
- Cutover requires backfill completion and quality-gate pass documented in migration state; partial backfill cutover is rejected.
- Offline evaluation compares retrieval metrics between versions; failing gates block cutover-ready transition.
- Rollback from v2-primary to v1-primary requires the retained v1 index to be intact, operator approval, and a temporary freeze on v2 ingest; serving switches to v1-only without mixed-version queries.
- Mixed-version retrieval in a single user query triggers operational alerts and is treated as a configuration defect.
Implementation guidance
- Maintain migration state in operational metadata consumed by ingest workers and query routers.
- Batch backfill with checkpointing and lag alerts; throttle to protect upstream embedding APIs.
- Run shadow queries against the candidate index during dual-read-eval without exposing blended results to users.
Failure handling and safeguards
- Embedding API failures during backfill retry with idempotent chunk keys. Exhausting the retry budget moves the batch or migration to an explicit failed/incomplete state; it never marks the target version complete, and cutover remains blocked.
- If v2 error rate spikes post-cutover, execute rollback playbook and freeze further ingest to v2 until root cause is identified.
- Decommission v1 only after retain window expires and rollback drills confirm v2 stability.
Verification and operations
- Track backfill percent, cutover events, rollback events, and mixed-version alert count.
- Fixture-test state transitions in
migration-states.yaml. - Verify retry exhaustion produces
backfill-failed, preserves its checkpoint, and cannot satisfy the backfill-complete gate. - Compare recall-at-k or equivalent offline metric before and after cutover for a golden query set.
The executable-looking examples in this package are fixtures and acceptance contracts. Run static validation on the package structure and metadata, then implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - Pinecone and OpenAI embedding docs illustrate migration patterns; no vector database is mandatory.
- Quality gate metrics and thresholds are profile choices tied to product risk.