Streaming responses: Requirements and policy

Stop aborts an in-flight generation stream, persists partial assistant text, keeps the user message, allows regenerate as a new attempt, and treats transport disconnect as distinct from an explicit user stop.

Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.

Decision

Stop aborts the active stream with the transport abort signal. The user message stays a committed turn. Received assistant tokens persist as a partial tagged stopped. Regenerating starts a new attempt_id; it does not rewrite history as if stop never happened. A dropped socket, idle timeout, or HTTP/2 reset is disconnect or timeout, not user stop. Those events may offer resume, but they must not use Stop copy. See sequence.md.

Scope

• LLM chat and other streaming APIs that emit incremental text.
• Abort signaling, partial persist, and regenerate-as-new-attempt.
• Records that distinguish stop, disconnect, and timeout.
• Tools already committed before stop; this block only records completion.

Outside this block

• Whether a tool may mutate production data.
• Prompt-injection defenses for retrieved text.
• Token billing beyond attempt identity.

Contract

• Activating Stop aborts the in-flight request so no further tokens append after acknowledgement.
• The user turn is retained; the assistant turn is stored with status stopped and the exact partial text.
• Regenerate creates a new attempt_id linked to the same user turn and does not delete the stopped partial unless the user discards it.
• A transport disconnect sets disconnected and must not fire Stop analytics or Stop confirmation copy.
• Idle with no tokens for idle_ms sets timed_out; reconnect may resume only with a valid stream cursor.
• Stop does not roll back a tool side effect that already committed.

Implementation

Wire Stop to AbortSignal and send a server cancel when the protocol allows. Persist on each chunk or on abort, whichever is earlier. Keep reconnect on a separate path from Stop. Label Stop generating versus Retry connection. Log attempt ids without prompt secrets.

Failure handling

If abort never reaches the server, freeze the UI and mark the local partial stopped; drop late chunks. If persist fails, keep the partial in session storage until retry. If regenerate races an open stream, refuse until abort completes. Extra tokens after cancel increment a cancel-leak metric.

Verification

Measure stop-to-last-token latency, cancel leaks, persist failures, and misclassified disconnects. Walk the three sequences in sequence.md. Announce generation stopped distinctly from a network error. Execute the packaged scenarios.

The executable-looking examples in this package are fixtures and acceptance contracts. Run python tools/validate.py from the collection root to check package structure and metadata; then implement and execute the scenarios in the target repository.

Adoption assumptions

• Names and numeric values in example.yaml are an adoption profile, not universal defaults.
• The adopting team must map actors, data classes, error vocabulary, and ownership to its system.

References

• fetch-aborting