Streaming check_output
The non-streaming check_output waits for the whole reply before it checks it. Streaming checks rolling windows as the LLM emits tokens, so a violation surfaces while the reply is still being written, and your agent can stop generating.
Non-streaming check_output |
Streaming | |
|---|---|---|
| When you hear about a violation | After the full reply is checked | Around 1–2 seconds after the text that contains it is written |
| Calls | One | One per ~200 characters, plus finalize |
| UX during the wait | Spinner | Progressive results, optional cancellation |
| Best for | Async review, audit | Real-time chat, voice agents, long generations |
See Latency for timings.
How it works
Section titled “How it works”You stream output chunks from your LLM into the SDK’s streamCheckOutput. The SDK keeps reading your stream while a check runs (it never pauses your LLM), and calls POST /api/v1/agent/governance/check-output/stream every ~200 chars (configurable), one call at a time, each with the full text so far. Each call:
- Detects on the new segment + a 200-char overlap window (so violations spanning chunk boundaries still get caught).
- Drops violations already reported for this reply, so each violation is reported once per reply, even one found again, longer, after a chunk boundary.
- Returns the new violations + a watermark to pass back next call.
At end-of-stream, the SDK calls POST /api/v1/agent/governance/check-output/finalize for the consolidated verdict (decision, suggestedRedaction, full violations list).
Each stream response also carries decision (allow / warn / block, on the same bars as post-inference) for that segment’s new violations, and checked. A segment that did not run (checked: false, with uncheckedReason) has decision: null and keeps the watermark where it was, so the next call or finalize covers it again. A chunk with fewer than 80 new characters is deferred the same way (no violations, decision: null, watermark unchanged) unless it is marked isFinal.
The SDK exposes the whole flow as an async iterable. You consume events with for await:
| Event | When |
|---|---|
violation |
A new violation, with the decision of the check that found it. |
progress |
A check finished, with its decision. |
unchecked |
A segment was not checked; finalize still covers it. |
complete |
Finalize returned the whole-output verdict (the default). |
final |
With finalize: false only: the closing chunk’s verdict for the whole reply. |
error |
A call failed. recoverable: true means the stream went on. |
An error with recoverable: true comes from a network error, a timeout, a 408, a 429 or a 5xx on an incremental chunk; finalize still checks the whole output. When an incremental chunk is refused with any other 4xx (a 400 bad field, 401 key not accepted, 402 budget used up, 403 check disabled, 404), the SDK emits one error with recoverable: false, status and code, plus budget (capCents, usedCents, periodEnd) on a 402. It then sends no more chunks and no finalize, reads your source to its end, and ends the stream with no complete or final event: treat the reply as not checked.
A 200 that the SDK cannot read (a null body, no numeric watermark, or a newViolations that is not a list) is also an error, with status: 200 and a message that starts malformed streaming check response (a body that is not JSON at all is an error without a status). It is recoverable: true on an incremental chunk, so finalize still checks the whole output, and recoverable: false on the closing isFinal chunk (with finalize: false), so treat the reply as not checked.
An empty or whitespace-only output is not checked, and the stream ends with no event at all, not even complete. Don’t wait for complete to release a reply.
The SDK sends one replyId on every chunk and on finalize (yours if you pass replyId, otherwise a new random UUID per reply), so two replies on the same sessionId are never confused. If you pass your own, use a new one for every reply: a replyId used again is treated as the same reply.
Quickstart — TypeScript SDK
Section titled “Quickstart — TypeScript SDK”import { InPolicyClient } from '@inpolicy/sdk';import OpenAI from 'openai';
const ip = new InPolicyClient({ apiKey: process.env.INPOLICY_API_KEY! });const openai = new OpenAI();
async function* outputChunks(stream: AsyncIterable<{ choices: { delta: { content?: string | null } }[];}>) { for await (const part of stream) { yield part.choices[0]?.delta?.content ?? ''; }}
async function respond(sessionId: string, userMessage: string) { // 1. Pre-inference (fast, ~100 ms) const pre = await ip.recordTurn({ sessionId, turn: { role: 'user', content: userMessage }, });
const completion = await openai.chat.completions.create({ model: 'gpt-4o', stream: true, messages: [ { role: 'system', content: pre.injectionBlock }, { role: 'user', content: userMessage }, ], });
// 2. Drive incremental detection on the streamed output. let unsafe = false; for await (const evt of ip.streamCheckOutput({ sessionId, source: outputChunks(completion as never), })) { switch (evt.type) { case 'violation': console.warn(`[InPolicy] ${evt.violation.matchedText}: ${evt.violation.explanation}`); // evt.decision === 'block': you can cancel generation here. break; case 'progress': // Useful for live UI: "checked X chars so far" break; case 'complete': // A check that did not run follows the key's on_check_error setting. if (!evt.verdict.checked) { console.warn('[InPolicy] not checked:', evt.verdict.uncheckedReason); } unsafe = evt.verdict.decision === 'block'; if (unsafe && evt.verdict.suggestedRedaction) { // Safe to send as is: every finding was rewritten. } else if (unsafe) { // No safe rewrite: regenerate, or route to a person. } break; case 'unchecked': // A segment was not checked; finalize checks it again. break; case 'error': // recoverable: a chunk failed and the stream went on; finalize // re-checks the whole output. Not recoverable: nothing more is // checked and no 'complete' follows. evt.status / evt.code say why. console.error('[InPolicy] streaming error:', evt.message, evt.recoverable); break; } }}When to use streaming vs. non-streaming
Section titled “When to use streaming vs. non-streaming”| Use case | Recommended |
|---|---|
| Real-time customer-service chat (interactive UI) | Streaming |
| Voice agent (sub-second latency required) | Streaming, cancelling on the first violation event whose decision is block |
| Email drafting tool (human reviews before sending) | Non-streaming check_output is fine |
| Compliance audit pipeline (offline) | Non-streaming, queued |
| Code review / copilot suggestions | Non-streaming |
| Long generation (e.g. document drafting) | Streaming: violations surface as the text is written |
Trade-offs
Section titled “Trade-offs”Streaming spends more detection. Each stream call checks its segment plus a 200-character overlap, and finalize checks the whole output once more. A streamed reply still counts as one message checked (see How a streamed reply counts).
Streaming gives you cancellation. If a block decision surfaces mid-stream, your agent can stop generation, regenerate or return suggestedRedaction from finalize (when it is not null), and save the rest of the LLM cost. Non-streaming can’t do this. By the time you see the violation, the LLM has already finished.
Streaming has a watermark contract. Pass back the watermark from the previous call as lastCheckedOffset on the next. The SDK does this automatically; if you call streamCheckOutputOnce directly, you manage it yourself.
Disable the final complete event
Section titled “Disable the final complete event”If you only care about per-chunk violation events and want to skip the end-of-stream finalize call:
for await (const evt of ip.streamCheckOutput({ sessionId, source: outputChunks(...), finalize: false, // ← skips /check-output/finalize})) { ... }The SDK then closes the stream with one last chunk marked isFinal, which checks the tail, and ends with a final event: { type: 'final', decision, checked, uncheckedReason, riskScore, watermark, traceId }. Its decision is for the whole reply, the worst across all of its chunks, the same decision the reply’s usage and findings record, and its riskScore is the reply’s highest, so you don’t need to keep a running maximum. The SDK takes the worst of every chunk result it received and the server’s closing answer, so a chunk that blocked is never lost. A retried closing chunk returns the same decision. If the closing chunk was not checked, checked is false and decision is null, unless an earlier chunk already blocked (then block): the stream route does not apply on_check_error, so treat a null decision the way your on_check_error says (with block, hold the reply). That chunk counts the reply as a message. If the closing chunk fails, there is no final event: the stream ends with an error whose recoverable is false, so treat the reply as not checked. Cost: you don’t get suggestedRedaction or policyCitations.
Direct API access (without the SDK helper)
Section titled “Direct API access (without the SDK helper)”If you need lower-level control:
POST /api/v1/agent/governance/check-output/streamAuthorization: Bearer inp_live_…Content-Type: application/json
{ "sessionId": "sess_abc", "fullText": "Sure, here is the refund for John Smith (john@acme.com). It should reach your card within five working days.", "lastCheckedOffset": 0, "isFinal": false, "replyId": "msg_8812"}Response shape:
{ "sessionId": "sess_abc", "newViolations": [ { "policyId": "policy_pii", "matchedText": "John Smith (john@acme.com)", "explanation": "Names the customer and gives their email address in a support reply.", "severity": 4.5, "confidence": 0.92, "enforcementType": "fix", "enforcementText": "the account holder", "offset": 29 } ], "watermark": 108, "isComplete": false, "riskScore": 0.92, "decision": "block", "checked": true, "uncheckedReason": null, "traceId": "trc_…"}Pass watermark (108 above) as lastCheckedOffset on the next call. On /check-output/stream, a lastCheckedOffset greater than the length of fullText gets a 400 and nothing is checked or charged. finalize checks the whole text and does not use the offset, so it never gets that 400.
The server trusts lastCheckedOffset only as far as the reply’s record shows was checked. An offset past the highest watermark the reply’s checked chunks returned is checked from that watermark. A reply with nothing on record, or whose text before that watermark is not the text that was checked, is checked from 0: for example a second reply in a session without a replyId that opens at an offset above 0. Sending back the watermark you were given is never affected.
An incremental chunk (isFinal: false) with fewer than 80 new characters since lastCheckedOffset is deferred: it returns newViolations: [], decision: null and the same watermark you sent, and the next call (or the isFinal chunk, or finalize) checks that text. That is why the example sends a whole sentence.
offset and watermark count Unicode code points. JavaScript strings index UTF-16 code units, so after an emoji or another character outside the Basic Multilingual Plane the two differ: convert before you slice or highlight. The SDK converts both ways for you.
enforcementType: "fix" means enforcementText is a rewrite of matchedText; "warning" means it is advice only. Those are the only two values.
replyId is optional (letters, digits and . _ : @ + = -, 1 to 200 characters): send the same one on every chunk of a reply and on its finalize, and a new one for every reply. A replyId used again is the same reply: it is counted once and keeps what the earlier reply found, its decision included. With it, re-sending a chunk at lastCheckedOffset: 0 keeps what the reply already found. Without it, start every reply at lastCheckedOffset: 0: a chunk at offset 0 starts a new reply.
A chunk re-sent after it was checked (a retry after a timeout, for example) does not report that chunk’s violations again. The isFinal chunk’s decision still counts them, and finalize returns every violation in the reply. Two copies of the same phrase found by one check are both reported, each with its own offset.
When the LLM finishes, close the reply in one of two ways:
POST /api/v1/agent/governance/check-output/finalizewith the full text, for the consolidated verdict (recommended), or- a last
/check-output/streamcall withisFinal: true, which checks the tail (even when nothing new is left). Itsdecisionis for the whole reply, the worst of every chunk, and itsriskScoreis the reply’s highest;newViolationslists only the tail’s new ones. If that chunk was not checked,decisionisnull(apply youron_check_error) unless an earlier chunk blocked, in which case it isblock.
Don’t do both: it checks the tail twice. The reply still counts once if you do, as long as both carry the same replyId (or neither does).
How a streamed reply counts
Section titled “How a streamed reply counts”A streamed reply is one message checked, counted at finalize or at the isFinal chunk, whichever comes first for the same reply. A reply is its replyId: two replies in one session with the same text but different replyIds are two messages, and a retried isFinal or finalize of the same replyId counts once. An isFinal chunk and a finalize of the same reply count as two messages when only one of them carries the replyId. Without a replyId, the same fullText in the same session within an hour counts once. Other chunks record spend against the workspace’s budget and count no message. A reply’s findings (policy, severity, decision; never text) are recorded once, by the call that counted it, and its usage row records the whole reply’s length and every violation its chunks found. A new reply starts at lastCheckedOffset: 0, with a new replyId if you send one; the same sessionId can carry many replies.
Latency expectations
Section titled “Latency expectations”Each /check-output/stream call and the end-of-stream /finalize take around 1–2 seconds; see Latency for the caps. The first violation can surface once you cross chunkSize of buffered text (200 characters by default).
Auth + gating
Section titled “Auth + gating”Same as non-streaming check_output: needs the governance:check scope and post_inference_checking: "allowed" on the API key (on by default for new keys). Streaming endpoints return 403 otherwise. fullText may be at most 100,000 characters.
Cookie PreferencesDo Not Sell or Share My Personal Information