Submit and recover Agents over authenticated HTTP.

Bind verified identity and resource policy

A mandatory AgentHttpAuthenticator validates signature or introspection evidence, issuer, audience, expiry and revocation. It maps a trusted tenant/principal and independent Submit, Read and Cancel grants. There is no default verifier or anonymous mode.

The existing AgentServiceAuthorizer still authorizes the exact Agent/request, Run or key digest before resource lookup. Never trust identity fields in JSON, an unverified JWT or arbitrary forwarding headers. Client budgets can only narrow host policy.

use std::sync::Arc;
use stateknot::agent_http::{AgentHttpOptions, AgentHttpService};

// service: AgentServiceV1 with exact deployments and resource policy.
// verifier: your real AgentHttpAuthenticator, not a test token check.
let options = AgentHttpOptions::new(["agents.example.com".to_owned()])?;
let http = AgentHttpService::new(service, Arc::new(verifier), options);
let router = http.router();
// Mount behind TLS. Run Workers separately. Own and drain the listener.

Connect a real identity provider with online verification

AgentHttpIntrospection connects to a fixed HTTPS OAuth 2.0 introspection endpoint and strictly verifies issuer, audience, subject, lifetime and operation scopes. Real Keycloak 26.7.3 TLS, secret rotation and revocation are qualified. No redirects, environment proxies, retries, token cache or insecure TLS switch. This is not JWT/JWKS local validation or complete OIDC.

TenantPolicy accepts exact principal-to-tenant bindings from the trusted control plane, intersecting token scopes with local grants. At most 1024 bindings have an explicit lease of at most one hour; replace atomically checks the expected generation. Expired policy fails closed. A tenant binding is not Run ownership: the existing AgentServiceAuthorizer must still authorize each resource.

Install secret-manager-delivered credentials with replace_client_secret. Readiness sends a random invalid token using real client authentication and requires an inactive response. It proves endpoint/credential access, not business authorization or resource policy health. Rotation without overlapping credentials creates a brief 503 window; never fall back to anonymous access.

Defaults are a 3-second total exchange, 32 concurrent requests and a 64 KiB response. Every HTTP request and SSE authorization cycle verifies online; plan IdP capacity accordingly. Buffered bytes cannot be revoked. Test identity services and accounts are never deployed with the website.

Complete wiring, rotation and recovery guide · RFC-0011 (Draft)

Verify access tokens locally with JWT/JWKS

AgentHttpJwtJwks verifies RS256 at+jwt access tokens using public keys delivered by your trusted control plane. Configure JwtJwksOptions with an exact issuer, resource audience and operation scopes. Signed identity still requires TenantPolicy and the independent AgentServiceAuthorizer. This profile is available in the current source checkout.

Provision at most 16 public keys and 16 KiB, with an explicit lease of at most one hour. replace_jwks validates the whole set and checks the expected generation before publication. Refresh from a newly verified source; key expiry returns 503. Removed keys fail subsequent verification and close their SSE streams. JWT lifetime defaults to 15 minutes, with no expiry-extending clock skew.

Local verification avoids a per-request IdP exchange. Your deployment owns authenticated key distribution and timely refresh. Use online introspection when per-token provider revocation must be observed immediately. Readiness checks local key and policy freshness; compose it with actual resource and database dependencies. PostgreSQL 16/17 tests cover tenant and resource refusal, rotation, SSE closure, expiry and fresh-verifier admission recovery.

Local JWT wiring and key-rotation guide · RFC-0018 (Accepted)

Run execution separately

The owned durable scheduling Worker now manages concrete tenant/fair schedulers, real dependency readiness, fixed concurrency and joined shutdown. Ingress remains independent; submitting a Run does not start a Worker or maintenance jobs.

Run the owned maintenance role separately for deadline cancellation, child settlement, Join publication and failure closure, including when no execution Run is runnable.

Inspect the host through separately authorized operations

AgentHostOperations::start owns a separate loopback listener over the actual host health view. Every request requires verified identity, InspectHost and a fresh AgentHostOperationsPolicy. Business Read permission is insufficient. Introspection requires explicit with_host_inspection_scope, the matching token scope and trusted tenant grant.

Only GET /v1/host/status, /v1/host/live and /v1/host/ready exist. Authorized status remains available during business outages and after host shutdown; unavailable identity or operator policy fails closed. No anonymous probes, browser Origin, request bodies or administrative writes.

At most 128 exact operators have a lease up to one hour, with generation-checked replacement and no automatic renewal. Defaults: 16 connections/requests, 5-second requests/drain, 3-second headers and 60-second connection lifetime. Snapshots contain only fixed labels and counters, never tenant/Run IDs or credentials; cumulative counters are decimal strings. Reads do not probe the database.

Stop and join the business host first, inspect its final state, then await operations.shutdown().await. A cancelled wait retains ownership; forced drain joins connections. PostgreSQL 16/17 and real TLS identity tests cover dedicated ops scope, expiry, revocation, limits and cleanup. These are qualification bounds, not production capacity or recovery SLOs. The website deployment remains static documentation only. Wiring, wire format and operations guide · RFC-0016 (Draft)

Own the three roles as one co-located host

AgentHostBindings::new consumes the actual ingress, Worker and maintenance bindings. AgentHostDependencies requires their real identity, policy and execution checks. AgentHost::launch claims shared ingress exclusively and returns an owner before asynchronous startup; host.wait_ready().await waits without losing ownership on cancellation. The application must qualify compatible databases, registries and tenant scope.

Startup is maintenance → Worker → HTTP. Queued durable work may execute during startup; later failure joins started roles but does not roll back committed work. Every hosted HTTP request checks actual sibling freshness, and Worker admission pauses while maintenance is unavailable. Already admitted work may complete. Do not mount input router clones on a separate listener.

host.health() is an internal payload-free view, not an anonymous operations endpoint. A role's unexpected exit closes ingress and drains siblings. host.shutdown().await joins in HTTP → Worker → maintenance order, preserving maintenance during Worker cleanup. Drop initiates cleanup but does not synchronously join it. The OS deadline must cover all role drain bounds plus cooperative destruction; process shutdown is not durable user cancellation.

PostgreSQL 16/17 and real TLS Keycloak exercise composed recovery, dependency gates, startup failures, role exits and joined cleanup. Protected operations are independently authorized above; measured deployment/capacity SLOs remain separate work. Only the documentation website is deployed here, not a fixture-backed Agent runtime. Wiring and recovery guide · RFC-0015 (Draft)

Real resource policy: exact grants and recoverable submission

runtime::agent_policy::AgentResourcePolicy provides offline default-deny authorization. Submission rules match tenant, issuer/subject, Agent revision and input schema, with explicit scopes and restrictive budgets. Run reads, cancellation and key lookup are separate permissions. Knowing a Run ID is not permission; TenantRuns is an explicitly privileged operator grant, never inferred ownership.

Build a PolicyArtifact from a PolicyDocument, retaining its canonical bytes privately with a trusted digest before activation. Limits are 1024 total rules and 256 KiB JSON; duplicate or overlapping selectors fail closed. Call register_agent_policy_evidence_schema before freezing the registry, then pass the shared policy to AgentServiceV1 and compose resource and identity readiness.

Absolute expiration survives restart, and the local freshness lease is at most one hour. replace uses expected-generation CAS. Expiration returns 503; no matching rule returns 403. SSE closes on subsequent authorization after revocation. Admission retains selected-rule, request and budget evidence; unrelated ACL changes preserve same-key recovery. If changing the applicable submission rule produces 409, recover through a currently authorized key lookup instead of automatically resubmitting under a new key.

Reads and denials are not automatically durably audited; cancellation retains policy and decision hashes only. Hosts must retain protected audit context as required. Real PostgreSQL 16/17 and Keycloak tests use this implementation; automatic ownership storage and full production-release acceptance remain separate. Configuration, wiring and operations guide · RFC-0012 (Draft)

Four operations, one durable service

POST /v1/agent-runs

Serialize AgentHttpSubmission: submission_key, exact versioned agent identity, and schema-bound request. 201 creates a Run; 200 recovers it.

GET /v1/agent-runs/{run_id}

No request body. 200 returns an integrity-verified public snapshot, not private checkpoints or transcript.

POST /v1/agent-runs/lookup

Serialize AgentHttpLookup with submission_key. Keep the key out of the URL and access logs.

POST /v1/agent-runs/{run_id}/cancellation

Serialize caller-retained AgentCancellationIds with event_id and failure_id. 202 records the request; 200 recovers the exact pair.

Use a Bearer credential and JSON POST bodies. Accept must be absent, application/json or */*. Rust wire types expose JSON Schema; unknown fields, duplicate keys (including nested keys), query strings and noncanonical paths are rejected.

Success returns AgentHttpRunResponse { request_id, snapshot }. A fresh request_id correlates this response; it is not an idempotency key. Active does not prove a Worker is executing and may include failure closure. CancellationRequested is not terminal cleanup.

Replay activity after a disconnect

Enable with_sse(AgentHttpSseOptions::default()) on the same authenticated host. Connect to GET /v1/agent-runs/{run_id}/events with Bearer authentication and Accept: text/event-stream. Reconnect with the last completely processed Last-Event-ID. Native EventSource cannot set this Authorization header; use an authenticated streaming HTTP client. No credentials in URLs or anonymous fallback.

activity

One public notification per retained journal entry. Its id is a database-verified replay cursor; JSON contains only a decimal-string sequence and recording time. No private payload, custom kind or transcript.

snapshot

A current verified AgentHttpRunResponse, without an id. It does not describe historical state at a cursor. Intermediate snapshots may coalesce; quarantine-only changes are still detected.

error / heartbeat

Neither advances replay progress. Errors are best-effort, heartbeats are comments, and EOF never proves task success.

Process an activity idempotently before saving its cursor. A new process resumes strictly after that exact retained event. Malformed cursors are 400; changed, crossed or unavailable history returns agent_http.invalid_cursor (409). Never silently reset progress. This is not token streaming, complete historical state replay, or an exactly-once delivery guarantee.

Defaults: 16 stream producers, 60-second lifetime, 1-second polling/heartbeats, 5-second queue wait, one queued bounded batch, 64 MiB cumulative output. Streams use separate capacity from JSON requests. Credentials and resource policy are rechecked each iteration; already authorized buffered bytes cannot be recalled. Drop, queue timeout and shutdown stop producer work. Configure socket limits and disable proxy buffering/compression on this route.

Full client recovery and production-host guide · RFC-0009

Own readiness, connections and graceful drain

AgentHttpServer::start owns a loopback-only HTTP/1 listener behind a co-located TLS proxy. Startup verifies the actual PostgreSQL schema and exact executable registry, plus a mandatory host-provided AgentHttpReadiness check of the same credential-verifier and policy dependencies used by requests. No placeholder verifier, migration, Worker or scheduler starts implicitly.

Single-flight checks repeat every 10 seconds after completion, with a 5-second whole-check timeout and 20-second evidence freshness by default. Failed, timed-out or stale readiness rejects new business requests with 503; a later successful check restores admission. This is not an authorization grant or a guarantee that a future request succeeds. Already admitted work retains its normal authorization and deadlines.

Transport defaults are 256 connections, 64 headers, a 32 KiB parser buffer, 5-second header reads and a 300-second absolute connection lifetime. Excess sockets close without a business response. JSON/SSE limits remain separate. TLS, proxy limits and replica-wide quotas are still deployment responsibilities.

server.health() exposes only local status and active connection counts. Wire these to a protected admin or supervisor surface; there is no public health exception in the Agent API. Unavailable is live but not ready.

server.begin_shutdown() closes readiness immediately. Await server.shutdown() to stop accepting, finish admitted work, then abort and join remaining connection tasks after the default 30-second drain. SSE may require force-close. Inspect the drain report and preserve original retry identities: closure never proves rollback. Dropping the handle aborts work but does not await graceful drain. Blocking callbacks still require an outer process-supervisor kill deadline; shared DB pools remain host-owned.

Real PostgreSQL 16/17 tests cover readiness loss/recovery, startup failure, commit during drain, slow-body/SSE force-close, idle/drop cleanup and transport ceilings. Real issuer/policy-backed role integration and stable release qualification remain separate gates. Complete wiring and operations guide · RFC-0010

Recover a lost response without creating more work

Persist a high-entropy submission key and exact logical request before sending. Persist both cancellation IDs before cancelling. On timeout, disconnect, shutdown or an oversized response, the database may already have committed.

Read by the original key/Run or retry the same content and identities with bounded backoff. Changed submission content or a competing cancellation pair is 409. Inspect durable state; do not generate a new key merely because the response was lost. A recovered snapshot may have a newer revision.

Finite limits and payload-safe errors

Defaults: 256 KiB request, 2 MiB response, 64 concurrent requests per shared instance and a 15-second cooperative deadline. Hard ceilings: 2 MiB request, 8 MiB response, 1,024 concurrent requests and 60 seconds. JSON also has structural bounds. Blocking host code cannot be preempted by an async timeout.

400 rejects malformed input; 401/403 reject credential or policy; 404 follows authorization; 405/406/415 reject unsupported HTTP forms; 409 reports conflict; 413 rejects body size/read failures; 429 signals capacity; 503 signals unavailable/deadline/shutdown; 500 signals integrity/configuration or response limits.

Every response is private/no-store JSON with a correlation ID and API-version header. Errors expose only a fixed agent_http.* code and request_id, never raw token, SQL or provider messages. 429 and 503 include Retry-After: 1.

Before exposing a production endpoint

1. Pin compatible source, lockfile, migrations, graphs and schemas. Run an independently configured scheduler/Worker; submission alone does not advance a Run.
2. Configure TLS and an exact Host allowlist, connection/header limits, per-tenant quotas and replica-wide admission control. Forwarded headers do not expand trust. Origin is denied by default; optional exact HTTPS origins do not enable CORS.
3. Install real credential verification and exact resource policy. Exercise expiry, revocation, wrong audience and tenant isolation. Never deploy the tests’ static credentials.
4. Keep authorization, keys, input/output and raw paths out of logs. Monitor bounded latency/error metrics and dependency readiness.
5. Await server.shutdown() when using the owned role. Router-only hosts still call http.shutdown() and bound their own connection drain. Recover possibly committed requests with original identities after replacement.

Qualification covers actual HTTP + PostgreSQL 16/17, 24 concurrent identical submissions, lost submit/cancel responses, no inline dispatch, hostile input, authorization before an unavailable database, slow requests, overload and shutdown. SSE additionally covers new-process replay, revoked access, quarantine-only changes and unpolled responses. This does not qualify a live issuer, live model or an entire production deployment.

Full wire and operations guide · RFC-0008 · Development status