Skip to content

Router overview

The router is stateless. It reads a static router.toml and a dynamic endpoint snapshot, then routes each request by priority, weight or latency with per-request failover.

  1. Request target check: a target that does not start with / (absolute-form without a path, such as http://host?x, or *) or that has a . or .. segment (also with a ; parameter, such as ..;), raw or percent-encoded, gets 400 with error type bad_request_target and never reaches an upstream, since the router appends the target to the endpoint URL.
  2. Route match on host, path tree and headers. With protocol: openai, the body is buffered (up to 1 MiB) to read the model key.
  3. Score endpoints with one function: priority_spillover (default), weighted, ewma_latency, locality, optional sticky affinity.
  4. Attempt engine: phase timeouts (connect 2 s, first byte 30 s, idle 60 s, total 10 m), retry before the first committed byte, retry budget, hedging in v0.2. Outcomes: Success, Capacity, Transient (including a provider edge error such as Modal’s 404 modal-http: for a stopped app, matched by the signature the snapshot carries per endpoint), Fatal, ClientAbort; the table is in Health and circuits.
  5. Auth: client Authorization stripped, provider header injected from the snapshot. Replicate endpoints receive only POST /predictions; other requests skip them, and a route with no other endpoint answers 404 not_forwarded.
Source When
grpcs:// stream from hull controller Teams; carries Degraded signals back
snapshot.json written by hull deploy Solo dev, CI
HTTPS or object-store URL polled Multi-region routers without controller reachability

The gRPC stream and HTTPS URLs take a CA bundle, a client certificate and a bearer token under [snapshot]; plaintext grpc:// and http:// need insecure = true. See Controller stream.

The router nacks a streamed snapshot older than the one it holds. From any source, a route that arrives with no endpoints keeps the endpoints the router holds; removing the route from the snapshot stops serving it. See Versions and empty routes.

The last snapshot is persisted to disk and served indefinitely on control-plane loss, with an alert after 1 h.

router.toml
listen = "0.0.0.0:8080"
admin_listen = "0.0.0.0:9090"
[snapshot]
source = "file:///var/lib/multihull/snapshot.json"
[probe]
interval = 5
timeout = 2

[timeouts], [circuit], [admission], [pressure], [probe] and [retry] tune the data plane; router/router.example.toml lists every key with its default. Everything else is dynamic. SIGHUP reloads TLS; SIGTERM drains for up to 10 m.

OTel spans router.request and upstream.attempt{provider, endpoint, n, outcome}. Prometheus: router_requests_total{route,endpoint,outcome}, router_responses_total{route,status}, router_failovers_total{from,reason}, router_circuit_state, router_probe_total{endpoint,outcome}, router_upstream_ttft_seconds, router_queue_wait_seconds, router_concurrency_limit, router_retry_budget_remaining, router_output_tokens_total. JSON access log with attempts[]. /debug/endpoints, /debug/sessions and /debug/config on the admin listener.

hull top polls the admin listener and renders the router in the terminal:

Terminal window
hull top --admin http://127.0.0.1:9090 --interval 1.0

Per endpoint: provider, state (serving, recovering, down), circuit, probe, requests per second, in-flight requests, p50 and p95 upstream time to first token. Per route: responses per second, errors per second (4xx and 5xx responses), the error total and failovers by reason. Rates come from counter deltas between polls, so the first frame shows dashes. When the router stops answering the view keeps the last frame, says router unreachable, retrying and keeps polling. Ctrl-C exits.

Set route.auth.apiKeys.from in the spec to env:NAME (comma-separated keys) or file:PATH (one key per line, lines starting with # are comments, a relative path resolves against the working directory of hull). Clients send Authorization: Bearer hull_<id>_<secret>.

  • <id> is ASCII letters and digits. <secret> is printable ASCII without spaces and may contain underscores. Generate one with echo "hull_$(openssl rand -hex 4)_$(openssl rand -hex 32)".
  • Hashing contract: lowercase hex blake3("<id>_<secret>"), the key without its hull_ prefix. The control plane publishes only these hashes in the snapshot’s auth.api_key_hashes; the router hashes the presented key the same way and compares it with every hash in constant time. proto/testdata/api-key-hash.json is a test vector both sides assert.
  • hull deploy, destroy, snapshot and controller reject a malformed key before acting. The error names the source and entry, never the key.
  • A missing, malformed or unknown key gets 401 with error type unauthorized.
  • A route with auth.apiKeys never runs open. If its source is unset, missing, empty or only comments, hull deploy, destroy, snapshot and controller stop with an error naming the route and the source, and a running controller keeps its last snapshot and logs the error.
  • The snapshot marks such a route required. On a required route with no key hashes the router answers every request with 401 and logs a warning naming the route. Snapshots without the flag, from older control planes, keep an empty hash list open.
  • A route without auth.apiKeys accepts every request.
  • Per-key rate limits and provider or service allowlists are planned, not enforced yet.

The Helm chart in charts/multihull runs the router as a Deployment with a Service, ConfigMap for router.toml, and an optional PodDisruptionBudget and controller. A grpc snapshot source needs TLS values or an explicit plaintext opt-in; see Controller stream.

Each v* release pushes the chart to oci://ghcr.io/mishraprafful/charts/multihull. The chart version and appVersion are the release version without the v. The router image defaults to ghcr.io/mishraprafful/multihull-router:<appVersion> and the controller image to ghcr.io/mishraprafful/multihull-controller:<appVersion>; the same release publishes both. See Controller in the cluster for the controller’s values.

The chart and images are public, so no registry login or pull secret is needed:

Terminal window
helm install multihull oci://ghcr.io/mishraprafful/charts/multihull --version 0.2.0 \
-n multihull --create-namespace

helm show values oci://ghcr.io/mishraprafful/charts/multihull --version 0.2.0 lists every value. A private fork publishing its own packages needs helm registry login ghcr.io and a pull secret named in imagePullSecrets.

hull init targets namespace: multihull. The chart renders that Namespace (workloads.namespace, workloads.createNamespace: true) with the chart labels plus multihull.dev/managed-by: multihull, so one helm install leaves the cluster ready for hull deploy; hull never creates namespaces. The Namespace carries helm.sh/resource-policy: keep, so helm uninstall leaves it and the workloads in it; remove it with kubectl delete namespace. Helm cannot create its own release namespace from a template: when workloads.namespace equals the release namespace, as in the install above, the chart skips the manifest and --create-namespace creates it. workloads.forceCreateNamespace: true renders it anyway.

Chart.yaml carries the artifacthub.io/* annotations (license, links, images, changes) and charts/artifacthub-repo.yml holds the repository metadata. The listing appears once the owner registers oci://ghcr.io/mishraprafful/charts/multihull on Artifact Hub and the GHCR packages are public; the next release tag then pushes the metadata to the artifacthub.io tag next to the chart.