Security
Report vulnerabilities privately as described in SECURITY.md: GitHub private vulnerability reporting, from the repository’s Security tab. Never in a public issue or pull request. The policy also covers supported versions, response targets and scope.
Operator notes
Section titled “Operator notes”- Provider deploy credentials come from environment variables (
MODAL_TOKEN_IDandMODAL_TOKEN_SECRET,RUNPOD_API_KEY,BASETEN_API_KEY,REPLICATE_API_TOKEN,REPLICATE_CLI_AUTH_TOKEN) or native config files (kubeconfig,~/.modal.toml), or from a target’scredentials: env:NAMEorcredentials: file:PATHoverride (The spec). They are never written to the spec, the state database, plans or logs, and only the RunPod and Baseten keys and the Replicate token reach the snapshot (next item).REPLICATE_CLI_AUTH_TOKENgoes tocog login --token-stdinon stdin; Cog then keeps it in Docker’s credential store. A target’scredentialsnever goes tocog login. - Service secrets appear in the spec by name only. Values are read at apply time from environment variables named after them (
HF_TOKENfromHF_TOKEN; other names in upper case with-and.as_) and stored in the provider’s own secret store.hull planoutput carries no secret values. - The snapshot does carry the headers the router injects upstream, such as Modal proxy auth tokens from
MODAL_PROXY_TOKEN_IDandMODAL_PROXY_TOKEN_SECRET, and RunPod’sAuthorization: Bearer: a target’scredentialskey when it has one, elseRUNPOD_ENDPOINT_API_KEY, elseRUNPOD_API_KEY. RunPod needs a key on every inference request, so give the router a Restricted key throughRUNPOD_ENDPOINT_API_KEYfor targets withoutcredentials; a target withcredentialsputs its own key in the snapshot. Baseten’sAuthorization: Api-Keyfollows the same order: a target’scredentialskey, elseBASETEN_INFERENCE_API_KEY, elseBASETEN_API_KEY; give the router an inference-only team key throughBASETEN_INFERENCE_API_KEYfor targets withoutcredentials. Replicate needsAuthorization: Beareron every prediction too, from a target’scredentialstoken, elseREPLICATE_API_TOKEN, and its tokens cannot be scoped, so the snapshot then holds a token that can also manage the account’s models and deployments. The router forwards onlyPOST /predictionsto a Replicate endpoint and rejects request targets without a leading/or with.or..segments, so a router client cannot reach that management API through it. Protectsnapshot.json, any snapshot URL and the controller stream like a secret. - The translator conformance suite (
python/tests/test_conformance.py) checks the three items above for every provider: deploy credential and service secret values never appear in plan output, state or the snapshot, and traffic credentials appear only in the snapshot. It runs against fakes on every PR and against kind, Docker, Modal and RunPod in the live workflows. - A snapshot file (
hull deploy,hull snapshot,hull controller --snapshot-out) that carries credentials must be mode0600and owned by the router’s user. - The state database (
--state,MULTIHULL_STATE_BACKEND) is trusted input; protect it like the spec. Its refs name provider resources and, for a target withcredentials, the override reference (never the value). Onlyhull destroy --target <name>of a target no longer in the spec uses a stored reference, and its confirmation shows the reference. A plainhull destroystops before destroying anything when such a target is in state and names it, so a routine teardown never loads a planted kubeconfig, whoseexeccommand would run;hull statusand the controller’s refresh report such a target without building a provider. - Route API keys (
hull_<id>_<secret>) never reach the snapshot in plain text, only lowercase hexblake3("<id>_<secret>"). The router compares hashes in constant time. A route withoutauth.apiKeysaccepts every request. A route with it never runs open:hullrefuses to publish it when its key source yields no keys, and the router rejects every request on a route markedrequiredthat carries no key hashes. Protect the key file or environment variable like any other secret. See Router overview. - The router strips the client’s
Authorization,Proxy-Authorizationand hop-by-hop headers, then sets the provider’s auth headers from the snapshot. - The admin listener (
/metrics,/healthz,/debug/*) has no auth and no TLS. It defaults to127.0.0.1:9090. The Helm chart binds it to0.0.0.0and adds it to the router Service, so changingrouter.service.typetoLoadBalancerorNodePortexposes it. Never expose it publicly. - The controller’s discovery stream serves TLS from
hull controller --tls-cert --tls-key.--client-camakes it require router client certificates signed by that CA (mTLS). A bootstrap token inMULTIHULL_DISCOVERY_TOKEN(renamed with--token-env) is checked in constant time on every stream asauthorization: Bearer <token>; it is never logged. TLS needs client certificates, the token or both. Without TLS flags the controller refuses to start;--insecureserves plaintext for local development and logs a warning. Certificates load at startup, so restart the controller to rotate them. - The router’s
[snapshot]table takesca(trusted instead of the public roots),client_certandclient_keyfor mTLS, andtoken_envnaming the variable that holds the token, forgrpcs://andhttps://sources. Plaintextgrpc://andhttp://sources start only withinsecure = true, which never disables certificate checks on a TLS source. - TLS: set
[tls]cert and key inrouter.toml(orrouter.tls.secretNamein the chart) and the router terminates TLS itself with rustls, reloading the certificate onSIGHUP. Otherwise it serves plain HTTP; terminate TLS at a load balancer or ingress in front. Upstream TLS trusts the webpki roots plus an optionalupstream_ca. - Releases are built in GitHub Actions from
v*tags. PyPI and crates.io uploads both use trusted publishing behind thereleaseenvironment, and the Helm chart goes to GHCR with the workflow’s ownGITHUB_TOKEN, so no registry tokens are stored as repository secrets. - Dependabot opens weekly update PRs for Python, Rust, npm and GitHub Actions dependencies, and for digest and patch updates of the Docker base images.