Skip to content

Security

Report vulnerabilities privately as described in SECURITY.md: GitHub private vulnerability reporting, from the repository’s Security tab. Never in a public issue or pull request. The policy also covers supported versions, response targets and scope.

  • Provider deploy credentials come from environment variables (MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, RUNPOD_API_KEY, BASETEN_API_KEY, REPLICATE_API_TOKEN, REPLICATE_CLI_AUTH_TOKEN) or native config files (kubeconfig, ~/.modal.toml), or from a target’s credentials: env:NAME or credentials: file:PATH override (The spec). They are never written to the spec, the state database, plans or logs, and only the RunPod and Baseten keys and the Replicate token reach the snapshot (next item). REPLICATE_CLI_AUTH_TOKEN goes to cog login --token-stdin on stdin; Cog then keeps it in Docker’s credential store. A target’s credentials never goes to cog login.
  • Service secrets appear in the spec by name only. Values are read at apply time from environment variables named after them (HF_TOKEN from HF_TOKEN; other names in upper case with - and . as _) and stored in the provider’s own secret store. hull plan output carries no secret values.
  • The snapshot does carry the headers the router injects upstream, such as Modal proxy auth tokens from MODAL_PROXY_TOKEN_ID and MODAL_PROXY_TOKEN_SECRET, and RunPod’s Authorization: Bearer: a target’s credentials key when it has one, else RUNPOD_ENDPOINT_API_KEY, else RUNPOD_API_KEY. RunPod needs a key on every inference request, so give the router a Restricted key through RUNPOD_ENDPOINT_API_KEY for targets without credentials; a target with credentials puts its own key in the snapshot. Baseten’s Authorization: Api-Key follows the same order: a target’s credentials key, else BASETEN_INFERENCE_API_KEY, else BASETEN_API_KEY; give the router an inference-only team key through BASETEN_INFERENCE_API_KEY for targets without credentials. Replicate needs Authorization: Bearer on every prediction too, from a target’s credentials token, else REPLICATE_API_TOKEN, and its tokens cannot be scoped, so the snapshot then holds a token that can also manage the account’s models and deployments. The router forwards only POST /predictions to a Replicate endpoint and rejects request targets without a leading / or with . or .. segments, so a router client cannot reach that management API through it. Protect snapshot.json, any snapshot URL and the controller stream like a secret.
  • The translator conformance suite (python/tests/test_conformance.py) checks the three items above for every provider: deploy credential and service secret values never appear in plan output, state or the snapshot, and traffic credentials appear only in the snapshot. It runs against fakes on every PR and against kind, Docker, Modal and RunPod in the live workflows.
  • A snapshot file (hull deploy, hull snapshot, hull controller --snapshot-out) that carries credentials must be mode 0600 and owned by the router’s user.
  • The state database (--state, MULTIHULL_STATE_BACKEND) is trusted input; protect it like the spec. Its refs name provider resources and, for a target with credentials, the override reference (never the value). Only hull destroy --target <name> of a target no longer in the spec uses a stored reference, and its confirmation shows the reference. A plain hull destroy stops before destroying anything when such a target is in state and names it, so a routine teardown never loads a planted kubeconfig, whose exec command would run; hull status and the controller’s refresh report such a target without building a provider.
  • Route API keys (hull_<id>_<secret>) never reach the snapshot in plain text, only lowercase hex blake3("<id>_<secret>"). The router compares hashes in constant time. A route without auth.apiKeys accepts every request. A route with it never runs open: hull refuses to publish it when its key source yields no keys, and the router rejects every request on a route marked required that carries no key hashes. Protect the key file or environment variable like any other secret. See Router overview.
  • The router strips the client’s Authorization, Proxy-Authorization and hop-by-hop headers, then sets the provider’s auth headers from the snapshot.
  • The admin listener (/metrics, /healthz, /debug/*) has no auth and no TLS. It defaults to 127.0.0.1:9090. The Helm chart binds it to 0.0.0.0 and adds it to the router Service, so changing router.service.type to LoadBalancer or NodePort exposes it. Never expose it publicly.
  • The controller’s discovery stream serves TLS from hull controller --tls-cert --tls-key. --client-ca makes it require router client certificates signed by that CA (mTLS). A bootstrap token in MULTIHULL_DISCOVERY_TOKEN (renamed with --token-env) is checked in constant time on every stream as authorization: Bearer <token>; it is never logged. TLS needs client certificates, the token or both. Without TLS flags the controller refuses to start; --insecure serves plaintext for local development and logs a warning. Certificates load at startup, so restart the controller to rotate them.
  • The router’s [snapshot] table takes ca (trusted instead of the public roots), client_cert and client_key for mTLS, and token_env naming the variable that holds the token, for grpcs:// and https:// sources. Plaintext grpc:// and http:// sources start only with insecure = true, which never disables certificate checks on a TLS source.
  • TLS: set [tls] cert and key in router.toml (or router.tls.secretName in the chart) and the router terminates TLS itself with rustls, reloading the certificate on SIGHUP. Otherwise it serves plain HTTP; terminate TLS at a load balancer or ingress in front. Upstream TLS trusts the webpki roots plus an optional upstream_ca.
  • Releases are built in GitHub Actions from v* tags. PyPI and crates.io uploads both use trusted publishing behind the release environment, and the Helm chart goes to GHCR with the workflow’s own GITHUB_TOKEN, so no registry tokens are stored as repository secrets.
  • Dependabot opens weekly update PRs for Python, Rust, npm and GitHub Actions dependencies, and for digest and patch updates of the Docker base images.