Skip to content

Baseten

Calls Baseten’s management API (https://api.baseten.co/v1) directly. apply renders a Truss config.yaml for a custom server (base_image, docker_server), uploads it as a config-only Truss archive and creates the model multihull-<svc>-<provider>, or a new deployment of it, pushed to the production environment. On main, unreleased: tested against a mocked API, not yet run against Baseten.

- provider: baseten-us
type: baseten
priority: 4
baseten: {}

Credentials: BASETEN_API_KEY, sent as Authorization: Api-Key. hull doctor reads /v1/users/me with it and fails on 401 (rejected key) or 403 (a key without management access, such as an inference-only key). Baseten requires a key on every inference request, so the router injects the same header from the snapshot, on traffic and probes. The deploy key needs full access, so it can manage every model in the workspace: set BASETEN_INFERENCE_API_KEY to an inference-only team key scoped to the production environment and this model to keep it out of the snapshot; hull doctor warns while it is unset. A target with credentials (The spec) always signs traffic with its own key and ignores BASETEN_INFERENCE_API_KEY. The router sends the key only to the model’s own inference host, model-<id>.api.baseten.co, which serves predict, async, wake and sync routes for that model; the management API is on api.baseten.co.

Requirements:

  • container.command is required: it becomes docker_server.start_command, which replaces the image entrypoint. It may not contain %: Baseten’s build writes it into supervisord.conf, where % starts an interpolation. The same holds for container.secrets names, because a name that is not a shell identifier reaches that command through exec env; hull plan notes both and apply refuses them. Supervisor also reads ; or # after whitespace as the start of a comment, which cuts the command short, and drops the indentation after a line break, so a command whose rendered form has either, or any line break, is refused the same way. Move such shell text into a script in the image.
  • Baseten builds on the image to add its proxy, so the image must be Debian based with Python on PATH. Port 8080 is reserved, and PORT is set to 8080 in every container: bind the server to container.port without reading PORT. PORT, HOST, HOSTNAME and PATH in container.env are dropped by Baseten; hull plan notes them.
  • The image must be public; private registry credentials are not wired yet.
  • A deployment is labelled with a digest of its Truss config. apply again with an unchanged config pushes nothing and only restores the production autoscaling settings if they drifted; changing only scaling.replicas is such an update. A changed config is a new deployment of the same model, promoted when it is ready, while the previous deployment keeps serving. A ready deployment for the same config, for example after a rollback, is promoted instead of pushed again. With no state, apply, destroy and rediscover find the model by name, and act only when it is the one model with that name in the organization (names are unique per team only) and one of its deployments is labelled with this service and provider; otherwise they stop with an error.
  • scale sets min_replica and max_replica on the production environment in one call, which applies to the deployment it serves; a 400 or 422 is a permanent refusal and changes nothing.
  • hull logs reads the deployment’s logs from --since (at most 7 days) until now.
  • destroy deletes the model, with all its deployments, and the secrets it mirrored. A 404 counts as gone.
Spec field Baseten
container.image Truss base_image.image
container.command, port docker_server.start_command, server_port
resources.gpu, gpuCount resources.accelerator (first class, :<count> above one), use_gpu; a CPU instance when empty
resources.memory resources.memory
scaling.replicas production autoscaling min_replica, max_replica, scale_down_delay: 300
scaling.concurrency concurrency_target and runtime.predict_concurrency
container.health readiness_endpoint, liveness_endpoint; hull deploy --wait and hull status report Ready only once the pushed deployment serves production, is ACTIVE with a replica, and <url><path> answers 2xx. A rollout that fails while the previous deployment still serves and answers is Degraded, not Failed, so the router keeps routing to it; hull deploy --wait stops at the failed rollout and reports the target not ok instead of waiting out its timeout
container.secrets Baseten secrets named multihull_<svc>_<provider>_<secret>_<hash> (non-alphanumerics become _; <hash> is 12 hex digits of SHA-256 over the exact service, provider and secret names, so targets whose names sanitise alike never share a secret), declared in the Truss secrets and read from /secrets/<name> by a sh -c wrapper that exports the variable the spec names, then execs the command. A name that is not a shell identifier, such as hf-token, goes through exec env, so its value sits in that process’s arguments until it execs the command; name secrets like HF_TOKEN to avoid that
endpoint https://model-<id>.api.baseten.co/environments/production/sync, which forwards every path to the server, plus Authorization: Api-Key injected by the router

No edge error signature yet: Baseten’s own errors carry a flat {"error": ...} body (502 or 503 when the container is unreachable), but a live run has to record them first.

Ref stored in state: model id, name and URL, deployment id, health path, the config digest and the mirrored secret names. Fallbacks stay warm because fallbackScaleToZero defaults to false, so min_replica is never 0 unless you opt in.