Baseten
Calls Baseten’s management API (https://api.baseten.co/v1) directly. apply renders a Truss config.yaml for a custom server (base_image, docker_server), uploads it as a config-only Truss archive and creates the model multihull-<svc>-<provider>, or a new deployment of it, pushed to the production environment. On main, unreleased: tested against a mocked API, not yet run against Baseten.
- provider: baseten-us type: baseten priority: 4 baseten: {}Credentials: BASETEN_API_KEY, sent as Authorization: Api-Key. hull doctor reads /v1/users/me with it and fails on 401 (rejected key) or 403 (a key without management access, such as an inference-only key). Baseten requires a key on every inference request, so the router injects the same header from the snapshot, on traffic and probes. The deploy key needs full access, so it can manage every model in the workspace: set BASETEN_INFERENCE_API_KEY to an inference-only team key scoped to the production environment and this model to keep it out of the snapshot; hull doctor warns while it is unset. A target with credentials (The spec) always signs traffic with its own key and ignores BASETEN_INFERENCE_API_KEY. The router sends the key only to the model’s own inference host, model-<id>.api.baseten.co, which serves predict, async, wake and sync routes for that model; the management API is on api.baseten.co.
Requirements:
container.commandis required: it becomesdocker_server.start_command, which replaces the image entrypoint. It may not contain%: Baseten’s build writes it intosupervisord.conf, where%starts an interpolation. The same holds forcontainer.secretsnames, because a name that is not a shell identifier reaches that command throughexec env;hull plannotes both andapplyrefuses them. Supervisor also reads;or#after whitespace as the start of a comment, which cuts the command short, and drops the indentation after a line break, so a command whose rendered form has either, or any line break, is refused the same way. Move such shell text into a script in the image.- Baseten builds on the image to add its proxy, so the image must be Debian based with Python on
PATH. Port 8080 is reserved, andPORTis set to 8080 in every container: bind the server tocontainer.portwithout readingPORT.PORT,HOST,HOSTNAMEandPATHincontainer.envare dropped by Baseten;hull plannotes them. - The image must be public; private registry credentials are not wired yet.
Lifecycle
Section titled “Lifecycle”- A deployment is labelled with a digest of its Truss config.
applyagain with an unchanged config pushes nothing and only restores theproductionautoscaling settings if they drifted; changing onlyscaling.replicasis such an update. A changed config is a new deployment of the same model, promoted when it is ready, while the previous deployment keeps serving. A ready deployment for the same config, for example after a rollback, is promoted instead of pushed again. With no state,apply,destroyandrediscoverfind the model by name, and act only when it is the one model with that name in the organization (names are unique per team only) and one of its deployments is labelled with this service and provider; otherwise they stop with an error. scalesetsmin_replicaandmax_replicaon theproductionenvironment in one call, which applies to the deployment it serves; a 400 or 422 is a permanent refusal and changes nothing.hull logsreads the deployment’s logs from--since(at most 7 days) until now.destroydeletes the model, with all its deployments, and the secrets it mirrored. A 404 counts as gone.
Translation
Section titled “Translation”| Spec field | Baseten |
|---|---|
container.image |
Truss base_image.image |
container.command, port |
docker_server.start_command, server_port |
resources.gpu, gpuCount |
resources.accelerator (first class, :<count> above one), use_gpu; a CPU instance when empty |
resources.memory |
resources.memory |
scaling.replicas |
production autoscaling min_replica, max_replica, scale_down_delay: 300 |
scaling.concurrency |
concurrency_target and runtime.predict_concurrency |
container.health |
readiness_endpoint, liveness_endpoint; hull deploy --wait and hull status report Ready only once the pushed deployment serves production, is ACTIVE with a replica, and <url><path> answers 2xx. A rollout that fails while the previous deployment still serves and answers is Degraded, not Failed, so the router keeps routing to it; hull deploy --wait stops at the failed rollout and reports the target not ok instead of waiting out its timeout |
container.secrets |
Baseten secrets named multihull_<svc>_<provider>_<secret>_<hash> (non-alphanumerics become _; <hash> is 12 hex digits of SHA-256 over the exact service, provider and secret names, so targets whose names sanitise alike never share a secret), declared in the Truss secrets and read from /secrets/<name> by a sh -c wrapper that exports the variable the spec names, then execs the command. A name that is not a shell identifier, such as hf-token, goes through exec env, so its value sits in that process’s arguments until it execs the command; name secrets like HF_TOKEN to avoid that |
| endpoint | https://model-<id>.api.baseten.co/environments/production/sync, which forwards every path to the server, plus Authorization: Api-Key injected by the router |
No edge error signature yet: Baseten’s own errors carry a flat {"error": ...} body (502 or 503 when the container is unreachable), but a live run has to record them first.
Ref stored in state: model id, name and URL, deployment id, health path, the config digest and the mirrored secret names. Fallbacks stay warm because fallbackScaleToZero defaults to false, so min_replica is never 0 unless you opt in.