Replicate
Replicate runs Cog models only, so a service with a Replicate target sets build.target: cog and every target runs the same Cog image. apply calls Replicate’s HTTP API (https://api.replicate.com/v1) and the Cog CLI: it creates the private model <account>/multihull-<svc> if missing, runs cog push r8.im/<account>/multihull-<svc> in build.context, then creates or updates the deployment multihull-<svc>-<provider> with the pushed version. On main, unreleased: tested against a mocked API, not yet run against Replicate.
container: build: {context: ., target: cog} port: 5000 health: {path: /health-check}
targets: - provider: replicate type: replicate priority: 2Credentials: REPLICATE_API_TOKEN, sent as Authorization: Bearer. hull doctor reads the token’s account and fails on 401 or when cog is not on PATH. cog push needs a Cog login: set REPLICATE_CLI_AUTH_TOKEN (from replicate.com/auth/token) and apply runs cog login --token-stdin with it, or run cog login once yourself. replicate.owner is optional; when set it must be the token’s account, since Replicate creates models and deployments only there. A target’s credentials: env:NAME or credentials: file:PATH replaces REPLICATE_API_TOKEN for API calls, hull doctor and the router’s Authorization. It never goes to cog login: Replicate issues a separate CLI token, so cog push keeps using REPLICATE_CLI_AUTH_TOKEN or an existing login, which must belong to the same account as the override.
Build context
Section titled “Build context”A relative build.context resolves against the directory of multihull.yaml. cog push copies the whole context into the image except what .dockerignore excludes, so Multihull reads the context the same way:
hull planhashes the files Docker would copy (skipping.git,.cogand__pycache__) and shows the hash aspush.digest; a changed hash plans aschanged, andapplypushes only then.applyrefuses a context that would ship.multihull/(state, plans and snapshots), a Multihull snapshot (it carries provider credentials), a Multihull state database or a file any target names incredentials: file:PATH(resolved like the override itself:~expands, relative paths against the working directory) or an existing native credentials file of any provider (eachKUBECONFIGentry,~/.kube/config, the in-cluster service account token,$MODAL_CONFIG_PATH,~/.modal.toml,$DOCKER_CONFIG/config.json,~/.docker/config.json,~/.dockercfg), matched by file identity so hard links and other spellings count, or any file holding a value of 8 characters or more from a target’s credentials, any provider’s native credential variable (REPLICATE_API_TOKEN,REPLICATE_CLI_AUTH_TOKEN,RUNPOD_API_KEY,RUNPOD_ENDPOINT_API_KEY,BASETEN_API_KEY,BASETEN_INFERENCE_API_KEY,MODAL_TOKEN_ID,MODAL_TOKEN_SECRET,MODAL_PROXY_TOKEN_ID,MODAL_PROXY_TOKEN_SECRET) or a native credentials file (up to 1 MiB). Copies count, in whole or in part: besides each file’s full contents, the scan looks for each kubeconfig user’stokenandclient-key-data(also decoded) and the files itsclient-keyandtokenFilename (tokens also base64-encoded, key files also askubectl config view --flattenwrites them),.modal.tomltoken ids and secrets, and Dockerauthsentries. Passwords (a kubeconfigpassword, a Docker registry password, also decoded fromauth) count only from 16 characters, so a short password that is also an ordinary word does not block a push; tokens and keys count from 8. A target’scredentialsin those formats is split the same way; an unreadable or malformed native file is skipped, never fatal. The message names the shipped file and the source (a credentials file’s path or a variable’s name), never the value, andhull plannotes it.- The scan reads every shipped file once and searches it for each value, so its cost grows with both: about 0.07 s per GB per value on an Apple M4 Pro (4.3 s per GB with 60 values). Large weights in the context add that to every
hull planandapply; a single-pattern regex search measured slower, not faster. Runninghullfrom the context directory is fine once.dockerignorelists.multihull; also listmultihull.yamlso editing the spec does not push a new image, and.env*so a local token file never ships.examples/replicate-echo/.dockerignorehas both. - Two Replicate targets with the same account share the model; their pushes run one at a time.
Routing
Section titled “Routing”Replicate has no HTTP passthrough: a deployment answers POST /v1/deployments/<owner>/<name>/predictions with a Cog prediction body. The endpoint in the snapshot is the deployment URL, so a client calling POST /predictions with {"input": {...}} reaches the predictions URL on Replicate and the Cog server’s own /predictions on every other target. Routes for Cog services use protocol: http; an OpenAI-compatible request has no Replicate equivalent.
The router forwards only POST /predictions to a Replicate endpoint, so a router client can never reach Replicate’s management API with the injected token. Other requests, such as GET /health-check, go to the route’s other targets; a route with only Replicate targets answers them 404 not_forwarded.
The router injects Authorization: Bearer and sets Prefer: wait=<n> with n five seconds below its first-byte timeout, between 1 and 60. The first-byte timeout is router-wide, [timeouts] first_byte in router.toml (30 s by default, so wait=25). Prefer: wait makes Replicate hold the response until the prediction finishes, like a Cog server does; a prediction still running at n seconds comes back as a success with status starting or processing and no output, and its polling URL is on Replicate, not the router. That avoids a first-byte timeout, which the router would retry on another target while Replicate keeps running and billing the first prediction. Keep predictions shorter than the wait or call Replicate directly for long ones.
Health: the router probes the deployment URL itself (GET), not container.health.path, because Replicate does not expose the container. A passing probe means only that the deployment exists and the token works. Replicate’s API reports no instance state, so a deployment whose instances are down or still booting passes the probe; only request outcomes (timeouts, 5xx) take it out of rotation.
Lifecycle
Section titled “Lifecycle”applysends onePATCHwith the fields that differ from the deployment’s current release (version, hardware, instances), or nothing; everyPATCHcreates a new release. With no state,applypushes and adopts the deployment by name.applyfails before any call whencontainer.envorcontainer.secretsis set (the deployments API takes no environment; build configuration into the image), whenscaling.replicasexceeds Replicate’s limits (min_instances5,max_instances20), when no GPU class maps to hardware, or when the build context is unsafe (above). It checks the chosen hardware againstGET /v1/hardwarebefore pushing.hull statusreports Failed whilemax_instancesis 0 (after a deferreddestroy), so the controller’s snapshot keeps the endpoint out of rotation too.hull deploy --waitandhull statusreport Ready once the deployment’s current release runs the applied version withmin_instancesof at least 1. That is configuration, not instance health (see Health above).scalepatchesmin_instancesandmax_instances; values above the limits, a 400 or a 422 are permanent refusals.hull logsprints nothing: Replicate has logs per prediction, not per deployment.destroysetsmin_instancesandmax_instancesto 0, then deletes the deployment. Replicate deletes only deployments offline and unused for 15 minutes; when it refuses (400, 409 or 422, assumed since Replicate documents no status),destroykeeps the state record marked Failed, which the snapshot publishes as down so the router sends it nothing (Draining would not: it only stops new sticky sessions), and you rundestroyagain later.destroyalso drops any floor the controller raised for the target, the controller never scales back a target that is not Ready, and a status call that fails meanwhile keeps the target Failed, so nothing brings the deployment back above 0/0. 401, 403, 429 and 5xx are errors. A 404 counts as gone. The model and its versions stay, like images in a registry.
Translation
Section titled “Translation”| Spec field | Replicate |
|---|---|
container.build |
cog push r8.im/<account>/multihull-<svc> in build.context; model created private |
container.command, port |
ignored; the Cog image serves its predictor (port 5000 on other targets) |
resources.gpu |
deployment hardware, first class Replicate offers: L4 to gpu-l40s, A10G to gpu-a40-small, A100-40 and A100-80 to gpu-a100-large, H100 to gpu-h100; cpu when empty |
scaling.replicas |
min_instances, max_instances |
scaling.concurrency |
not applicable |
container.health |
not sent; the router probes the deployment URL |
container.env, secrets |
refused |
| endpoint | https://api.replicate.com/v1/deployments/<owner>/<name>, plus Authorization: Bearer from the snapshot and Prefer: wait=<n> from the router |
Ref stored in state: owner, model, deployment name, pushed version and the build context hash. Fallbacks stay warm because fallbackScaleToZero defaults to false, so min_instances is never 0 unless you opt in.
Example: examples/replicate-echo, a Cog echo model on Docker and Replicate.