# Durable functions Deploy functions that checkpoint progress and resume from the last completed step for up to 366 days. ## What it is A durable function checkpoints its progress and resumes from the last completed step, so one execution can run for up to 366 days instead of the seconds a normal invocation allows. Use one for work that has to survive a restart: a multi-step order pipeline, a long agent run, a nightly batch that calls out to a slow API. Durable functions run locally and in the cloud on runtimes that ship the durable authoring API (JavaScript, TypeScript and Python). They are a separate collection from standard functions: - You **start** an execution instead of invoking a function. The start returns a handle immediately; the result arrives later. - Each **execution** is a resource with its own id, status, and result. - The two collections never accept each other's names or ids. A function's kind is fixed when it is created. ## How it relates - Belongs to a **project**, and reads **variables**, **databases**, and **storage** in that project like any function. - Its sources live in `volcano/functions/` beside standard ones. Only `volcano-config.yaml` says which are durable, so `volcano functions deploy --all` skips them and `volcano cloud durable deploy --all` picks them up. `-f` deploys one the manifest does not mention, and refuses a name the manifest declares standard — a kind cannot be changed once the function exists. - Every region the project deploys to has to offer durable execution, or the deploy is refused up front. - Durable execution needs the durable authoring API. A source is deployed on the newest runtime for its language that ships one, which is not always that language's default — a Python durable function needs a newer runtime than a standard one gets. A language that has no such runtime at all is refused before anything is uploaded. ```yaml version: 1 project: name: my-app functions: - name: hello - name: order-pipeline kind: durable ``` A durable entry takes `variable_scope` and `variables` like any other, and `durable deploy` sends what the manifest declares. Declaring nothing leaves an existing function's scope alone. ## CLI operations | Operation | Command | |---|---| | Deploy all declared, or one | `volcano cloud durable deploy --all \| -f `; `--visibility` takes `-f` | | List | `volcano cloud durable list [--page 1] [--limit 100]` | | Get | `volcano cloud durable get ` | | Delete | `volcano cloud durable delete [--yes]` | | Logs | `volcano cloud durable logs [deployment-id] --type build\|runtime [--follow] [--limit 100]` | | Start an execution | `volcano cloud durable start [--input …] [--name …]` | | List executions | `volcano cloud durable executions list [--status …] [--page 1] [--limit 100]` | | Get one execution | `volcano cloud durable executions get ` | | Stop an execution | `volcano cloud durable executions stop [--yes]` | | Schedule executions | `volcano cloud durable schedulers create --cron "0 * * * *" [--name …] [--input …] [--regions …]` | | List schedulers | `volcano cloud durable schedulers list ` | | Pause or resume one | `volcano cloud durable schedulers disable\|enable ` | | Delete one | `volcano cloud durable schedulers delete [--yes]` | | List approvals | `volcano cloud durable approvals list [--function …] [--status pending] [--execution …] [--since …] [--page 1] [--limit 100]` | | Get one approval | `volcano cloud durable approvals get ` | | Approve or deny one | `volcano cloud durable approvals approve\|deny [--comment …] [--yes]` | | Approval stats | `volcano cloud durable approvals stats [--function …] [--since 30d]` | | Send a message | `volcano cloud durable messages send [--data … \| --data-file …] [--idempotency-key …]` | | List messages | `volcano cloud durable messages list [--direction in\|out] [--name …] [--after …] [--limit 100] [--follow]` | | Open message waits | `volcano cloud durable messages waits ` | | Succeed a callback | `volcano cloud durable callbacks succeed --result … \| --result-file …` | | Fail a callback | `volcano cloud durable callbacks fail --type … [--message …]` | | Heartbeat a callback | `volcano cloud durable callbacks heartbeat ` | Wherever a command takes a function, it takes the name or the id. `deploy` needs exactly one of `--all` and `-f`. `--yes` skips the confirmation prompt the three destructive commands and the two approval decisions ask for. Every `approvals`, `messages`, and `callbacks` command takes `--json`. Schedulers require `SUPERAGENT` and are capped at 5 per project across standard and durable functions together; a create beyond that answers `403`. The table shows cloud commands. Remove `cloud` to use the same commands against the local environment, for example `volcano durable deploy --all`. Local waits resolve immediately by default while preserving checkpoint and replay behavior. Set `LOCAL_DURABLE_REAL_TIME=true` before `volcano start` when wait timing must match the deployed function. ## Examples [`volcano deploy`](https://docs.volcano.dev/cli/deploy.md) deploys the durable functions after the standard ones, together with the rest of the project. ```bash # Deploy every function volcano-config.yaml declares durable volcano cloud durable deploy --all # Deploy one, and let your project's signed-in users start it volcano cloud durable deploy -f order-pipeline --visibility authenticated # Start an execution and follow it volcano cloud durable start order-pipeline --input '{"order_id":4417}' volcano cloud durable executions get order-pipeline 66666666-6666-4666-8666-666666666666 # Only the executions still running volcano cloud durable executions list order-pipeline --status running # End one where it is volcano cloud durable executions stop order-pipeline 66666666-6666-4666-8666-666666666666 ``` `--status` takes the status as the API spells it: `pending`, `running`, `succeeded`, `failed`, `timed_out`, `stopped` or `unknown`. The dashboard labels those for reading, so a `pending` execution appears there as "Starting". `unknown` is terminal, like the four outcomes before it: the platform could not establish how the execution ended, so it carries no result and nothing will settle it later. Retry it under the same name, which picks that execution back up rather than starting a second one. ## Logs `durable logs` reads the same two log streams the standard collection has, on a durable function: ```bash # Why the deploy ended in "failed" volcano cloud durable logs order-pipeline --type build # What the function logged while its executions ran volcano cloud durable logs order-pipeline --type runtime --follow ``` `--type build` reads the deployment the function is on, which is the one that just failed. Name an earlier deployment to read its build logs instead: `volcano cloud durable logs order-pipeline --type build`. `--type runtime` spans every execution, including every resume: the code between context operations runs again each time, so a line logged there appears once per resume while a line inside a completed `step` does not. To filter runtime logs by level, text or time, or read several durable functions at once, use [`volcano cloud logs --kind durable`](https://docs.volcano.dev/cli/logs.md). `--follow` works on both, and stops at a different point for each: following build logs ends when the deploy does, following runtime logs runs until you interrupt it. A function that has never deployed has no build logs, and `--type build` says so rather than printing nothing. `volcano cloud functions logs` does not accept a durable function's name: the two collections never accept each other's names or ids. ## Scheduling executions A scheduler ticks on a cron expression and starts an execution instead of invoking the function, so every tick produces an execution you can list, follow, and stop like any other: ```bash # Start one execution an hour, with the same input each time volcano cloud durable schedulers create order-pipeline --cron "0 * * * *" --input '{"scope":"hourly"}' volcano cloud durable schedulers list order-pipeline volcano cloud durable executions list order-pipeline ``` Each tick names its execution after the run, so a tick Volcano has to retry resolves to the execution it already started rather than beginning a second one. Ticks draw on the same durable execution and operation allowances and the same concurrency cap a manual start does; a tick that would exceed the cap fails that run rather than queueing. A tick's run counts as failed when its execution fails, times out, or is stopped. `schedulers list` shows each scheduler's health and how many runs in a row have failed, and the project is alerted when the failures continue; see [notifications.md](https://docs.volcano.dev/cli/notifications.md). `schedulers disable` stops the ticks and leaves the scheduler in place; executions it already started keep running. `schedulers delete` removes the scheduler and its run history, and also leaves running executions alone — stop those with `executions stop`. ## Approvals A workflow that calls `ctx.waitForApproval` pauses until a person approves or denies it, then resumes with the decision. See [durable functions in the JavaScript SDK](https://docs.volcano.dev/sdk/js/durable-functions.md) for the authoring side. The `approvals` commands are the deciding side: ```bash # What is waiting on someone volcano cloud durable approvals list # Read what the workflow is asking, with the details it attached volcano cloud durable approvals get 0b6f3c1e-8a4d-4f7e-9c2b-5d1a7e3f9b20 # Decide it; the comment reaches the workflow and stays in the history volcano cloud durable approvals approve 0b6f3c1e-8a4d-4f7e-9c2b-5d1a7e3f9b20 --comment "Checked stock" volcano cloud durable approvals deny 0b6f3c1e-8a4d-4f7e-9c2b-5d1a7e3f9b20 ``` ```text ID: 0b6f3c1e-8a4d-4f7e-9c2b-5d1a7e3f9b20 Title: Ship order 4417? Name: ship-order Status: approved Workflow: order-pipeline Execution: order-4417 (66666666-6666-4666-8666-666666666666, running) Requested: 2026-10-06T11:00:00Z Expires: when its execution ends Details: { "order": 4417 } Decided By: owner@example.com Decided At: 2026-10-06T11:04:12Z Comment: Checked stock ✓ Approved "Ship order 4417?"; the workflow resumes with the decision ``` `list` shows pending approvals unless `--status` says otherwise. It takes `pending`, `approved`, `denied`, `expired`, `cancelled`, or `all`. An approval ends in one of four ways: | Status | What happened | What the workflow gets | |---|---|---| | `approved` | A person approved it | `approved: true` and the comment | | `denied` | A person denied it | `approved: false` and the comment | | `expired` | Its timeout passed with nobody deciding | `status: 'expired'`; the wait resolves rather than failing | | `cancelled` | Its execution ended first | Nothing; the execution is already over | `approve` and `deny` show the title and ask before deciding, because a decision cannot be changed. Repeating the decision an approval already has changes nothing and succeeds, so a retried command is safe. A different decision, or any decision on an expired or cancelled approval, fails with what happened, for example `approval … was already approved by owner@example.com at …`. An approval past its timeout counts as expired, even while it still reads `pending`. Approvals are decided by a person. A command run with a project access token is refused before it asks; run `volcano login` with `VOLCANO_TOKEN` unset, then decide again. `stats` counts the approvals requested in a window, 30 days unless `--since` says otherwise, up to 366 days: ```bash volcano cloud durable approvals stats --since 7d ``` It prints the count in each status, the approval rate (approved out of approved and denied), the median time from request to decision, and the workflows that request the most. In local mode, drop `cloud`: `volcano durable approvals approve …` decides as the local user. An approval timeout runs in real time locally, even though plain waits resolve immediately. ## Messages A workflow that calls `ctx.waitForMessage` pauses until a message with that name arrives, and `ctx.publish` sends progress out of a running execution. See [durable functions in the JavaScript SDK](https://docs.volcano.dev/sdk/js/durable-functions.md) for the authoring side. The `messages` commands send to an execution and read what it received and published: ```bash # What the execution is waiting for volcano cloud durable messages waits order-pipeline 66666666-6666-4666-8666-666666666666 # Send it; the workflow resumes with the data volcano cloud durable messages send order-pipeline 66666666-6666-4666-8666-666666666666 \ payment-confirmed --data '{"amount":4200}' --idempotency-key pay-4417 # Watch what it publishes until it ends volcano cloud durable messages list order-pipeline 66666666-6666-4666-8666-666666666666 \ --direction out --follow ``` ```text ID: 8f3b6c2a-1d4e-4a7b-9c5f-2e8d1a6b3c70 Seq: 3 Direction: in Name: payment-confirmed Status: queued Sender: project Sent: 2026-10-12T09:00:00Z Data: { "amount": 4200 } ✓ Sent "payment-confirmed" to execution 66666666-6666-4666-8666-666666666666 ``` A message sent before the workflow waits for its name is `queued` until it does, and waits take messages oldest first. Without `--data` or `--data-file` the workflow receives `null`. An execution that has ended takes no messages: the send fails with `execution … has ended and takes no more messages`, and queued messages nobody took are marked `dropped`. `--idempotency-key` makes a send safe to retry. Repeating it with a key the execution has already seen prints the first message and `Already sent with this idempotency key; nothing changed`. Without the flag, `send` generates a key and prints it to stderr before sending, so a send whose outcome you did not see can be retried with that key. `--json` prints the message as the API returns it, the same for a repeat as for the first send. `list` reads both directions oldest first: `in` for messages sent to the execution, `out` for the ones it published. Each message has a `seq`; `--after` reads only the ones after it, `--limit` takes 1 to 500 per request, and a long list ends with the command for the rest. `--follow` keeps reading until the execution ends; with `--json` it prints one message per line. It prints each message once, with the status it had when first read, so a sent message shown as `queued` stays that way after a wait takes it. Anyone with access to the project can send and read messages, including a project access token. A read-only token can list but not send. In local mode, drop `cloud`: `volcano durable messages send …` sends to an execution running on the local engine. ## Callbacks A workflow that calls `ctx.waitForCallback` hands a `vcb_` token to another system and pauses until that system succeeds or fails the wait. See [durable callbacks](https://docs.volcano.dev/platform/functions/durable-callbacks.md) for the authoring side. The `callbacks` commands are the token holder's side: ```bash # Still working: restart the wait's heartbeat timeout volcano cloud durable callbacks heartbeat vcb_0000000000000000000000000000000000000000000 # Done: the workflow resumes with the result volcano cloud durable callbacks succeed vcb_0000000000000000000000000000000000000000000 \ --result '{"charge_id":"ch_1"}' # Or it failed: the workflow raises CallbackFailedError with this type and message volcano cloud durable callbacks fail vcb_0000000000000000000000000000000000000000000 \ --type CardDeclined --message "Insufficient funds" ``` ```text ✓ Succeeded the callback; the execution resumes with the result ``` The token is the only credential these commands send, so they need no login or project. Whoever holds the token can answer the wait, so treat it as a secret. Pass `-` in place of the token to read it from stdin, which keeps it out of shell history and the process list: ```bash printf '%s' "$CALLBACK_TOKEN" | volcano cloud durable callbacks succeed - --result-file charge.json ``` `--result` or `--result-file` is required: any JSON value other than `null`, up to 64 KiB. `null` is refused because a wait that times out resolves with `null`. `fail` requires `--type`, a name the workflow can branch on; `--message` is optional. `succeed` and `fail` return once the execution has the outcome. When handing it over takes longer, they report that Volcano is still delivering it. Volcano finishes the delivery on its own, so do not send it again. A token takes one answer. A later `succeed`, `fail`, or `heartbeat` fails with `the callback already has an outcome`, including a retry of the answer that went through, so a holder retrying after a lost response can treat it as done. Once the wait times out or its execution ends, every command fails and says so; nothing will take an answer, so do not retry. A `heartbeat` that cannot reach the execution right now leaves the wait unchanged; retry it with backoff. In local mode, drop `cloud`: `volcano durable callbacks succeed …` answers a callback on the local engine. A callback's timeout and heartbeat timeout run in real time locally. ## Starting is asynchronous `start` returns as soon as the execution is accepted, with the execution's id and name. Read the execution to get its status and, once it has finished, its result: ```bash volcano cloud durable start order-pipeline --input order.json --name order-4417 volcano cloud durable executions get order-pipeline ``` `--input` takes an inline JSON object or the path to a file holding one, and is handed to the function verbatim. Omit it to start with no input at all, which is not the same as starting with `{}` — and not the same as `--input null`, which is refused, because it would arrive as neither. `--name` is the execution's idempotency key. Starting again under a name that already names an execution returns the existing one instead of beginning a second, and is not charged again — so a retried start is safe. Omit it and Volcano generates one. A name is letters, digits, `-`, `_` and `.`, up to 255 characters; anything else is refused. A deploy finishes asynchronously, so a start that follows one straight away can be refused while the function is still provisioning. Wait for `get` to report `active` and start again. ## Visibility Visibility decides who can start executions of a function: | Visibility | Who can start executions | |---|---| | `private` | Service keys and schedulers | | `authenticated` | Also your project's signed-in users, including [anonymous sign-ins](https://docs.volcano.dev/platform/authentication/anonymous-users.md) | | `public` | Also anon keys with `functions.invoke` | A caller a `private` function refuses gets `404` with `durable function not found`, the same as for a function that does not exist. If your app gets 404 for a durable function you deployed, check its visibility with `volcano cloud durable get `. A durable function is never invocable over HTTP the way a public standard function is, whatever its level: starting an execution is all the level allows. An anon key reads and stops only the executions whose `client_token` it sends, and a signed-in user also the ones they started. See [Durable functions](https://docs.volcano.dev/platform/functions/durable-functions.md#wait-for-the-result). A new durable function starts `private`, and `durable get` shows the level. To change it without rebuilding, declare `visibility` in [`volcano-config.yaml`](https://docs.volcano.dev/cli/project-configuration.md#function-visibility) and run `volcano cloud config deploy`: ```yaml functions: - name: order-pipeline kind: durable visibility: authenticated ``` A redeploy with `--visibility` also sets the level, for the one function `-f` names. Omit the flag and a redeploy keeps the level the function has. When a deploy creates a function without `--visibility`, it prints both ways to let signed-in users in. `--public` is the same as `--visibility public`. `--private` is no longer accepted: it used to let signed-in users in, which is `authenticated` now, so pick the level you mean. `--visibility` is not accepted with `--all`. Visibility is a per-function decision, so one flag applied to a whole manifest would take a redeploy of every function to undo. Declare levels in the manifest instead. ## Stopping and deleting `executions stop` requests the end of one execution. Cancellation happens behind the request, so the execution it prints back often still reads `running`; `executions get` is how you watch it reach `stopped`. Steps already completed are not undone. Stopping one that has already finished reports the state it is in rather than failing. `durable delete` tears down the function and its execution history. It does not wait for work in flight, so stop an execution you need ended first.