The Waitpoint System enables task runs to pause execution and wait for specific conditions before continuing. It provides the implementation behind wait.for(), wait.createToken(), and manual waitpoints, enabling time-based delays, human-in-the-loop workflows, and inter-task dependencies. When a run blocks on a waitpoint, concurrency is managed efficiently to allow other tasks to execute.
The system powers several user-facing features:
wait.for({ seconds: 10 }) - pauses execution for a durationwait.createToken() and wait.forToken() - enables approval workflows and external callbackstriggerAndWait() - waits for child tasks to completebatchTriggerAndWait() - waits for multiple tasks to finishFor information about checkpoints (which suspend execution with memory snapshots), see Checkpoint and Resume System. For concurrency management during waitpoints, see Concurrency Management.
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts1-25 docs/wait-for-token.mdx1-10
The wait.for() function pauses execution for a specified duration:
Internally, this creates a DATETIME waitpoint that completes automatically when the time elapses. The run transitions to EXECUTING_WITH_WAITPOINTS state, allowing monitoring systems to track that the run is intentionally waiting.
Sources: docs/wait.mdx (referenced), internal-packages/run-engine/src/engine/systems/waitpointSystem.ts178-255
The token-based API enables approval workflows and external callbacks:
The token's url property can be used as an HTTP callback endpoint. When a POST request is made to this URL, the waitpoint is completed with the request body as the output. This enables integration with external services like Replicate, Stripe webhooks, or custom approval UIs.
Sources: docs/wait-for-token.mdx10-74 internal-packages/run-engine/src/engine/systems/waitpointSystem.ts260-363
When a task triggers another task with triggerAndWait(), it blocks on the child task's associated RUN waitpoint:
Every task run automatically has an associated RUN-type waitpoint created during trigger. When the child completes, its associated waitpoint is completed with the task's output, unblocking the parent.
Sources: internal-packages/run-engine/src/engine/index.ts488-540
The system supports four types of waitpoints, each with distinct completion criteria:
| Type | Description | Completion Trigger | User-Facing API |
|---|---|---|---|
DATETIME | Time-based waitpoint | Completed automatically at specified time | wait.for() |
MANUAL | External completion required | Explicit API call or HTTP callback | wait.createToken(), wait.forToken() |
RUN | Associated with a task run | Task run completion | triggerAndWait() |
BATCH | Associated with batch execution | All batch items complete | batchTriggerAndWait() |
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts1-715
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts1-25 internal-packages/run-engine/src/engine/index.ts488-493
A common pattern using waitpoint tokens for approval workflows:
Key features:
url acts as an HTTP callback endpointpublicAccessToken enables secure completion without API keysSources: docs/wait-for-token.mdx1-74 internal-packages/run-engine/src/engine/systems/waitpointSystem.ts70-363
The WaitpointSystem class coordinates all waitpoint operations:
WaitpointSystem
├── createDateTimeWaitpoint() - Creates DATETIME waitpoints for wait.for()
├── createManualWaitpoint() - Creates MANUAL waitpoints for wait.createToken()
├── blockRunWithWaitpoint() - Blocks a run until waitpoint(s) complete
├── completeWaitpoint() - Marks waitpoint as completed
├── continueRunIfUnblocked() - Resumes run if no pending waitpoints
├── clearBlockingWaitpoints() - Removes all blocking waitpoints (on failure)
└── buildRunAssociatedWaitpoint() - Creates RUN waitpoint for triggerAndWait()
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts40-49
The WaitpointSystem is initialized in the RunEngine constructor and depends on:
ExecutionSnapshotSystem - for creating snapshots when blocking/unblockingEnqueueSystem - for re-queueing runs after waitpoints completeSystemResources - for database, locks, event bus, and worker queueSources: internal-packages/run-engine/src/engine/index.ts295-299
The system schedules a finishWaitpoint worker job to automatically complete the waitpoint at the specified time.
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts178-255
MANUAL waitpoints require explicit completion via completeWaitpoint(). They support optional timeout via the completedAfter field, which schedules a timeout error if not completed in time:
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts260-363
Every task run automatically gets an associated RUN waitpoint created during trigger:
This waitpoint is completed when the run finishes (success or failure) and used for triggerAndWait() patterns.
Sources: internal-packages/run-engine/src/engine/index.ts488-493
The blocking operation uses a raw SQL query to atomically insert waitpoint records and count pending ones, avoiding race conditions:
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts368-497
Key state transitions:
EXECUTING → EXECUTING_WITH_WAITPOINTS: Run blocks on waitpoint(s) via wait.for() or wait.forToken(). Concurrency is not released as the run is still actively executing code.EXECUTING_WITH_WAITPOINTS → EXECUTING: All waitpoints complete (via timeout, manual completion, or child task completion). Run continues execution seamlessly.EXECUTING_WITH_WAITPOINTS → SUSPENDED: Checkpoint created while waiting. Concurrency is fully released at this point, allowing other runs to execute.SUSPENDED → QUEUED: Waitpoints complete while suspended. Run is re-enqueued with its checkpoint and completed waitpoint data.Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts433-463 internal-packages/run-engine/src/engine/statuses.ts8-14 internal-packages/run-engine/src/engine/systems/checkpointSystem.ts195-202
The completion uses updateMany with a WHERE status = 'PENDING' filter to ensure idempotency - completing an already-completed waitpoint is safe.
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts70-172
The continueRunIfUnblocked() method checks if all blocking waitpoints are complete and resumes execution:
This method is called asynchronously via worker jobs, debounced at 50ms to handle rapid waitpoint completions efficiently.
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts499-728
When a run enters EXECUTING_WITH_WAITPOINTS, concurrency tokens are not released because the run is still actively executing code. However, the state signals to monitoring systems that the run is waiting and may take longer.
When a checkpoint is created while waitpoints are pending, the run transitions to SUSPENDED and concurrency is fully released:
This allows other runs to execute while waiting, making efficient use of environment-level and queue-level concurrency limits.
Sources: internal-packages/run-engine/src/engine/systems/checkpointSystem.ts195-240
Waitpoints support user-provided idempotency keys for deduplication:
When an idempotency key expires, the old waitpoint's key is moved to inactiveIdempotencyKey and a new random key is assigned, allowing the same idempotency key to be reused.
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts195-305
The RUN-type associated waitpoint enables triggerAndWait():
child.triggerAndWait()blockRunWithWaitpoint()Sources: internal-packages/run-engine/src/engine/index.ts527-540 internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts771-780
When a run attempt fails, all blocking waitpoints are cleared to prevent the run from remaining blocked indefinitely:
This ensures that retried attempts start with a clean slate.
Sources: internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts861-864
The system uses run locks and atomic database operations to prevent race conditions:
updateMany WHERE status = 'PENDING' for idempotencySources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts397-510
The continueRunIfUnblocked job is scheduled with a 50ms delay and uses the job ID as a deduplication key. This debounces rapid waitpoint completions:
Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts148-492
The system uses three worker jobs defined in the worker catalog:
| Job | Purpose | Trigger |
|---|---|---|
finishWaitpoint | Complete DATETIME/MANUAL waitpoints | Scheduled at completion time |
continueRunIfUnblocked | Check if run can continue | Debounced after waitpoint completion |
| N/A (inline) | Process completed waitpoints | Loop in completeWaitpoint() |
Sources: internal-packages/run-engine/src/engine/workerCatalog.ts4-58 internal-packages/run-engine/src/engine/index.ts188-227
Batch operations use BATCH-type waitpoints to coordinate multiple child runs:
The BatchSystem calls tryCompleteBatch() after each child completes, which checks if all batch items are finished and completes the BATCH waitpoint if so.
Sources: internal-packages/run-engine/src/engine/index.ts862-916 internal-packages/run-engine/src/engine/systems/batchSystem.ts (referenced but not provided)
Sources: internal-packages/run-engine/src/engine/index.ts488-493
Sources: internal-packages/run-engine/src/engine/tests/waitpoints.test.ts98-105
Sources: internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts775-780
Refresh this wiki