This document explains the dual-storage architecture that combines PostgreSQL and Redis, the Prisma ORM configuration with primary/replica separation, the Redis partitioning strategy for queues, locks, and cache, and the integration with additional storage systems like ClickHouse and S3/R2.
For information about database schema and models, see Database Schema Overview. For Redis queue management details, see Queue Management. For database migrations, see Database Migrations.
Trigger.dev employs a multi-tier storage architecture that separates concerns across different storage systems optimized for their specific use cases.
Sources: apps/webapp/app/services/worker.server.ts4 apps/webapp/app/v3/services/triggerTask.server.ts1-14
| Storage System | Primary Use Cases | Access Pattern | Data Volatility |
|---|---|---|---|
| PostgreSQL Primary | Task runs, deployments, organizations, write operations | Strong consistency, ACID transactions | Persistent |
| PostgreSQL Replica | Dashboard queries, read-heavy operations | Eventually consistent reads | Persistent |
| Redis Queues | Task run queues, fair selection algorithm | Sorted sets with atomic operations | Transient |
| Redis Locks | Distributed coordination, run locking | Redlock algorithm, short TTL | Transient |
| Redis Cache | API keys, metadata, session data | Memory + Redis tiered cache | Transient |
| GraphileWorker | Background job queue (legacy) | PostgreSQL-backed queue | Persistent |
| ClickHouse | Analytics, trace events, metrics | Time-series append-only | Persistent |
| S3/R2 | Deployment contexts, build artifacts | Object storage | Persistent |
Sources: apps/webapp/app/services/worker.server.ts136-145
Trigger.dev uses Prisma as the primary database abstraction layer with a dual-client pattern for read/write separation.
Sources: apps/webapp/app/services/worker.server.ts4
The codebase follows a consistent pattern for choosing between prisma and $replica:
| Operation Type | Client Choice | Example Code Location |
|---|---|---|
| Write operations | prisma | Task creation, status updates, deployments |
| Consistent reads | prisma | Reading immediately after writes |
| Dashboard queries | $replica | Run lists, organization queries |
| Count queries | $replica | Pagination, statistics |
| Background jobs | Both | ZodWorker configured with both clients |
The ZodWorker initialization demonstrates this pattern:
apps/webapp/app/services/worker.server.ts134-145
Sources: apps/webapp/app/services/worker.server.ts134-145
Trigger.dev implements singleton pattern for Prisma clients in development to prevent connection exhaustion:
Sources: apps/webapp/app/services/worker.server.ts115-122
The connection string configuration supports connection pooling modes:
DATABASE_URL: Primary connection stringDIRECT_URL: Direct connection (bypasses poolers like PgBouncer)noPreparedStatements: Set to true when using connection poolersapps/webapp/app/services/worker.server.ts139-143
Sources: apps/webapp/app/services/worker.server.ts115-143
Redis serves three distinct purposes in the Trigger.dev architecture, each with its own key namespace and access patterns.
Sources: Based on system architecture diagrams
Redis sorted sets store task run queues with scores based on priority and enqueue time. The hierarchical queue structure enables fair selection algorithms:
marqs:queue:prod): Top-level queue containing environment referencesmarqs:queue:env:{envId}): Per-environment queues containing task referencesmarqs:queue:task:{taskId}): Per-task queues containing run IDsThis three-tier structure enables:
Sources: Referenced in system architecture diagrams for Queue Management
The system uses the Redlock algorithm for distributed locking, ensuring only one worker processes a task run at a time:
run:lock:{runId}RunLocker class with RedlockSources: Referenced in system architecture diagrams
The @internal/cache package provides a tiered caching strategy:
Typical cache patterns:
| Cache Key Pattern | TTL | Use Case |
|---|---|---|
api-key:{key} | Long (hours) | API key validation |
env:{id} | Medium (minutes) | Environment metadata |
idempotency:{key} | Configurable | Request deduplication |
Sources: Referenced in system architecture diagrams
The background job system uses GraphileWorker, which stores jobs in a dedicated PostgreSQL schema.
Sources: apps/webapp/app/services/worker.server.ts33-145
The worker system is configured through several environment variables:
WORKER_SCHEMA: Database schema for GraphileWorker tables (default: graphile_worker)WORKER_ENABLED: Enable/disable worker processingWORKER_CONCURRENCY: Number of concurrent jobsWORKER_POLL_INTERVAL: Job polling frequencyGRACEFUL_SHUTDOWN_TIMEOUT: Shutdown grace periodapps/webapp/app/services/worker.server.ts128-147
The workerCatalog object defines all available background jobs with Zod schemas:
apps/webapp/app/services/worker.server.ts33-103
Each job maps to a handler with configuration:
priority: Job priority (0 = highest)maxAttempts: Retry limithandler: Async function processing the jobSources: apps/webapp/app/services/worker.server.ts33-147
ClickHouse stores time-series analytics data:
Access pattern: Write-heavy, append-only, time-series queries.
Object storage for large binary data:
Access pattern: Infrequent reads, long-term retention.
Sources: Referenced in system architecture diagrams
Services encapsulate data access logic and coordinate between storage systems:
Sources: apps/webapp/app/v3/services/triggerTask.server.ts51-120
Model functions provide reusable data access utilities:
Example from apps/webapp/app/models/runtimeEnvironment.server.ts10-63:
This pattern:
Sources: apps/webapp/app/models/runtimeEnvironment.server.ts10-63
For operations requiring consistency across multiple tables or storage systems, services use Prisma transactions:
Note: Redis operations are not included in Prisma transactions. Services must handle Redis rollback logic explicitly if needed.
Sources: Common pattern across service classes
The storage layer integrates with structured logging to provide observability:
Sources: packages/core/src/logger.ts19-139 apps/webapp/app/services/logger.server.ts1-82
The Logger class automatically enriches logs with:
Configuration in apps/webapp/app/services/logger.server.ts49-59:
Sources: packages/core/src/logger.ts19-139 apps/webapp/app/services/logger.server.ts49-82
Refresh this wiki
This wiki was recently refreshed. Please wait 1 day to refresh again.