Infrastructure providers
Infrastructure providers store your project's data and deliver its events and background jobs. Choose one implementation for each required slot in Project configuration.
Open a package's README for installation and configuration options.
Object storage
The storage provider stores objects, relationships, telemetry, authentication state, and execution history.
| Provider | Use for |
|---|---|
| SQLite | Persistent local development. |
| PostgreSQL | Shared storage for production services. |
Dataset storage
The lakeStorage provider stores datasets and their versions.
| Provider | Use for |
|---|---|
| DuckLake | A dataset lake with a local or PostgreSQL catalog and local or object storage. |
| Local lake | File-based datasets on local disk. |
File storage
The blobStorage provider stores the bytes behind file references.
| Provider | Use for |
|---|---|
| Local files | Files on your development machine. |
| S3 | Shared files in S3 or a compatible service. |
| Azure Blob Storage | Shared files in Azure. |
Events and jobs
The broker delivers events. The queues provider distributes jobs to workers. They serve different purposes even when both use Redis.
| Slot | Provider |
|---|---|
broker | Redis Streams |
broker | NATS JetStream |
queues | BullMQ |
For local development and tests, @sixb/core includes InMemoryStorage, InMemoryLakeStorage, InMemoryBlobStorage, InMemoryBroker, and InMemoryQueues. Their state is lost when the process stops and is not shared with other processes.
Choose a production setup
A typical setup uses PostgreSQL for objects, DuckLake with a PostgreSQL catalog and shared object storage for datasets, S3 for files, and Redis for both events and BullMQ jobs.
All services must use the same providers and project ID. Separate dataset and file storage locations so their contents can be managed independently. See Deployment for building and starting the services.