| Safe Haskell | None |
|---|---|
| Language | GHC2024 |
Keiro.Inbox
Description
Idempotent inbox for cross-bounded-context integration events.
The inbox lives in the consuming bounded context. When a Kafka consumer receives an integration event, the inbox records the event's stable external identity and runs the local handler in the same Postgres transaction. Duplicate redeliveries (Kafka offset retry, rebalance, producer republish) become observable as duplicates instead of re-running the handler.
The wrapper is a single-transaction primitive: the completed inbox row and the handler's local writes commit atomically. If the handler raises or condemns the transaction, the inbox row never appears and the next delivery starts fresh.
Completed-row retention defines the deduplication window. After
garbageCollectCompleted removes a row, a later delivery of the same key is
processed again. A concurrent GC can also delete a conflicting completed row
between the insert attempt and its lookup; the handler then commits without a
replacement deduplication row, so a later redelivery can run it again. These
cases preserve at-least-once delivery, not permanent exactly-once processing;
size retention beyond the maximum redelivery delay and keep handlers
idempotent.
Synopsis
- module Keiro.Inbox.Types
- lookupInbox :: forall (es :: [Effect]). Store :> es => Text -> Text -> Eff es (Maybe InboxRow)
- listInbox :: forall (es :: [Effect]). Store :> es => Text -> Eff es [InboxRow]
- garbageCollectCompleted :: forall (es :: [Effect]). Store :> es => NominalDiffTime -> UTCTime -> Eff es Int
- countInboxBacklog :: forall (es :: [Effect]). Store :> es => Eff es Int
- markFailedTx :: Text -> Text -> Text -> UTCTime -> Transaction ()
- runInboxTransaction :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a))
- runInboxTransactionWith :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> InboxPersistence -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a))
- runInboxTransactionWithKey :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Text -> Text -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (InboxResult a)
- runInboxTransactionWithRetries :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a))
- runInboxTransactionWithRetriesWith :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> InboxPersistence -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a))
- runInboxTransactionWithRetriesKey :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> Text -> Text -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (InboxResult a)
- runInboxTransactionBatch :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> InboxDedupePolicy -> InboxPersistence -> [(IntegrationEvent, Maybe KafkaDeliveryRef)] -> (IntegrationEvent -> Transaction a) -> Eff es [Either InboxError (InboxResult a)]
- sampleInboxBacklog :: forall (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Eff es ()
Re-exports
module Keiro.Inbox.Types
Storage primitives
lookupInbox :: forall (es :: [Effect]). Store :> es => Text -> Text -> Eff es (Maybe InboxRow) Source #
Read one inbox row.
listInbox :: forall (es :: [Effect]). Store :> es => Text -> Eff es [InboxRow] Source #
List inbox rows for a source, ordered by received_at. Test helper.
garbageCollectCompleted :: forall (es :: [Effect]). Store :> es => NominalDiffTime -> UTCTime -> Eff es Int Source #
Delete completed inbox rows older than keepFor from now.
Returns the number of rows deleted. The retention window defines the
duplicate-detection window: a redelivery that arrives after retention
GC has run will be processed again, so the window must exceed the
maximum delivery delay tolerated by the operator. The user guide
recommends 30 days as a default. See tryInsertCompletedTx for the related
concurrent-GC race and its at-least-once consequence.
countInboxBacklog :: forall (es :: [Effect]). Store :> es => Eff es Int Source #
Count inbox rows in a non-terminal state (backlog gauge source).
Backlog = rows still processing (in flight) or failed (awaiting a
retry decision). Completed rows are terminal and excluded.
markFailedTx :: Text -> Text -> Text -> UTCTime -> Transaction () Source #
Mark an inbox row failed inside the same transaction as the handler.
Transactional handler wrapper
runInboxTransaction :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a)) Source #
Run handler at most once for each (source, dedupe_key).
Computes the dedupe key from policy and kafka, then in one
transaction:
- Inserts the inbox row with status
completed. - If the row already exists, branches on its status:
InboxCompleted→InboxDuplicate;InboxProcessing→InboxInProgress;InboxFailed→InboxPreviouslyFailed. - Otherwise runs
handler; the row commits only if the handler succeeds.
The handler is invoked with the decoded IntegrationEvent so it does
not need to redecode bytes. On exception or condemn the whole
transaction rolls back, including the inbox row insert — the next
delivery sees no row and can retry.
runInboxTransactionWith :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> InboxPersistence -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a)) Source #
Variant of runInboxTransaction that controls success-path
envelope persistence.
PersistDedupeOnly keeps enough columns for dedupe and operator
correlation but stores an empty payload and omits schema, trace, and
attribute columns for successfully processed rows.
runInboxTransactionWithKey :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Text -> Text -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (InboxResult a) Source #
Lower-level variant that takes the dedupe key directly.
Use when the policy is not enough to express the identity scheme — for example, when the consumer joins fields from multiple headers or derives the key from the payload itself.
runInboxTransactionWithRetries :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a)) Source #
Run handler with opt-in poison-message accounting.
This wrapper behaves like runInboxTransaction for fresh messages,
duplicates, and in-flight rows, but changes the behavior for handler
exceptions and previously failed rows:
- A synchronous exception from
handlerrolls back the handler transaction, then records a failed attempt in a second transaction and returnsInboxHandlerFailedwith the new attempt count. - A previously failed row with
attempt_count < ceilingis retried. - A previously failed row with
attempt_count >= ceilingreturnsInboxPreviouslyFailedwithout running the handler. The consumer can commit its offset and move on; the failed inbox row is the dead-letter record for operator review.
condemn is not treated as a handler failure by this wrapper. It
keeps the original rollback semantics from runInboxTransaction.
runInboxTransactionWithRetriesWith :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> InboxPersistence -> InboxDedupePolicy -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (Either InboxError (InboxResult a)) Source #
Variant of runInboxTransactionWithRetries that controls success-path persistence.
runInboxTransactionWithRetriesKey :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> Text -> Text -> IntegrationEvent -> Maybe KafkaDeliveryRef -> (IntegrationEvent -> Transaction a) -> Eff es (InboxResult a) Source #
Lower-level retrying variant that takes the dedupe key directly.
runInboxTransactionBatch :: forall a (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Int -> InboxDedupePolicy -> InboxPersistence -> [(IntegrationEvent, Maybe KafkaDeliveryRef)] -> (IntegrationEvent -> Transaction a) -> Eff es [Either InboxError (InboxResult a)] Source #
Process a batch of inbox deliveries with a single transactional fast path.
The fast path computes each (source, dedupe_key), suppresses repeated
keys within the batch as duplicates, then runs all remaining deliveries
in one Postgres transaction. If any handler throws or condemns that
transaction, the whole batch rolls back and every original delivery is
retried through runInboxTransactionWithRetries. That fallback preserves
per-message failure accounting and prevents one poison message from
discarding unrelated batch mates.
condemn rolls the transaction back at commit but returns normally,
so it cannot be observed from the transaction's return value alone.
The batch detects it by re-reading one row it should have committed:
every write in the fast path belongs to a delivery classified
InboxProcessed (fresh insert as completed or retry promotion to
completed), so if the first such row is not completed after the
transaction returns, the whole batch was condemned and the per-message
fallback runs. A batch with no InboxProcessed rows performed no writes,
so a condemned transaction loses nothing.
sampleInboxBacklog :: forall (es :: [Effect]). (IOE :> es, Store :> es) => Maybe KeiroMetrics -> Eff es () Source #
Count the inbox backlog and record the gauge when metrics are enabled.
The backlog is non-terminal rows: legacy processing rows plus failed rows.
Schedule this on its own interval; it is intentionally not part of the
per-message intake path.