What's Changed

• build(deps): bump golang.org/x/crypto from 0.24.0 to 0.31.0 by @dependabot[bot] in #410
• azure: Invalidate storage client if storage account name changes by @jshearer in #409
• build(deps): bump jinja2 from 3.1.4 to 3.1.5 in /docs by @dependabot[bot] in #412
• support for journal hibernation by @jgraettinger in #411
• build(deps): bump golang.org/x/net from 0.26.0 to 0.33.0 by @dependabot[bot] in #413
• broker: tear down a stale append pipeline before proxying by @jgraettinger in #414
• broker: tear down and restart a demoted replica by @jgraettinger in #415
• scale allocator member ItemLimit to the relative share of ItemSlots, and refine suspension eligibility by @jgraettinger in #416
• build(deps): bump github.com/golang/glog from 1.2.1 to 1.2.4 by @dependabot[bot] in #418
• increase zone name maximum length by @mdibaiee in #419
• consumer: resolution should allow omitted ProcessId by @jgraettinger in #417
• broker: require the topology is stable for longer before auto-suspending by @jgraettinger in #420
• message: nil out dequeued Envelope.Message after consumption by @jgraettinger in #422
• Phil/max msg size by @psFried in #426
• shard: convert errors to runes before truncating by @jgraettinger in #424
• build(deps): bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 by @dependabot[bot] in #425
• build(deps): bump jinja2 from 3.1.5 to 3.1.6 in /docs by @dependabot[bot] in #428
• build(deps): bump golang.org/x/net from 0.37.0 to 0.38.0 by @dependabot[bot] in #430
• consumer: Apply RPC takes optional hints to set on upsert by @jgraettinger in #429
• store_azure: if token refresh fails, allow the credential to be invalid by @jgraettinger in #431
• add client support for pre-shared authorization tokens by @jgraettinger in #432
• build(deps): bump requests from 2.32.0 to 2.32.4 in /docs by @dependabot[bot] in #433
• Update Apply APIs to authorize Upserts against full label sets by @jgraettinger in #434
• fix two rare bugs observed in Etcd response handling by @jgraettinger in #435
• fragment: slightly relax offset skip behavior by @jgraettinger in #437
• Add Robust Storage Health Monitoring to Gazette by @jgraettinger in #438
• feat: handle additional non-retryable errors in append service by @jgraettinger in #439
• stores: increase OK health-check interval and jitter by @jgraettinger in #440
• server: offer "http/1.1" and "h2" via ALPN by @jgraettinger in #445
• build(deps): bump golang.org/x/oauth2 from 0.20.0 to 0.27.0 by @dependabot[bot] in #441
• gazctlcmd: register stores in gazctl by @mdibaiee in #446
• message: change Clock.Tick() to increment by 1 microsecond by @jgraettinger in #448
• shards_prune: skip unhealthy stores when pruning shards by @mdibaiee in #449
• update many dependencies by @jgraettinger in #443

Full Changelog: v0.100.0...v0.101.0

This release includes a number of different bugfixes and improvements. Most notably, it includes support for mTLS, fine-grained authorizations, and streaming List responses for when you need to watch for changes to many journals. Also pulls in a fix to the etcd client to resolve a possible broker crash when an etcd server restarts after a period of no journal or topology changes.

What's Changed

• build(deps): bump requests from 2.22.0 to 2.32.0 in /docs by @dependabot in #383
• build(deps): bump pygments from 2.7.4 to 2.15.0 in /docs by @dependabot in #376
• build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.3.0 to 1.6.0 by @dependabot in #385
• Phil/colons in names by @psFried in #386
• build(deps): bump urllib3 from 1.26.5 to 1.26.19 in /docs by @dependabot in #387
• build(deps): bump jinja2 from 2.11.3 to 3.1.4 in /docs by @dependabot in #381
• build(deps): bump google.golang.org/protobuf from 1.31.0 to 1.33.0 by @dependabot in #380
• build(deps): bump idna from 2.8 to 3.7 in /docs by @dependabot in #375
• build(deps): bump k8s.io/apimachinery from 0.0.0-20190620073744-d16981aedf33 to 0.16.13 by @dependabot in #382
• build(deps): bump certifi from 2019.9.11 to 2023.7.22 in /docs by @dependabot in #373
• build(deps): bump k8s.io/client-go from 0.0.0-20190620074045-585a16d2e773 to 0.17.16 by @dependabot in #378
• fix: Add missing endian conversion macros to enable MacOS builds by @jshearer in #392
• azure: Don't panic when token refresh fails by @jshearer in #391
• build(deps): bump certifi from 2023.7.22 to 2024.7.4 in /docs by @dependabot in #394
• build(deps): bump github.com/gorilla/schema from 1.2.0 to 1.4.1 by @dependabot in #393
• small bug fix for mutex on store_azure by @ddowker in #395
• TLS, fine-grain Authorization, and List subscriptions by @jgraettinger in #390
• broker/client: surface JOURNAL_NOT_FOUND from RetryReader by @jgraettinger in #397
• make the maximum accepted gRPC message size configurable by @jgraettinger in #398
• broker/client: quiet more noisy logging by @jgraettinger in #399
• add grpc-web gateways for REST access to Journal and Shard APIs by @jgraettinger in #400
• grpc-gateway: emit default (zero-value) fields by @jgraettinger in #401
• server: update TLS config to not request a client cert at all by @jgraettinger in #403
• Add more information to se1quencer panic message by @psFried in #404
• use separate log statements for sequencer diagnostics by @psFried in #405
• message: account for duplicates across a recovered checkpoint boundary by @jgraettinger in #406
• keyspace: Etcd watch can retry on Unknown gRPC errors by @jgraettinger in #407
• update etcd dependency by @psFried in #408

Full Changelog: v0.99.0...v0.100.0

It's been a while since we've officially tagged a release. Sorry about that.

What's Changed

• client: AppendService must release buffers of error'd AsyncAppends by @jgraettinger in #298
• This refactors the gazctl command to allow importing the cli parsing to another project by @snowzach in #296
• client: AppendService owns its buffer sync.Pool by @jgraettinger in #299
• brokertest: fix graceful stop test flake by @jgraettinger in #302
• consumer: flexible transaction boundaries by @jgraettinger in #303
• performance instrumentation and configurable read channel sizes by @jgraettinger in #306
• build(deps): bump babel from 2.7.0 to 2.9.1 in /docs by @dependabot in #308
• protocols: explicitly mark messages for Equal generation by @jgraettinger in #310
• multiple improvements to facilitate local testing contexts by @jgraettinger in #311
• Add shard unassign cmd by @saterus in #312
• consumer: shard transitions account for delete-then-create race by @jgraettinger in #315
• Improve Unassign Shard Workflow by @saterus in #316
• client: add new metric instrumentation for reads & appends by @jgraettinger in #317
• Register Protobuf Enums by @saterus in #320
• Add DeferPublishUncommitted to message.Publisher by @psFried in #319
• work around for platform dependent test setup by @michaelschiff in #321
• parse SSEKMSKeyId from store url and pass to store config by @davidgmonical in #323
• address WithBalancerName deleted in grpc v1.46.0 by @apesternikov in #324
• gazctl: add shards recoverylog-play command by @mdibaiee in #325
• shards unassign command no longer stalls by @saterus in #322
• Allow customizing server Listener by @psFried in #327
• consumer: synchronously fail if StartCommit returns a pre-resolved error by @jgraettinger in #329
• error when s3 client is missing region config by @psFried in #330
• Fix keyspace missing watch events by @psFried in #336
• keyspace: update MemberId/RaftTerm on watch events by @psFried in #338
• consumer: add Shard.PrimaryLoop() client.OpFuture by @jgraettinger in #339
• broker: allow further banking of unused flow-control credit by @jgraettinger in #340
• feature: Implement azure-ad:// as a fragment store URI scheme by @jshearer in #341
• chore: Bump github.com/Azure/azure-storage-blob-go to v0.15.0 by @jshearer in #342
• shard unassign: handle non-PRIMARY assignments which are FAILED by @jgraettinger in #343
• Fix azure pre-signed urls by @psFried in #344
• message: Sequencer incrementally tracks pending partial sequences by @jgraettinger in #345
• broker: add optional ReadRequest.BeginModTime constraint by @jgraettinger in #346
• Phil/recoverylog pruning by @psFried in #347
• gazctl: re-use shard and journal clients by @psFried in #348
• Azure AD: fix my dumb mistake when refreshing user delegation credentials by @jshearer in #349
• gazctl: shard pruning should consider all hints by @jgraettinger in #350
• allocator: make assignment much much faster by @jgraettinger in #351
• align with grpc base/balancer to trigger reconnect in Idle state (when we move to gRPC 1.41+) by @ddowker in #335
• go mod: update grpc module to latest 1.59.0 by @mdibaiee in #352
• fix: Azure fragment store with multiple tenants by @jshearer in #354
• recoverylog: relax FSM to allow updating the content of a property by @jgraettinger in #356
• broker/client: discard async appends of deleted journals by @jgraettinger in #357
• take two: relax ErrJournalNotFound only when writing recovered ACKs by @jgraettinger in #358
• AppendService should retry on JOURNAL_NOT_FOUND by @jgraettinger in #359
• SegmentSet.Add allow missing LastOffset by @psFried in #360
• broker/consumer: add CreateRevision to ListResponse by @jgraettinger in #361
• Phil/pruning is hard by @psFried in #362
• Phil/pruning paranoia by @psFried in #363
• go.mod: update pinned gorocksdb and x/net by @jgraettinger in #364
• broker/fragment: region support for s3 stores by @williamhbaker in #365
• consumer: bound the size of an error stored as Etcd shard status by @jgraettinger in #366
• generalize first-write persistence and relax byte-zero read skipping by @jgraettinger in #367
• journal client: force using http1.1 when fetching fragments from clou… by @psFried in #368
• Phil/http client errors by @psFried in #369
• journal client: use http1.1 for fetching fragments, for real this time by @psFried in #370
• mainboilerplate: use a maximum-size connection flow control window by @jgraettinger in #371
• build(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 by @dependabot in #372

New Contributors

• @snowzach made their first contribution in #296
• @saterus made their first contribution in #312
• @davidgmonical made their first contribution in #323
• @apesternikov made their first contribution in #324
• @mdibaiee made their first contribution in #325
• @jshearer made their first contribution in #341
• @ddowker made their first contribution in #335
• @williamhbaker made their first contribution in #365

Full Changelog: v0.89.0...v0.99.0

• Optional consumer.MessageProducer interface allows applications
  to provide an alternate driver of message reading & parsing,
  outside of the built-in mechanisms for reading journals named
  in the ShardSpec.
• Binary-compatible enhancement of recovery logs allows
  for cheap "forks", where historical content of one log is used
  from within another.
  This introduced a breaking change to the recoverylog.Recorder,
  which now has a dedicated constructor and non-exported fields.
• Allow journal names in Read and Append RPCs to be extended with a
  'name;suffix', where the suffix is captured into consumer checkpoint offsets.
  This allows a single consumer to have multiple, concurrent reads of a journal.
• New instrumentation for testing and building custom consumer executables.
• Fix a memory leak in append flow control.
• Fix an allocator bug which could cause delayed convergence under pretty uncommon conditions.
• Fix some corner cases on timely reporting of shard progress in the Stat RPC.
• Update the versions of all dependencies.

• Behavior change: journal registers are a small set of labels which are used
  for cooperative, transactional locking. Where before a set of checked registers
  must always match current journal registers for an append to proceed, now an
  exception exists in the case where a journal has a fully empty set
  of registers (as is the case on recovery if journal consistency is lost).
• gazctl journals reset-head now issues concurrent resets to enumerated journals,
  and no longer fails if an attempt to reset a consistent journal loses an append race.
• Makefiles now build local Docker images gazette/broker:latest and
  gazette/examples:latest, instead of gazette-broker:latest and gazette-examples:latest.
• Additional sanity checks to ensure badly-behaved clients or peers cannot
  block an exiting broker from draining its server.
• Gazette no longer logs "wrong fragment format" warnings when encountering "directory" objects
  (those where the object key ends with a '/') in cloud storage. These were just adding noise and
  were not actually anything to worry about.
• Fix various test flakes.

• BUGFIX: Add a catch-all timeout on server GracefulStop, to ensure poorly
  behaved clients cannot keep a terminating broker around indefinitely (GH-266).

• % is now an allowed character in label values and journal names. This makes their grammar compatible with URL-encoding.
• Added grpc-prometheus instrumentation to client & server gRPC's.
• Several new metrics added:
  • gazette_shard_*: operational metrics of consumer shards
  • gazette_spool_*: metrics of journal fragment spools
  • gazette_journal_server_*: metrics of gRPC JournalServer invocations
• Some metrics are marked as DEPRECATED, and will be removed in the future:
  • gazette_commits_total
  • gazette_committed_bytes_total
  • gazette_consumer_*
• Some already-broken or unused metrics were removed.
• Improvements to benchmarks and fixed test flakes.

• BUGFIX: Fix potential UUID sequencing race in concurrent calls to PublishCommitted of the same message.Publisher, and add new regression testing (GH-259).
• Instrument message.Sequencer with Prometheus metrics.
• Add curl to release images, making it much simpler for node-zone.sh mappings to directly query cloud metadata APIs.
• Fix omitted registration of broker WriteHead metric.
• Various improvements to tests, addressing or instrumenting flakes.

• Update RocksDB to 6.7.3, and Go to 1.14.2, along with many other package dependencies.
• The etcdtest package no longer embeds an Etcd server. Instead, an etcd
  binary must be available on the PATH and is invoked as a sub-process,
  using Unix domain sockets. Users who use etcdtest themselves, or are running
  Gazette tests outside of the hermetic Docker build environment, must provide a
  reasonably recent version of etcd.
• Relatedly, the gazette client package no longer depends on Etcd (including the Etcd client).
• JSON-framed messages are now able to use custom marshal/demarshal routines.
• gazctl journals/shards apply now interprets revision -1 (previously
  disallowed) to mean "don't care", allowing specs to be applied which will always
  overwrite what's in Etcd.
  This makes sense for specs managed in another source-of-truth (eg, git).
• Mitigations and improved logging for issue GH-248.
• Remove explicit TCP keep-alive management, as this is now Go default.
• Various doc improvements & cleanups.

• Fix: as-ci target downloads go modules before invoking inner make, to allow external repos to include Makefiles of the gazette/core repo which are dynamically determined using the go mod tool.
• Update to Go 1.13.4