fix(meta): limit pending barriers in partial graphs

[wenym1]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

The global barrier worker previously applied streaming.in_flight_barrier_nums only to barriers still in flight in the database partial graph. Collected but uncommitted barriers did not count toward the limit, and snapshot-backfill partial graphs were not bounded by their total pending count.

This PR:

• applies in_flight_barrier_nums to the full database partial-graph pending count: in-flight, collected but not committed, and currently completing barriers;
• bounds snapshot-backfill pending barriers by in_flight_barrier_nums * snapshot_backfill_barrier_amplification_factor;
• changes the amplification factor's default to 1 and sets it to 10 in every CI configuration that explicitly sets in_flight_barrier_nums to 10, giving those CI snapshot-backfill graphs a pending limit of 100 while database graphs remain limited to 10;
• retains explicit command and required lifecycle barriers as soft-limit exceptions;
• queues snapshot and log-store catch-up barriers when the snapshot-backfill partial graph has no available capacity;
• keeps the tombstone backfill regression test on arrangement backfill by explicitly disabling snapshot backfill for its MV creation session.

Release investigation found that #26423 was not cherry-picked into either release-2.8 or release-3.0. Both release branches still have the underlying inflight-only/unbounded-catch-up behavior, but neither has the amplification configuration. Backports of this PR therefore need to account for the missing configuration.

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests.
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
• I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

• My PR needs documentation updates.

The generated example configuration and configuration reference document the pending-barrier semantics of in_flight_barrier_nums and the new multiplier semantics and default of snapshot_backfill_barrier_amplification_factor.

Release note

Barrier injection now limits the total number of pending barriers per database partial graph, including barriers that have been collected but not yet committed. Snapshot-backfill partial graphs use the same base limit multiplied by streaming.snapshot_backfill_barrier_amplification_factor, which defaults to 1.

Tests

• ./risedev generate-example-config
• cargo test -p risingwave_meta pending_barrier --lib
• cargo check -p risingwave_meta_node
• cargo fmt --all
• cargo clippy --all-targets --all-features
• bash -n ci/scripts/run-backfill-tests.sh
• git diff --check
• ./risedev slt -p 4566 -d dev './e2e_test/streaming/bug_fixes/stack_overflow_17342.slt'
• ./risedev slt -p 4566 -d dev './e2e_test/streaming/unaligned-join.slt'

The latest CI-configuration-only update was checked with git diff --check and an audit confirming that every ci*.toml with in_flight_barrier_nums = 10 also has snapshot_backfill_barrier_amplification_factor = 10. Per request, Clippy and local e2e tests were not rerun for that update.

[wenym1]

• fix(meta): limit pending barriers in partial graphs #26467 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[hzxa21]

LGTM

[github-actions]

❌ Cherry-pick failed for one or more branches. Please check the workflow run logs and consider retrying or manually cherry-picking.