Skip to content

Claude/backpressure repartitionexec jv hp9#20762

Closed
Dandandan wants to merge 4 commits intoapache:mainfrom
Dandandan:claude/backpressure-repartitionexec-JvHP9
Closed

Claude/backpressure repartitionexec jv hp9#20762
Dandandan wants to merge 4 commits intoapache:mainfrom
Dandandan:claude/backpressure-repartitionexec-JvHP9

Conversation

@Dandandan
Copy link
Contributor

@Dandandan Dandandan commented Mar 6, 2026

Which issue does this PR close?

  • Closes #.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

claude added 2 commits March 6, 2026 18:54
@claude
feat: implement per-channel backpressure in RepartitionExec distribut…
20fce3a 
…or channels

Replace the global gate mechanism with per-channel bounded capacity.
Each channel independently enforces a buffer limit (CHANNEL_CAPACITY=2),
so senders targeting a full channel block without affecting senders to
other channels. This eliminates head-of-line blocking and provides
more granular backpressure control.

https://claude.ai/code/session_01MiFZEjX5FyFwJGAVjv8tJ3
@claude
Fix deadlocks in per-channel backpressure for RepartitionExec
e32e390 
Two issues caused deadlocks with per-channel backpressure:

1. wait_for_task sent completion signals to channels sequentially. If one
   channel was full, it blocked sending to all remaining channels. Fixed by
   using futures::future::join_all to send concurrently.

2. Unclaimed output partitions (when only a subset of partitions are
   executed) would fill to capacity and block senders indefinitely. Fixed by
   tracking receiver_active state - backpressure is only enforced once the
   receiver has actively consumed data. Channels with inactive receivers
   allow unbounded buffering, preventing deadlocks while still providing
   backpressure for active channels.

https://claude.ai/code/session_01MiFZEjX5FyFwJGAVjv8tJ3
@github-actions github-actions bot added the physical-plan Changes to the physical-plan crate label Mar 6, 2026
@Dandandan
Copy link
Contributor Author

Dandandan commented Mar 6, 2026

run benchmarks

@alamb-ghbot
Copy link

alamb-ghbot commented Mar 6, 2026

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing claude/backpressure-repartitionexec-JvHP9 (e32e390) to 2cbee47 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb-ghbot
Copy link

alamb-ghbot commented Mar 6, 2026

🤖: Benchmark completed

Details 

Comparing HEAD and claude_backpressure-repartitionexec-JvHP9
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃       Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │  2419.14 ms │                                2453.67 ms │    no change │
│ QQuery 1 │   985.80 ms │                                1046.52 ms │ 1.06x slower │
│ QQuery 2 │  2002.99 ms │                                1976.80 ms │    no change │
│ QQuery 3 │  1091.75 ms │                                1092.31 ms │    no change │
│ QQuery 4 │  2437.92 ms │                                2865.89 ms │ 1.18x slower │
│ QQuery 5 │ 28150.14 ms │                               32144.33 ms │ 1.14x slower │
│ QQuery 6 │  4004.74 ms │                                4033.68 ms │    no change │
│ QQuery 7 │  3087.60 ms │                                3810.55 ms │ 1.23x slower │
└──────────┴─────────────┴───────────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 44180.09ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │ 49423.74ms │
│ Average Time (HEAD)                                      │  5522.51ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │  6177.97ms │
│ Queries Faster                                           │          0 │
│ Queries Slower                                           │          4 │
│ Queries with No Change                                   │          4 │
│ Queries with Failure                                     │          0 │
└──────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃       Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.61 ms │                                   2.66 ms │    no change │
│ QQuery 1  │    52.32 ms │                                  52.30 ms │    no change │
│ QQuery 2  │   168.23 ms │                                 169.33 ms │    no change │
│ QQuery 3  │   164.95 ms │                                 181.21 ms │ 1.10x slower │
│ QQuery 4  │  1102.42 ms │                                1308.26 ms │ 1.19x slower │
│ QQuery 5  │  1342.09 ms │                                1390.58 ms │    no change │
│ QQuery 6  │     6.37 ms │                                   6.22 ms │    no change │
│ QQuery 7  │    56.58 ms │                                  56.29 ms │    no change │
│ QQuery 8  │  1508.38 ms │                                1696.39 ms │ 1.12x slower │
│ QQuery 9  │  1887.31 ms │                                1892.97 ms │    no change │
│ QQuery 10 │   364.38 ms │                                 364.94 ms │    no change │
│ QQuery 11 │   409.13 ms │                                 417.26 ms │    no change │
│ QQuery 12 │  1245.17 ms │                                1271.38 ms │    no change │
│ QQuery 13 │  2013.53 ms │                                2394.57 ms │ 1.19x slower │
│ QQuery 14 │  1250.96 ms │                                1288.81 ms │    no change │
│ QQuery 15 │  1288.95 ms │                                1457.90 ms │ 1.13x slower │
│ QQuery 16 │  2645.99 ms │                                2942.72 ms │ 1.11x slower │
│ QQuery 17 │  2618.93 ms │                                2957.90 ms │ 1.13x slower │
│ QQuery 18 │  5673.92 ms │                                5984.32 ms │ 1.05x slower │
│ QQuery 19 │   123.64 ms │                                 128.55 ms │    no change │
│ QQuery 20 │  1986.27 ms │                                1972.96 ms │    no change │
│ QQuery 21 │  2202.01 ms │                                2201.93 ms │    no change │
│ QQuery 22 │  3912.14 ms │                                3908.52 ms │    no change │
│ QQuery 23 │ 12631.95 ms │                               12462.53 ms │    no change │
│ QQuery 24 │   212.20 ms │                                 203.31 ms │    no change │
│ QQuery 25 │   449.34 ms │                                 449.01 ms │    no change │
│ QQuery 26 │   206.54 ms │                                 214.08 ms │    no change │
│ QQuery 27 │  2902.76 ms │                                2777.99 ms │    no change │
│ QQuery 28 │ 25125.24 ms │                               24910.32 ms │    no change │
│ QQuery 29 │  1032.95 ms │                                1084.93 ms │ 1.05x slower │
│ QQuery 30 │  1280.33 ms │                                1342.71 ms │    no change │
│ QQuery 31 │  1418.27 ms │                                1704.41 ms │ 1.20x slower │
│ QQuery 32 │  5043.19 ms │                                5956.46 ms │ 1.18x slower │
│ QQuery 33 │  6183.70 ms │                                6279.35 ms │    no change │
│ QQuery 34 │  6576.42 ms │                                6690.60 ms │    no change │
│ QQuery 35 │  2066.49 ms │                                2092.58 ms │    no change │
│ QQuery 36 │   188.24 ms │                                 195.43 ms │    no change │
│ QQuery 37 │    74.25 ms │                                  71.36 ms │    no change │
│ QQuery 38 │   112.20 ms │                                 113.74 ms │    no change │
│ QQuery 39 │   344.03 ms │                                 358.71 ms │    no change │
│ QQuery 40 │    38.51 ms │                                  42.76 ms │ 1.11x slower │
│ QQuery 41 │    33.91 ms │                                  35.70 ms │ 1.05x slower │
│ QQuery 42 │    31.00 ms │                                  30.62 ms │    no change │
└───────────┴─────────────┴───────────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │  97977.83ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │ 101064.55ms │
│ Average Time (HEAD)                                      │   2278.55ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │   2350.34ms │
│ Queries Faster                                           │           0 │
│ Queries Slower                                           │          13 │
│ Queries with No Change                                   │          30 │
│ Queries with Failure                                     │           0 │
└──────────────────────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 126.88 ms │                                 128.36 ms │     no change │
│ QQuery 2  │  33.95 ms │                                  34.72 ms │     no change │
│ QQuery 3  │  40.95 ms │                                  39.81 ms │     no change │
│ QQuery 4  │  34.30 ms │                                  47.61 ms │  1.39x slower │
│ QQuery 5  │  91.08 ms │                                 108.99 ms │  1.20x slower │
│ QQuery 6  │  24.73 ms │                                  24.65 ms │     no change │
│ QQuery 7  │ 159.29 ms │                                 154.72 ms │     no change │
│ QQuery 8  │  41.31 ms │                                  39.24 ms │ +1.05x faster │
│ QQuery 9  │ 108.52 ms │                                 114.97 ms │  1.06x slower │
│ QQuery 10 │  70.24 ms │                                  81.81 ms │  1.16x slower │
│ QQuery 11 │  18.92 ms │                                  19.37 ms │     no change │
│ QQuery 12 │  68.42 ms │                                  69.22 ms │     no change │
│ QQuery 13 │  55.49 ms │                                  63.84 ms │  1.15x slower │
│ QQuery 14 │  16.26 ms │                                  17.11 ms │  1.05x slower │
│ QQuery 15 │  34.21 ms │                                  34.82 ms │     no change │
│ QQuery 16 │  30.42 ms │                                  32.12 ms │  1.06x slower │
│ QQuery 17 │ 172.15 ms │                                 171.26 ms │     no change │
│ QQuery 18 │ 301.90 ms │                                 353.12 ms │  1.17x slower │
│ QQuery 19 │  53.99 ms │                                  54.54 ms │     no change │
│ QQuery 20 │  62.26 ms │                                  66.28 ms │  1.06x slower │
│ QQuery 21 │ 205.24 ms │                                 194.67 ms │ +1.05x faster │
│ QQuery 22 │  24.27 ms │                                  27.30 ms │  1.13x slower │
└───────────┴───────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 1774.78ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │ 1878.52ms │
│ Average Time (HEAD)                                      │   80.67ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │   85.39ms │
│ Queries Faster                                           │         2 │
│ Queries Slower                                           │        10 │
│ Queries with No Change                                   │        10 │
│ Queries with Failure                                     │         0 │
└──────────────────────────────────────────────────────────┴───────────┘

@Dandandan
Copy link
Contributor Author

Dandandan commented Mar 6, 2026

run benchmarks

@alamb-ghbot
Copy link

alamb-ghbot commented Mar 6, 2026

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing claude/backpressure-repartitionexec-JvHP9 (31d2da3) to 2cbee47 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb-ghbot
Copy link

alamb-ghbot commented Mar 6, 2026

🤖: Benchmark completed

Details 

Comparing HEAD and claude_backpressure-repartitionexec-JvHP9
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃    Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │  2396.04 ms │                                2473.94 ms │ no change │
│ QQuery 1 │   991.53 ms │                                1011.22 ms │ no change │
│ QQuery 2 │  1994.73 ms │                                1931.56 ms │ no change │
│ QQuery 3 │  1094.40 ms │                                1124.27 ms │ no change │
│ QQuery 4 │  2429.64 ms │                                2486.04 ms │ no change │
│ QQuery 5 │ 27944.67 ms │                               27982.31 ms │ no change │
│ QQuery 6 │  4089.57 ms │                                4044.93 ms │ no change │
│ QQuery 7 │  3125.88 ms │                                3271.31 ms │ no change │
└──────────┴─────────────┴───────────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 44066.46ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │ 44325.58ms │
│ Average Time (HEAD)                                      │  5508.31ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │  5540.70ms │
│ Queries Faster                                           │          0 │
│ Queries Slower                                           │          0 │
│ Queries with No Change                                   │          8 │
│ Queries with Failure                                     │          0 │
└──────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.58 ms │                                   2.61 ms │     no change │
│ QQuery 1  │    51.31 ms │                                  54.59 ms │  1.06x slower │
│ QQuery 2  │   164.92 ms │                                 167.83 ms │     no change │
│ QQuery 3  │   175.47 ms │                                 174.87 ms │     no change │
│ QQuery 4  │  1087.16 ms │                                1235.92 ms │  1.14x slower │
│ QQuery 5  │  1364.18 ms │                                1374.03 ms │     no change │
│ QQuery 6  │     7.22 ms │                                   6.52 ms │ +1.11x faster │
│ QQuery 7  │    55.61 ms │                                  56.69 ms │     no change │
│ QQuery 8  │  1537.00 ms │                                1688.13 ms │  1.10x slower │
│ QQuery 9  │  1883.77 ms │                                1979.45 ms │  1.05x slower │
│ QQuery 10 │   361.15 ms │                                 362.45 ms │     no change │
│ QQuery 11 │   409.70 ms │                                 422.05 ms │     no change │
│ QQuery 12 │  1254.10 ms │                                1324.71 ms │  1.06x slower │
│ QQuery 13 │  2090.16 ms │                                2117.73 ms │     no change │
│ QQuery 14 │  1273.47 ms │                                1292.70 ms │     no change │
│ QQuery 15 │  1288.72 ms │                                1409.13 ms │  1.09x slower │
│ QQuery 16 │  2747.77 ms │                                2875.99 ms │     no change │
│ QQuery 17 │  2709.29 ms │                                2825.60 ms │     no change │
│ QQuery 18 │  6073.88 ms │                                5451.72 ms │ +1.11x faster │
│ QQuery 19 │   131.53 ms │                                 130.07 ms │     no change │
│ QQuery 20 │  1987.61 ms │                                1934.38 ms │     no change │
│ QQuery 21 │  2324.77 ms │                                2264.76 ms │     no change │
│ QQuery 22 │  4629.19 ms │                                3911.80 ms │ +1.18x faster │
│ QQuery 23 │ 15457.87 ms │                               12591.30 ms │ +1.23x faster │
│ QQuery 24 │   209.00 ms │                                 204.23 ms │     no change │
│ QQuery 25 │   464.72 ms │                                 462.20 ms │     no change │
│ QQuery 26 │   206.92 ms │                                 199.57 ms │     no change │
│ QQuery 27 │  2949.22 ms │                                2829.14 ms │     no change │
│ QQuery 28 │ 24896.92 ms │                               24714.84 ms │     no change │
│ QQuery 29 │  1037.35 ms │                                1082.62 ms │     no change │
│ QQuery 30 │  1290.79 ms │                                1342.30 ms │     no change │
│ QQuery 31 │  1414.81 ms │                                1453.93 ms │     no change │
│ QQuery 32 │  4987.52 ms │                                5101.98 ms │     no change │
│ QQuery 33 │  6219.50 ms │                                5907.37 ms │ +1.05x faster │
│ QQuery 34 │  6574.46 ms │                                7034.84 ms │  1.07x slower │
│ QQuery 35 │  2038.29 ms │                                2121.97 ms │     no change │
│ QQuery 36 │   195.49 ms │                                 182.24 ms │ +1.07x faster │
│ QQuery 37 │    72.24 ms │                                  77.56 ms │  1.07x slower │
│ QQuery 38 │   113.10 ms │                                 114.18 ms │     no change │
│ QQuery 39 │   351.36 ms │                                 343.24 ms │     no change │
│ QQuery 40 │    41.49 ms │                                  44.23 ms │  1.07x slower │
│ QQuery 41 │    35.28 ms │                                  39.32 ms │  1.11x slower │
│ QQuery 42 │    31.08 ms │                                  32.07 ms │     no change │
└───────────┴─────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 102197.95ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │  98942.83ms │
│ Average Time (HEAD)                                      │   2376.70ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │   2301.00ms │
│ Queries Faster                                           │           6 │
│ Queries Slower                                           │          10 │
│ Queries with No Change                                   │          27 │
│ Queries with Failure                                     │           0 │
└──────────────────────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 127.00 ms │                                 129.03 ms │     no change │
│ QQuery 2  │  33.67 ms │                                  33.73 ms │     no change │
│ QQuery 3  │  42.64 ms │                                  39.94 ms │ +1.07x faster │
│ QQuery 4  │  36.39 ms │                                  35.48 ms │     no change │
│ QQuery 5  │  90.41 ms │                                  91.20 ms │     no change │
│ QQuery 6  │  25.25 ms │                                  25.25 ms │     no change │
│ QQuery 7  │ 152.17 ms │                                 143.67 ms │ +1.06x faster │
│ QQuery 8  │  40.26 ms │                                  39.33 ms │     no change │
│ QQuery 9  │ 107.87 ms │                                 111.27 ms │     no change │
│ QQuery 10 │  73.43 ms │                                  72.94 ms │     no change │
│ QQuery 11 │  18.86 ms │                                  19.22 ms │     no change │
│ QQuery 12 │  68.09 ms │                                  63.93 ms │ +1.07x faster │
│ QQuery 13 │  54.73 ms │                                  54.09 ms │     no change │
│ QQuery 14 │  15.83 ms │                                  16.87 ms │  1.07x slower │
│ QQuery 15 │  34.14 ms │                                  33.48 ms │     no change │
│ QQuery 16 │  30.31 ms │                                  31.42 ms │     no change │
│ QQuery 17 │ 168.34 ms │                                 172.09 ms │     no change │
│ QQuery 18 │ 289.84 ms │                                 311.02 ms │  1.07x slower │
│ QQuery 19 │  54.71 ms │                                  57.15 ms │     no change │
│ QQuery 20 │  61.33 ms │                                  62.64 ms │     no change │
│ QQuery 21 │ 201.02 ms │                                 188.53 ms │ +1.07x faster │
│ QQuery 22 │  25.98 ms │                                  24.97 ms │     no change │
└───────────┴───────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 1752.30ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │ 1757.25ms │
│ Average Time (HEAD)                                      │   79.65ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │   79.88ms │
│ Queries Faster                                           │         4 │
│ Queries Slower                                           │         2 │
│ Queries with No Change                                   │        16 │
│ Queries with Failure                                     │         0 │
└──────────────────────────────────────────────────────────┴───────────┘

@Dandandan
Increase CHANNEL_CAPACITY to 16
97e8c5a 
Increased the channel capacity from 8 to 16 to improve throughput.
@Dandandan
Copy link
Contributor Author

Dandandan commented Mar 7, 2026

run benchmarks

@alamb-ghbot
Copy link

alamb-ghbot commented Mar 7, 2026

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing claude/backpressure-repartitionexec-JvHP9 (97e8c5a) to 2cbee47 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb-ghbot
Copy link

alamb-ghbot commented Mar 7, 2026

🤖: Benchmark completed

Details 

Comparing HEAD and claude_backpressure-repartitionexec-JvHP9
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃       Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │  2417.77 ms │                                2344.56 ms │    no change │
│ QQuery 1 │   991.80 ms │                                1009.67 ms │    no change │
│ QQuery 2 │  2013.92 ms │                                1983.37 ms │    no change │
│ QQuery 3 │  1126.85 ms │                                1127.92 ms │    no change │
│ QQuery 4 │  2423.01 ms │                                2384.96 ms │    no change │
│ QQuery 5 │ 28297.76 ms │                               28031.42 ms │    no change │
│ QQuery 6 │  4058.31 ms │                                4038.00 ms │    no change │
│ QQuery 7 │  3072.28 ms │                                3232.88 ms │ 1.05x slower │
└──────────┴─────────────┴───────────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 44401.71ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │ 44152.78ms │
│ Average Time (HEAD)                                      │  5550.21ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │  5519.10ms │
│ Queries Faster                                           │          0 │
│ Queries Slower                                           │          1 │
│ Queries with No Change                                   │          7 │
│ Queries with Failure                                     │          0 │
└──────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.56 ms │                                   2.61 ms │     no change │
│ QQuery 1  │    51.13 ms │                                  58.43 ms │  1.14x slower │
│ QQuery 2  │   163.75 ms │                                 187.97 ms │  1.15x slower │
│ QQuery 3  │   169.79 ms │                                 190.88 ms │  1.12x slower │
│ QQuery 4  │  1042.79 ms │                                1201.65 ms │  1.15x slower │
│ QQuery 5  │  1321.27 ms │                                1393.65 ms │  1.05x slower │
│ QQuery 6  │     6.59 ms │                                   6.13 ms │ +1.08x faster │
│ QQuery 7  │    53.69 ms │                                  57.14 ms │  1.06x slower │
│ QQuery 8  │  1506.81 ms │                                1602.97 ms │  1.06x slower │
│ QQuery 9  │  1894.73 ms │                                1995.94 ms │  1.05x slower │
│ QQuery 10 │   359.26 ms │                                 360.50 ms │     no change │
│ QQuery 11 │   407.69 ms │                                 413.85 ms │     no change │
│ QQuery 12 │  1215.14 ms │                                1274.15 ms │     no change │
│ QQuery 13 │  2059.92 ms │                                2104.14 ms │     no change │
│ QQuery 14 │  1273.38 ms │                                1280.93 ms │     no change │
│ QQuery 15 │  1286.88 ms │                                1346.30 ms │     no change │
│ QQuery 16 │  2629.34 ms │                                2703.33 ms │     no change │
│ QQuery 17 │  2633.96 ms │                                2718.21 ms │     no change │
│ QQuery 18 │  6169.57 ms │                                5232.67 ms │ +1.18x faster │
│ QQuery 19 │   128.76 ms │                                 127.78 ms │     no change │
│ QQuery 20 │  2013.14 ms │                                1943.13 ms │     no change │
│ QQuery 21 │  2326.58 ms │                                2209.18 ms │ +1.05x faster │
│ QQuery 22 │  6446.99 ms │                                3932.12 ms │ +1.64x faster │
│ QQuery 23 │ 21445.36 ms │                               12865.25 ms │ +1.67x faster │
│ QQuery 24 │   196.42 ms │                                 205.76 ms │     no change │
│ QQuery 25 │   469.60 ms │                                 466.40 ms │     no change │
│ QQuery 26 │   201.11 ms │                                 207.51 ms │     no change │
│ QQuery 27 │  2904.97 ms │                                2948.60 ms │     no change │
│ QQuery 28 │ 25470.68 ms │                               25377.57 ms │     no change │
│ QQuery 29 │  1050.27 ms │                                1043.95 ms │     no change │
│ QQuery 30 │  1303.37 ms │                                1310.43 ms │     no change │
│ QQuery 31 │  1422.29 ms │                                1420.00 ms │     no change │
│ QQuery 32 │  4942.51 ms │                                5162.54 ms │     no change │
│ QQuery 33 │  6144.27 ms │                                6076.29 ms │     no change │
│ QQuery 34 │  6293.84 ms │                                6491.51 ms │     no change │
│ QQuery 35 │  2027.43 ms │                                2029.83 ms │     no change │
│ QQuery 36 │   194.53 ms │                                 180.93 ms │ +1.08x faster │
│ QQuery 37 │    75.84 ms │                                  76.83 ms │     no change │
│ QQuery 38 │   114.61 ms │                                 114.31 ms │     no change │
│ QQuery 39 │   350.08 ms │                                 346.26 ms │     no change │
│ QQuery 40 │    40.54 ms │                                  38.22 ms │ +1.06x faster │
│ QQuery 41 │    35.57 ms │                                  34.09 ms │     no change │
│ QQuery 42 │    32.21 ms │                                  31.55 ms │     no change │
└───────────┴─────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 109879.28ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │  98771.46ms │
│ Average Time (HEAD)                                      │   2555.33ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │   2297.01ms │
│ Queries Faster                                           │           7 │
│ Queries Slower                                           │           8 │
│ Queries with No Change                                   │          28 │
│ Queries with Failure                                     │           0 │
└──────────────────────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ claude_backpressure-repartitionexec-JvHP9 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 127.11 ms │                                 124.39 ms │     no change │
│ QQuery 2  │  32.76 ms │                                  32.62 ms │     no change │
│ QQuery 3  │  36.23 ms │                                  39.15 ms │  1.08x slower │
│ QQuery 4  │  34.28 ms │                                  33.91 ms │     no change │
│ QQuery 5  │  91.82 ms │                                  85.35 ms │ +1.08x faster │
│ QQuery 6  │  24.46 ms │                                  24.46 ms │     no change │
│ QQuery 7  │ 151.49 ms │                                 163.08 ms │  1.08x slower │
│ QQuery 8  │  40.80 ms │                                  41.49 ms │     no change │
│ QQuery 9  │ 107.59 ms │                                 106.74 ms │     no change │
│ QQuery 10 │  72.54 ms │                                  77.01 ms │  1.06x slower │
│ QQuery 11 │  19.26 ms │                                  19.34 ms │     no change │
│ QQuery 12 │  66.71 ms │                                  73.11 ms │  1.10x slower │
│ QQuery 13 │  53.56 ms │                                  62.46 ms │  1.17x slower │
│ QQuery 14 │  15.38 ms │                                  17.27 ms │  1.12x slower │
│ QQuery 15 │  33.35 ms │                                  38.25 ms │  1.15x slower │
│ QQuery 16 │  29.91 ms │                                  32.34 ms │  1.08x slower │
│ QQuery 17 │ 166.42 ms │                                 181.22 ms │  1.09x slower │
│ QQuery 18 │ 295.70 ms │                                 308.22 ms │     no change │
│ QQuery 19 │  53.81 ms │                                  54.71 ms │     no change │
│ QQuery 20 │  61.62 ms │                                  61.13 ms │     no change │
│ QQuery 21 │ 194.16 ms │                                 187.79 ms │     no change │
│ QQuery 22 │  24.26 ms │                                  24.38 ms │     no change │
└───────────┴───────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                        │ 1733.22ms │
│ Total Time (claude_backpressure-repartitionexec-JvHP9)   │ 1788.43ms │
│ Average Time (HEAD)                                      │   78.78ms │
│ Average Time (claude_backpressure-repartitionexec-JvHP9) │   81.29ms │
│ Queries Faster                                           │         1 │
│ Queries Slower                                           │         9 │
│ Queries with No Change                                   │        12 │
│ Queries with Failure                                     │         0 │
└──────────────────────────────────────────────────────────┴───────────┘

@Dandandan Dandandan closed this Mar 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Dandandan @alamb-ghbot @claude