Skip to content

Reuse in-flight Metal buffers on the same stream - #4852

Open
XXXXRT666 wants to merge 3 commits into
ml-explore:mainfrom
XXXXRT666:exp/metal-stream-buffer-reuse
Open

XXXXRT666 wants to merge 3 commits into
ml-explore:mainfrom
XXXXRT666:exp/metal-stream-buffer-reuse

Conversation

@XXXXRT666

Copy link
Copy Markdown
Contributor
  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: Benchmarking and Testing

Summary

Track inflight Metal buffer use separately from array ownership. Reuse released storage on the sole stream still using it, with fences for earlier reads and writes and allocator checks for host writes and donation.

Benchmarks

Values are medians over 48 trials per build for small models, 6 for Qwen3 and Parakeet, and 8 for vision and Qwen3-ASR.

Extra active peak memory does not include model footprint.

Workload Wall time Total active peak (MB) Extra active peak (MB)
Small Transformer encoder, 512 tokens 8.305 → 8.073 56.715 → 12.652 49.500 → 5.438 (−89.0%)
Small LM prefill, 384 tokens 4.617 → 4.576 34.954 → 13.470 28.291 → 6.808 (−75.9%)
Small LM decode, 16 steps 13.786 → 12.920 13.978 → 12.201 3.941 → 2.165 (−45.1%)
Small residual CNN, 160 × 160 1.021 → 1.016 3.245 → 2.635 2.272 → 1.662 (−26.8%)
Small U-Net, 128 × 128 1.215 → 1.178 8.251 → 5.501 7.875 → 5.125 (−34.9%)
Two-layer GRU, 128 steps 14.571 → 12.682 0.678 → 0.566 0.286 → 0.174 (−39.3%)
Qwen3-0.6B prefill, 2,048 tokens 820.45 → 810.14 1,868.88 → 1,404.88 732 → 268 (−63.4%)
Qwen3-0.6B prefill, 4,096 tokens 1,863.32 → 1,860.83 2,112.89 → 1,672.89 976 → 536 (−45.1%)
Qwen3-0.6B prefill, 6,144 tokens 3,338.36 → 3,353.98 2,348.91 → 1,940.91 1,212 → 804 (−33.7%)
Qwen3-0.6B decode, 8 tokens 149.95 → 154.42 1,630.71 → 1,614.20 269.83 → 253.33 (−6.1%)
Parakeet encoder, 33 s 221.03 → 222.37 1,656.07 → 1,341.83 419.58 → 105.06 (−75.0%)
ResNet-50, batch 1 7.903 → 8.028 72.89 → 57.31 23.71 → 8.12 (−65.7%)
ResNet-50, batch 32 175.34 → 169.00 768.41 → 322.53 710.34 → 264.45 (−62.8%)
ViT-L/16, batch 1 59.00 → 58.58 814.81 → 587.38 234.04 → 6.61 (−97.2%)
ViT-L/16, batch 16 745.47 → 766.32 1,222.04 → 671.70 636.97 → 86.64 (−86.4%)
Qwen3-ASR encoder, 33 s 169.03 → 168.82 2,483.54 → 1,841.06 982.18 → 339.70 (−65.4%)
Qwen3-ASR transcription, 33 s 1,700.34 → 1,627.52 2,483.67 → 1,841.07 985.94 → 343.35 (−65.2%)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant