Skip to content

Fix zero GPU case when GPUs assigned to worker - #1398

Merged
shuds13 merged 10 commits into
developfrom
bugfix/gpu_fix
Aug 13, 2024
Merged

shuds13 merged 10 commits into
developfrom
bugfix/gpu_fix

Conversation

@shuds13

@shuds13 shuds13 commented Aug 9, 2024 •

Copy link
Copy Markdown
Member

When GPUs have been assigned to a worker, and num_gpus=0 is sent, it was assigning all available GPUs.

  • Check single node
  • Check multiple nodes

This PR also changes some scenarios where resources are under-utilized for a submit, such that the given configuration will be fit to the minimum nodes required.

E.g., if a worker has been assigned 2 nodes but gets a run configuration of num_procs=2 and num_gpus=2, and this can be accommodated on one node, then only one node will be used, rather than splitting over two.

@codecov

codecov Bot commented Aug 10, 2024 •

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.61%. Comparing base (eefb0af) to head (8783c82).
Report is 107 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1398      +/-   ##
===========================================
+ Coverage    76.51%   76.61%   +0.09%     
===========================================
  Files           88       88              
  Lines         8019     8055      +36     
  Branches      1435     1442       +7     
===========================================
+ Hits          6136     6171      +35     
- Misses        1665     1666       +1     
  Partials       218      218              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@shuds13 shuds13 mentioned this pull request Aug 12, 2024
5 tasks done
@shuds13
shuds13 requested review from jlnav and jmlarson1 and removed request for jlnav August 12, 2024 20:02
@n01r

n01r commented Aug 13, 2024

Copy link
Copy Markdown

I tested this PR with a scaled-down version of my setup, and it fixed the issue. Thanks a bunch, @shuds13! 🚀

@shuds13
shuds13 merged commit 36fb0e3 into develop Aug 13, 2024
@jmlarson1
jmlarson1 deleted the bugfix/gpu_fix branch August 15, 2024 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

4 participants