[arpit974]

feat(slurm): MIG Phase 3 — Flexible MIGs (Machine-Type Fallback & Multi-Zone Scavenging) for Static Nodes

Overview

This PR implements MIG Phase 3, adding support for Flexible Managed Instance Groups (instance_flexibility_policy) and Multi-Zone Capacity Scavenging & Cross-Zone Auto-Repair (zone_target_shape) on static Slurm MIG NodeSets (provisioning_engine = "MIG", node_count_dynamic_max = 0), along with lifecycle hardening for MIG auto-repair in slurmsync.py and teardown in cleanup_compute.sh.

---

Key Changes

1. Ranked Machine-Type Fallback (instance_flexibility_policy)

• Schema & Plumbing: Exposes instance_flexibility_policy.instance_selections (rank, machine_types, and optional name defaulting to "selection-<idx+1>") across schedmd-slurm-gcp-v6-nodeset, schedmd-slurm-gcp-v6-partition, and schedmd-slurm-gcp-v6-controller (partition.tf).
• Primary Shape Auto-Inclusion: Automatically prepends a rank-0 "primary-default" selection containing var.machine_type when var.machine_type is not already listed in instance_selections.
• MIG Resource Wiring: Emits instance_flexibility_policy and sets instance_lifecycle_policy.force_update_on_repair = "YES" on google_compute_region_instance_group_manager.nodeset_mig.

2. Slurm Resource Flooring (cloud.conf) & Blueprint Machine Configs (pkg/config)

• Preventing Slurm DRAIN on Smaller Fallback VMs: Lookup.nodeset_machine_conf() in util.py (used by conf.py) computes the floor across var.machine_type and all candidate fallback machine_types for cpus, memory, sockets, and cores_per_socket (including heterogeneous SMT / non-SMT families like n4a, c4a, t2d). If any fallback shape cannot be resolved, it fails loudly with a RuntimeError rather than emitting an unfloored cloud.conf.
• Blueprint Expansion (pkg/config/machine_configs.go): Extends GetOutputConfig to query GCE MachineTypes.Get for all fallback shapes in instance_flexibility_policy and merge their CPU/GPU entries into machine_configs. Skips 404 Not Found (when a fallback shape is only available in another zone of the region) while failing fast on non-404 errors (403, 429, network failures).

3. Multi-Zone Capacity Scavenging & Cross-Zone Auto-Repair (zone_target_shape)

• Multi-Zone Distribution: Wires zone_target_shape (ANY_SINGLE_ZONE default, ANY, BALANCED) into distribution_policy_target_shape on nodeset_mig. When a multi-zone shape (ANY or BALANCED) is set on a static MIG without explicit zones or reservations, zones_allow automatically expands to all available zones in the region.
• Cross-Zone Auto-Repair: Sets on_repair.allow_changing_zone = "YES" and force_update_on_repair = "YES" when zone_target_shape != "ANY_SINGLE_ZONE", allowing preempted or failed VMs to be recreated in another zone with available capacity.
• DWS Flex Multi-Zone Selection: Removes the legacy restriction blocking zones with dws_flex.enabled = true, while enforcing zone_target_shape == "ANY_SINGLE_ZONE" for DWS Flex MIGs (mig_flex.py, metadata.yaml, outputs.tf) because GCE resizeRequests require ANY_SINGLE_ZONE.

4. MIG Lifecycle & Teardown Hardening (slurmsync.py & cleanup_compute.sh)

• slurmsync.py Auto-Repair Alignment: When a MIG-backed VM enters TERMINATED, slurmsync.py marks the Slurm node DOWN (Reason="MIG Auto-Healing instance repair in progress") and waits for GCE MIG auto-repair (delete + insert) instead of calling start_instances.
• cleanup_compute.sh Destroy Race Fix: Skips Terraform-managed <cluster>-<nodeset>-mig-<N> MIGs and their member VMs (metadata.items.created-by.basename()) so cleanup_compute.sh does not race google_compute_region_instance_group_manager.nodeset_mig deletion during terraform destroy.

5. Fail-Fast Guardrails (GCE Flex MIG Platform Constraints)

Enforced at both gcluster create (metadata.yaml) and terraform plan (outputs.tf / variables.tf):

• Requires provisioning_engine = "MIG" and node_count_dynamic_max = 0.
• Rejects instance_flexibility_policy with dws_flex.enabled = true, accelerator_topology, specific reservations (reservation_name / future_reservation), or a3-ultragpu / a4 / a4x families (unsupported by GCE's instanceFlexibilityPolicy API).
• Enforces matching CPU architecture (x86_64 vs Arm64), matching attached GPU model/count, disk-type compatibility across all fallback shapes, unique selection names, and GCE's limit of <= 10 distinct machine types.
• Enforces zone_target_shape == "ANY_SINGLE_ZONE" when enable_placement = true or accelerator_topology is configured on a MIG.

---

Testing & Verification Matrix

1. Unit Tests & Static Pre-Flight Analysis

• Pytest: 245 passed (test_util.py, test_resume.py, test_suspend.py, test_conf.py, test_repair.py, test_topology.py).
• Mypy: 0 issues found across 28 source files (util.py, slurmsync.py, mig_flex.py, conf.py).
• Go Unit Tests: Passed across ./pkg/config/... and ./pkg/validators/... (including TestExtractFallbackMachineTypes, TestAddMachineTypeToOutputConfig_MergesFallback, and TestGetOutputConfig_FallbackLookupErrors).
• Pre-Commit Suite: All 22 repository pre-commit hooks passed (terraform_fmt, terraform_tflint, terraform_validate, terraform-readme, go-cyclo, golangci-lint, shfmt, shellcheck, yamllint, PyMarkdown, codespell).

2. 66-Case Terraform Plan & Validation Matrix

• Phase 3 Positive & Negative Permutations: Verified instance_flexibility_policy rank ordering, auto-prepended primary-default, ANY / BALANCED multi-zone expansion, allow_changing_zone = "YES", DWS Flex with zones, and rejection of all invalid combinations (specific reservations, a3-ultragpu/a4/a4x, mismatched GPU/arch/disk types, >10 shapes).
• Phase 1 & Phase 2 Non-Regression: Verified all existing BULK_INSERT, AUTO, and Phase 2 a4x 1x72 slice MIG plan cases pass unchanged.

3. Live GCP End-to-End Verification & Teardown Audit

• Live Cluster Deployment: Deployed multi-partition Slurm cluster in us-central1 with BALANCED multi-zone Flex MIGs (n2-standard-4 rank 1, n2d-standard-2 rank 2), BULK_INSERT nodesets, and multi-zone DWS Flex nodesets.
• Runtime Verification:
  • Confirmed BALANCED multi-zone instance distribution across us-central1-a and us-central1-b.
  • Confirmed Slurm cloud.conf floored node definitions (CPUs=1 RealMemory=7552) and clean srun job execution across all partitions.
  • Triggered live VM stop on a running MIG node; verified slurmsync.py marked the node DOWN without calling start_instances, GCE MIG auto-repaired the VM, and slurmsync.py automatically restored the node to IDLE in ~28s.
• Teardown Audit: Executed gcluster destroy (52 destroyed); verified 0 residual MIGs, 0 instances, and 0 resource policies.

---

Impact & Backward Compatibility

• Zero Regression on Default & BULK_INSERT Workloads: instance_flexibility_policy defaults to null and zone_target_shape defaults to "ANY_SINGLE_ZONE". Standard AUTO and BULK_INSERT nodesets behave identically to prior releases.
• Phase 1 & Phase 2 MIG Compatibility: Existing static MIG and A4X 1x72 slice MIG blueprints require no configuration changes.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds support for machine-type fallback (flexibility policy) for static MIG NodeSets in the Slurm on GCP toolkit. By allowing users to specify a ranked list of machine types, the system can automatically fall back to alternative shapes when primary capacity is unavailable. The implementation includes robust validation to prevent incompatible configurations and updates to the Slurm configuration generation to maintain consistency across the different machine shapes.

Highlights

• Instance Flexibility Policy: Introduced the instance_flexibility_policy variable, allowing users to define a ranked list of fallback machine types for static MIG NodeSets.
• Configuration Validation: Added extensive validation logic in Terraform and metadata to ensure the flexibility policy is correctly configured and compatible with existing features.
• Slurm Configuration Logic: Updated Slurm configuration scripts to handle fallback machine types, ensuring node geometry is floored to the smallest shape across all selections.
• Machine Config Expansion: Enhanced the Go-based machine configuration logic to support fallback machine type expansion in output configurations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for instance_flexibility_policy (machine-type fallback) in MIG NodeSets across the Slurm GCP v6 modules (nodeset, partition, and controller). It adds the necessary Terraform variables, outputs, and validation rules, updates the Python configuration scripts to floor node definitions to the smallest shape in the flexibility policy, and extends the Go configuration package to parse and include fallback machine types. As there are no review comments provided, I have no additional feedback to offer.

[arpit974]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces an opt-in machine-type fallback policy (instance_flexibility_policy) for MIG NodeSets across the Slurm GCP v6 modules, allowing MIGs to fall back to alternative machine types when capacity is unavailable. The changes span Terraform configurations, Go configuration generation, and Python controller scripts, including new validation rules and unit tests. Feedback on the changes suggests: (1) refining error handling in pkg/config/machine_configs.go to avoid silently ignoring non-404 errors from GetMachineType, (2) avoiding autovivification on cached NSDict templates in util.py by using .get(), and (3) ensuring consistent defensive handling of the nodeset variable in mig_flex.py to prevent potential AttributeError crashes.

[AdarshK15]

Few other Edge cases that might arise:

1. Two issues when mixing SMT (threads_per_core=2) and non-SMT (threads_per_core=1) machine types in the same nodeset:

• Mixing t2d-standard-16 (16 cores, 1 thread/core) and n2d-standard-16 (8 cores, 2 threads/core) writes CPUs=16 CoresPerSocket=8 ThreadsPerCore=2 to cloud.conf. When a t2d-standard-16 VM boots with 1 thread/core, Slurm's cgroup core pinning maps jobs assuming 2 threads/core, causing adjacent jobs to overlap on CPUs 1..7 while leaving CPUs 9..15 unused.
• At lines 2365–2371, multiplying min_total_cores from an SMT shape by threads_per_core=1 from a non-SMT shape under-counts CPUs. For example, mixing t2d-standard-16 (16 vCPUs, 16 cores, tpc=1) with c3-standard-22 (22 vCPUs, 11 cores, tpc=2) sets cpus = 11 * 1 = 11, even though both shapes have at least 16 vCPUs.

2. Two small issues when var.machine_type isn't listed in instance_selections and main.tf:44–48 auto-adds primary-default with rank = 0:

• Line 751 allows rank >= 0, so if a fallback selection also uses rank = 0, it ties with primary-default and GCE won't prioritize var.machine_type first.
• Line 780 checks <= 10 machine types across instance_selections, so passing 10 fallback types without var.machine_type passes here and then fails outputs.tf:261 once primary-default is added as the 11th type.

3. In GCE Regional MIGs, fallback VMs still inherit visibleCoreCount from the instance template. Passing None for fallback shapes at line 2346 ignores visible_cores and uses the full guest_cpus count instead, which can configure cloud.conf with more CPUs than a fallback VM actually boots with and cause Slurm to drain the node.
4. template_machine_conf still calls getThreadsPerCore(template) at line 2317, which accesses template.advancedMachineFeatures.threadsPerCore directly at line 1581 when supports_smt is True. For SMT-capable machine types (n2, c2, c3, etc.), this still mutates the cached template when advancedMachineFeatures is missing and fails with AttributeError when advancedMachineFeatures is None.
5. test_machine_type_in_zone only checks machine_type, and Terraform doesn't validate fallback machine types against the Compute API during plan. Skipping 404s here means a typo in instance_selections won't be caught by ./ghpc create or terraform plan and will only fail during terraform apply.
   Also, if a valid fallback GPU machine type only exists in another zone in zones (not the primary zone), skipping it here leaves its GPU info out of machine_configs, which can cause the GPU validation in outputs.tf:237 to fail.

[arpit974]

Replying to above top level comment:

1. Mixing SMT (threads_per_core=2) and non-SMT (threads_per_core=1) machine types: Addressed.
   • Added check_flex_smt_consistency in outputs.tf to reject mixing SMT-capable (2 threads/core) and non-SMT (1 thread/core) machine families in instance_flexibility_policy.
   • Updated nodeset_machine_conf() in util.py to compute cpus = min(c["cpus"] for c in confs) directly across shapes rather than min_total_cores * threads_per_core.
2. var.machine_type omitted from instance_selections (primary-default at rank = 0):
   • Rank 0 tie (2a): Addressed. Added check_flex_rank0_primary_consistency in outputs.tf so that if any selection explicitly uses rank = 0, var.machine_type must be included in a rank = 0 selection (preventing an unintended tie with the auto-added primary-default at rank = 0).
   • <= 10 machine types check in variables.tf:L780 vs outputs.tf (2b): Kept as-is because Terraform variable validation blocks cannot reference another variable (var.machine_type). variables.tf:L780 catches > 10 shapes in instance_selections early, while outputs.tf (length(setunion([var.machine_type], local.flex_machine_types)) <= 10) catches the 10-fallback + 1-primary (11 total) case during terraform validate / terraform plan.
3. visibleCoreCount inheritance on fallback shapes: Addressed. Updated nodeset_machine_conf() in util.py to clamp visible_cores per fallback shape via min(visible_cores, mt_cores) (where mt_cores = mt.guest_cpus // (2 if mt.supports_smt else 1)) instead of passing None.
4. getThreadsPerCore(template) in util.py:L1578-L1581: Left untouched to avoid scope creep—getThreadsPerCore and template_machine_conf are pre-existing helpers on develop, and our new nodeset_machine_conf() path reads amf = template.get("advancedMachineFeatures") directly without calling getThreadsPerCore.
5. pkg/config/machine_configs.go 404 handling & multi-zone lookup:
   • PopulateMachineConfigs only pre-fetches hardware metadata for machine_configs, whereas live zone/machine-type existence validation lives in pkg/validators/cloud.go (test_machine_type_in_zone).
   • If a valid fallback GPU machine type only exists in a secondary zone of zones (not the primary zone) and is skipped on 404 during expansion, main.tf:L83-L89 provides the regex fallback in inferred_gpu_signature so outputs.tf GPU validation still succeeds across multi-zone Regional MIGs.

[arpit974]

Verification & Test Summary

1. Unit Tests & Static Validation

• Python Controller Unit Tests (pytest): 323 passed (0 failed) across community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/scripts/tests/, including unit tests for:
  • nodeset_machine_conf() minimum-shape flooring (RealMemory, CPUs, Boards, SocketsPerBoard, CoresPerSocket, ThreadsPerCore) across instance_flexibility_policy selections.
  • visibleCoreCount clamping per fallback shape (min(visible_cores, mt_cores)).
  • Non-SMT (tpc=1) and SMT (tpc=2) handling, explicit threadsPerCore=1 override, and fail-fast ValueError when all shapes are unresolvable.
  • DWS Flex mig_flex.py zone_target_shape propagation (ANY_SINGLE_ZONE, ANY, and BALANCED $\rightarrow$ ANY clamping).
• Go Unit Tests (go test): PASS across ./pkg/config/..., ./pkg/validators/..., and ./pkg/modulereader/... (including PopulateMachineConfigs expansion for instance_flexibility_policy fallback machine types, 404 handling for zone-specific shapes, and propagation of non-404 API errors).
• Type Checking & Formatting:
  • mypy --ignore-missing-imports on controller scripts: Success: no issues found in 29 source files.
  • terraform fmt -check -recursive & terraform validate across schedmd-slurm-gcp-v6-nodeset, schedmd-slurm-gcp-v6-partition, and schedmd-slurm-gcp-v6-controller: PASS.

---

2. Terraform Plan & Negative Guardrail Matrix (68/68 PASS)

Validated 68 blueprint permutations through ./gcluster create and terraform plan:

• Zero-Regression Baseline (5/5 PASS):
  • Default provisioning_engine omitted (AUTO), explicit AUTO, explicit BULK_INSERT (static & dynamic), standard static MIG, DWS Flex (enable_dynamic_workload_scheduler = true), and GPU compact-placement topology MIG (a4x-highgpu-4g, max_distance = 2, mig_resize_requests_enabled = true).
• Flex MIG & zone_target_shape Capabilities (25/25 PASS):
  • Multi-rank fallback ladders (rank 0, rank 1, rank 2) and multi-machine-type ranks.
  • Auto-injected primary-default (rank = 0) when machine_type is omitted from instance_selections, vs. explicit rank = 0 override containing machine_type.
  • zone_target_shape permutations (BALANCED, ANY, ANY_SINGLE_ZONE) across single-zone and multi-zone Regional MIGs.
  • instance_flexibility_policy combined with enable_spot_vm = true (instance_termination_action = "DELETE").
  • Homogeneous GPU fallback configurations (matching accelerator type and count across primary and fallback shapes).
• Negative Fail-Fast Guardrails (38/38 PASS):
  • Rejected at terraform plan with expected check_flex_compatibility / variable validation errors:
    • Heterogeneous vCPU counts across primary and fallback shapes.
    • Mixing SMT-capable (2 threads/core, e.g., n2-standard-16) and non-SMT (1 thread/core, e.g., t2d-standard-16) families in instance_flexibility_policy.
    • Explicit rank = 0 fallback selection without var.machine_type included at rank = 0 (preventing rank-0 tie with primary-default).
    • GPU count/type mismatch across primary and fallback shapes.
    • Exceeding 10 distinct machine types (10 fallbacks + 1 primary = 11 shapes).
    • Invalid rank keys (negative, leading zeros, non-numeric) or >10 ranks.
    • Incompatible feature combinations with instance_flexibility_policy: BULK_INSERT, node_count_dynamic_max > 0, enable_placement = true, enable_dynamic_workload_scheduler = true, future_reservation, enable_maintenance_reservation, accelerator_topology, and unsupported <machine_family>:<disk_type> combinations (e.g., n4/c4/h4d with pd-* disks).
    • zone_target_shape set on BULK_INSERT or dynamic non-DWS nodesets, or ANY_SINGLE_ZONE without enable_spot_vm or instance_flexibility_policy.

---

3. Live GCE End-to-End Cluster Verification (HA Multi-Engine Cluster)

Deployed a full HA Slurm cluster (137 resources: Regional HA Controller MIG, Cloud SQL, 2× Filestore instances, Login VM, 9 NodeSets, and 9 Partitions) exercising all provisioning engines and Flex MIG modes concurrently:

Partition	Provisioning Engine / Mode	Configuration	Live Verification Result
mig	MIG (BALANCED)	2 static (n2-standard-2)	Booted idle; srun -N 2 -p mig PASS
migzone	MIG (ANY_SINGLE_ZONE)	2 static + enable_spot_vm	Co-located in single zone; srun -N 2 -p migzone PASS
flex	MIG + instance_flexibility_policy (BALANCED)	3 static; Rank 0: [n2-standard-2], Rank 1: [n2d-standard-2, e2-standard-2], Rank 2: [n1-standard-2]	Organic GCE zone stockout triggered live fallback (2× n2-standard-2 + 1× n1-standard-2); cloud.conf floored RealMemory=7070 CPUs=1; all 3 registered idle (0 Low RealMemory drain) and ran srun -N 3 -p flex PASS
flexzone	MIG + instance_flexibility_policy (ANY_SINGLE_ZONE)	2 static; Rank 0: [n2-standard-2], Rank 1: [n2d-standard-2]	Both nodes fell back to Rank 1 (2× n2d-standard-2) co-located in a single zone; srun -N 2 -p flexzone PASS
flexspot	MIG + instance_flexibility_policy (ANY + SPOT)	2 static; Rank 0: [n2-standard-2, n2d-standard-2], Rank 1: [e2-standard-2, n1-standard-2]	Provisioned as SPOT VMs; srun -N 2 -p flexspot PASS
bulk	BULK_INSERT (static)	2 static (n2-standard-2)	Booted idle; srun -N 2 -p bulk PASS
burst	BULK_INSERT (static + dynamic)	1 static + 2 dynamic	Static + dynamic burst scale-up; srun -N 2 -p burst PASS
debug	Default AUTO (dynamic)	0 static + 2 dynamic	Dynamic scale-up via BulkInsert & idle scale-down; srun -N 2 -p debug PASS
auto	Explicit AUTO (dynamic)	0 static + 2 dynamic	Dynamic scale-up via BulkInsert & idle scale-down; srun -N 1 -p auto PASS

Post-Deployment Lifecycle & Corner Cases Verified on Live Cluster

• Heterogeneous Fallback Scheduling & Queueing: Submitted 5 --exclusive jobs and a 12-job burst flood against the 3-node heterogeneous flex partition (2× n2-standard-2 + 1× n1-standard-2); all jobs queued and completed (ExitCode 0:0) with 0 node drains.
• Fault Handling: Verified rapid scancel, multi-node (-N 3) mid-flight scancel, task crash (kill -9), walltime enforcement (--time=00:01:00 $\rightarrow$ TIMEOUT), and scontrol DRAIN/RESUME routing on flex nodes.
• MIG Auto-Healing & slurmsync.py Recovery: Triggered gcloud compute instance-groups managed recreate-instances on a live flex fallback VM; verified slurmsync.py transitioned the node to DOWN+CLOUD (MIG Auto-Healing instance repair in progress) and automatically resumed it to IDLE once the recreated VM rejoined.
• Scale-Down (suspend.py) & Clean Teardown: Verified automatic SuspendTime scale-down on dynamic partitions, manual POWER_DOWN instance deletion across all 5 Compute MIGs, and clean gcluster destroy (137 resources destroyed, 0 orphaned MIGs, instance templates, resource policies, or VMs).

[AdarshK15]

LGTM

[arpit974]

CI Triage: Failing / Stuck Checks (Unrelated to PR Changes)

None of the 5 failing or stuck CI checks below reached the Slurm modules (schedmd-slurm-gcp-v6-{nodeset,partition,controller}) modified in this PR:

1. PR-test-slurm-storage (cd18e8d5-c860-4c77-8076-3e6f7eeac51d) — Failed in group: primary (modules/file-system/cloud-storage-bucket):

   • examples/storage-slurm.yaml defines two sequential deployment groups: group: primary (modules/network/vpc and modules/file-system/cloud-storage-bucket with storage_class: RAPID, placement_zones, and anywhere_cache) followed by group: slurm-cluster.
   • gcluster deploy failed during terraform apply of group: primary before group: slurm-cluster was ever initialized or applied:

     Error: Error creating AnywhereCache: googleapi: Error 400: Zonal buckets are incompatible with Anywhere Cache., invalid
       with module.Anywhere_cache.google_storage_anywhere_cache.cache[0],
       on main.tf line 149, in resource "google_storage_anywhere_cache" "cache":
     

2. PR-test-ml-a4-highgpu-onspot-slurm (d710f8be-cc1d-45fd-af23-71d1e96e2fb3 & re-run 757766f3-4899-47d3-b41c-93727ace3144) — Failed in pre-build GCE A4 GPU capacity probe:

   • In tools/cloud-build/daily-tests/builds/ml-a4-highgpu-onspot-slurm.yaml, wait_for_available_zone.sh (find_available_zone.sh probing gcloud compute instances bulk create for 4× a4-highgpu-8g) runs prior to tools/get_binary.sh (make) and gcluster deploy.
   • Every candidate zone failed with either Does not currently have sufficient capacity for the requested resources (us-central1-b, europe-west1-b, asia-southeast1-b) or Quota 'GPUS_PER_GPU_FAMILY' exceeded. Limit: 0.0 (europe-west4, us-south1, us-west2, us-west3), terminating at --- DEPLOYMENT FAILED(Couldn't find a zone to deploy) --- before gcluster was built or invoked.

3. PR-test-ml-a3-highgpu-onspot-slurm (90032616-8428-4baa-93c1-bc7f9403026e) — Stuck in pre-build GCE A3-HighGPU capacity retry loop:

   • Admitted into test-kueue-cluster (pod/ml-a3-highgpu-onspot-slurm-900326-5b85s), where it is looping every 5 minutes on line 149 of tools/cloud-build/daily-tests/builds/ml-a3-highgpu-onspot-slurm.yaml (wait_for_available_zone.sh running gcloud compute instances bulk create --machine-type=a3-highgpu-8g --min-count=2) prior to tools/get_binary.sh (make) and gcluster deploy.
   • Every candidate zone is returning Does not currently have sufficient capacity for the requested resources (--- DEPLOYMENT FAILED(Couldn't find a zone to deploy) --- -> --- RETRYING in 5 minutes... ---).

4. PR-test-ml-a3-ultragpu-onspot-slurm (f96cfeb6-81ae-40f3-af0c-7ade36fd668d) — Suspended in GKE Kueue waiting on lock:

   • GKE Job ml-a3-ultragpu-onspot-slurm-f96cfe is still spec.suspend: true in test-kueue-cluster (no Pod has been scheduled yet) waiting for test-locks/ml-a3-ultragpu-onspot-slurm: 1 (nominalQuota: "1").
   • That lock has been held for 9+ hours by an earlier build (job/ml-a3-ultragpu-onspot-slurm-cbda06, pod ml-a3-ultragpu-onspot-slurm-cbda06-vf8hp), which is itself looping in wait_for_available_zone.sh for 4× a3-ultragpu-8g due to Does not currently have sufficient capacity for the requested resources.

5. PR-test-packer (36e6d402-9de2-4504-a045-16645262d6de) — Deadlocked in GKE Kueue by a 6-day orphaned job:

   • GKE Job packer-36e6d4 is still spec.suspend: true in test-kueue-cluster (no Pod has been scheduled yet) waiting for test-locks/packer: 1 (nominalQuota: "1").
   • That lock has been held since 2026-09-23T11:06:36Z (6+ days) by an orphaned GKE Job packer-a3fb29 (pod/packer-a3fb29-zmz82, from build a3fb29e1-15c8-4d1a-86b2-318f0db7b803, hung at TASK [Execute gcluster deploy]) because tools/cloud-build/daily-tests/builds/packer.yaml omits activeDeadlineSeconds on the Kubernetes Job spec.