[Neelabh94]

Problem

gcluster job logs --main-only (also the implicit default for jobs with more than 5 pods) had two problems:

1. It matched the wrong pods. The selector

   jobset.sigs.k8s.io/jobset-name=<job>,jobset.sigs.k8s.io/job-index=0,batch.kubernetes.io/job-completion-index=0
   

   uses indices that are scoped per replicated job, so on a Pathways JobSet it matches <job>-pathways-head-0-0 and <job>-worker-0-0, and kubectl logs -c workload-container fails on the worker:

   Error: failed to get logs: error: container workload-container is not valid for pod <job>-worker-0-0-xxxxx
   

   It only starts working once the worker pods are garbage-collected after the JobSet completes. In PR-test-gke-tpu-v5e this shows up as a race with the test's retry window: build 900172f3 hit the error 6 times before passing on attempt 7, and builds 5214db0f and 5a2a7769 ran out of retries even though the workload itself succeeded.

2. It guessed the container. -c came from matching workload-container / pathways- name prefixes, falling back to the first container. Custom templates (--gke-custom-templates-path) could get the wrong container, or the same error on another replicated job.

Fix

• Pods: the main-only selector adds jobset.sigs.k8s.io/replicatedjob-name=<spec.replicatedJobs[0].name>, read from the live JobSet: main-job, pathways-head, or whatever a custom template lists first.
• Container: main-only mode no longer passes -c or --all-containers, so kubectl (1.24+) reads the pod's default container: the one named by the kubectl.kubernetes.io/default-container annotation, otherwise spec.containers[0]. Native sidecars (Pathways proxy/rm, the colocated Python sidecar, GCS FUSE) are init containers and never the default, so every built-in template resolves to the same container as before. Custom templates can steer it with the standard annotation; the job guide now says so.
• Fallback: if the JobSet can't be read, main-only reads rank-0 of every replicated job, each with its default container (previously --all-containers, sidecars included).
• --main-only=false, the 5-pod auto-default, the --max-log-requests guard and the retry loop's behaviour are unchanged.
• findWorkloadContainer, the name prefixes and extractJobSetNameFromSelector are removed (the JobSet name is passed down instead of re-parsed from the selector); the JobSet query only reads {.spec.replicatedJobs[0].name}. The --main-only help text matches the docs.

Verification

Unit tests

• TestGetJobLogs: standard, Pathways, renamed-first-job and unreadable JobSets; the 5/6-pod default and 10/11-pod limit boundaries; a pod with no output yet; --main-only=false; and both follow modes (MockExecutor now records streamed commands).
• TestResolveLogsSelector_MainOnlyTargetsMainReplicatedJob parses the selector with k8s.io/apimachinery/pkg/labels and checks it rejects workers, rank-1 pods, other replicated jobs and other JobSets, including in the unreadable-JobSet fallback.
• TestFetchLogsWithRetry: containers still starting (also with Windows line endings), no matching pods, empty output, fail-fast errors, and the 12-attempt timeout with JobSet warning events.
• TestFirstReplicatedJobName and TestGetJobPodCount: trimming, Windows line endings and failures.
• go test ./... and pre-commit pass.

Windows: gcluster.exe and both test packages cross-compile for windows/amd64 and go vet is clean with GOOS=windows. kubectl is run with an argument list (no shell), and all output parsing tolerates CRLF.

JobSet v0.12.0: labels, the v1alpha2 API, spec.replicatedJobs[].name and the Indexed default are unchanged from v0.8.1; the selector and container choice were also checked against a v0.12.0 controller.

Live test on a GKE 1.36 cluster (JobSet v0.8.1) with kubectl 1.36, against JobSets with the same container layouts as the built-in templates:

Case	develop	this PR
Pathways, --main-only	❌ container workload-container is not valid for pod …-worker-0-0-…	✅ head workload-container
Headless Pathways (two regular containers, no annotation), --main-only	❌ container pathways-proxy is not valid for pod …-worker-0-0-…	✅ head pathways-proxy
Custom template, sidecar listed first plus the default-container annotation, --main-only	❌ container metrics-exporter is not valid for pod …-evaluator-0-0-…	✅ reads my-trainer
Parallel containers (two regular containers, no annotation), 6 pods, no flag	not run	✅ auto --main-only, rank-0 workload-container-0
Pathways, --main-only -f	not run	✅ streams head workload-container
Pathways, --main-only=false	not run	✅ every pod and container, sidecars included (unchanged)

Not addressed here (same on develop), planned for a follow-up PR:

• Waiting for pods that don't exist yet, in both modes including -f, while telling apart jobs that are queued, finished, or not found, and applying the main-only default once pods exist.
• With -f --main-only=false, kubectl counts every container stream, sidecars included, against --max-log-requests, so a JobSet with 10 or fewer pods can still exceed the limit.

Submission Checklist

• Have you installed and run this change against pre-commit? (pre-commit install)
• Are all tests passing? (make tests)
• If applicable, have you written additional unit tests to cover this change?
• Is unit test coverage still above 80%?
• Have you updated all applicable documentation?
• Have you followed the guidelines in our Contributing document?

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the reliability of the gcluster job logs --main-only command by refining the Kubernetes label selector used to identify the main job pod. By adding a constraint on the replicated job name, the command now correctly ignores auxiliary worker pods that previously caused log retrieval failures due to label collisions. This change ensures consistent behavior in JobSet environments, particularly when worker pods are present.

Highlights

• Selector Refinement: Updated the --main-only log selector to explicitly include jobset.sigs.k8s.io/replicatedjob-name in (main-job, pathways-head), ensuring that only the rank-0 pod of the main replicated job is targeted.
• Race Condition Resolution: Prevents log retrieval errors occurring when the previous selector incorrectly matched worker pods that had not yet been garbage-collected.
• Test Coverage: Added a new unit test TestResolveLogsSelector_MainOnlyTargetsMainReplicatedJob to verify the selector logic and updated existing test expectations to reflect the more precise filtering.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refactors the GKE job orchestrator's log selector logic to narrow down the target pods when mainOnly is enabled. It introduces a selector suffix that restricts matches to the main replicated jobs (main-job and pathways-head), preventing a race condition where auxiliary worker pods with identical job and completion indices were incorrectly matched. The changes are accompanied by updated unit tests and a new test targeting this specific scenario. I have no feedback to provide.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the GKE orchestrator's log retrieval logic to correctly target the main replicated job in a JobSet (such as pathways-head in Pathways) when using the --main-only flag. It replaces the resolveLogsSelector and getFirstContainerName methods with resolveLogsTarget and inspectJobSetMainJob, which leverage a new logsTarget struct to capture both the target selector and container. Additionally, comprehensive unit tests and regression tests have been added or updated to validate these changes. No review comments were provided, so there is no additional feedback to address.