DPTP-4801: cmd/ci-secret-bootstrap: Log Secret change keys#5130
DPTP-4801: cmd/ci-secret-bootstrap: Log Secret change keys#5130wking wants to merge 1 commit intoopenshift:mainfrom
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
|
@wking: This pull request references DPTP-4801 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: wking The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughCompute per-key diffs between expected and existing Kubernetes Secret Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 12 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (12 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@cmd/ci-secret-bootstrap/main.go`:
- Around line 805-818: The key-change classification has three issues: comparing
[]byte with !=, a missing short-declaration in the second loop, and an
inconsistent format string. Replace the direct byte-slice comparison in the
first loop by reusing existingValue and calling bytes.Equal(existingValue,
value) (import "bytes" if not present) and invert the result; fix the second
loop to use short declaration if _, ok := secret.Data[k]; !ok { ... } so ok is
declared; and normalize the format string in change to "added: %v, changed: %v,
removed: %v". Use the identifiers secret.Data, existingSecret.Data,
existingValue, addedKeys/changedKeys/removedKeys, and change to locate the
edits.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e449238e-715d-408d-9477-b370aa849dc9
📒 Files selected for processing (1)
cmd/ci-secret-bootstrap/main.go
933dec9 to
0e91221
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
cmd/ci-secret-bootstrap/main.go (1)
805-818: Make key-change summaries deterministic before logging.
addedKeys,changedKeys, andremovedKeyscome from map iteration, so output order is unstable. Sort each slice before formatting to keep logs/errors consistent across runs.Proposed patch
for k := range existingSecret.Data { if _, ok := secret.Data[k]; !ok { removedKeys = append(removedKeys, k) } } + sort.Strings(addedKeys) + sort.Strings(changedKeys) + sort.Strings(removedKeys) change := fmt.Sprintf("added: %v, changed: %v, removed: %v", addedKeys, changedKeys, removedKeys)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@cmd/ci-secret-bootstrap/main.go` around lines 805 - 818, The key-change summary is nondeterministic due to map iteration; before building the summary string (where change := fmt.Sprintf(...)), sort the slices addedKeys, changedKeys, and removedKeys (e.g., using sort.Strings) so their order is stable; locate the block that compares secret.Data and existingSecret.Data (uses equality.Semantic.DeepEqual) and insert sort.Strings(addedKeys), sort.Strings(changedKeys), and sort.Strings(removedKeys) just prior to the fmt.Sprintf call.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@cmd/ci-secret-bootstrap/main.go`:
- Around line 805-818: The key-change summary is nondeterministic due to map
iteration; before building the summary string (where change :=
fmt.Sprintf(...)), sort the slices addedKeys, changedKeys, and removedKeys
(e.g., using sort.Strings) so their order is stable; locate the block that
compares secret.Data and existingSecret.Data (uses equality.Semantic.DeepEqual)
and insert sort.Strings(addedKeys), sort.Strings(changedKeys), and
sort.Strings(removedKeys) just prior to the fmt.Sprintf call.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 343b0b42-54bc-48be-8a9d-7152bd7610d8
📒 Files selected for processing (1)
cmd/ci-secret-bootstrap/main.go
e8a721e to
a636842
Compare
Today there was a Deck outage, when both of two Pods restarted their 'deck' containers simultaneously at 22:23, causing: Application is not available errors to https://prow.ci.openshift.org/ users. Timestamps and exit 0: $ oc -n ci get -l component=deck -o json pods | jq -r '.items[].status.containerStatuses[] | select(.restartCount > 0) | {name, restartCount, lastState}' { "name": "deck", "restartCount": 1, "lastState": { "terminated": { "containerID": "cri-o://21786efafb65fd5b67e72eb2a1a91405b182d3b20daeb549980217210bf0e22a", "exitCode": 0, "finishedAt": "2026-04-23T22:23:37Z", "reason": "Completed", "startedAt": "2026-04-23T20:04:03Z" } } } { "name": "deck", "restartCount": 1, "lastState": { "terminated": { "containerID": "cri-o://7af2f63b995729d1e8e64b1776a6a2aa3439076e9d55616182ac61b1c64fe855", "exitCode": 0, "finishedAt": "2026-04-23T22:23:58Z", "reason": "Completed", "startedAt": "2026-04-23T19:28:51Z" } } } The container exits were because of Kubeconfig changes: $ oc -n ci logs -c deck --previous deck-54c8d55b65-6fxcn | tail -n16 | head -n2 {"component":"deck","file":"sigs.k8s.io/prow/cmd/deck/main.go:392","func":"main.main.func2","level":"info","msg":"Kubeconfig changed, exiting to trigger a restart","severity":"info","time":"2026-04-23T22:23:36Z"} {"component":"deck","file":"sigs.k8s.io/prow/pkg/interrupts/interrupts.go:63","func":"sigs.k8s.io/prow/pkg/interrupts.handleInterrupt","level":"info","msg":"Received signal.","severity":"info","signal":2,"time":"2026-04-23T22:23:36Z"} The Secret update was this ci-secret-bootstrapper in [1]: {"cluster":"app.ci","component":"ci-secret-bootstrap","file":"/go/src/github.com/openshift/ci-tools/cmd/ci-secret-bootstrap/main.go:815","func":"main.updateSecrets","level":"debug","msg":"secret updated","name":"deck","namespace":"ci","severity":"debug","time":"2026-04-23T22:23:02Z","type":"Opaque"} This commit adds more logs to that "secret updated" entry, to make it easier for us to figure out which change triggered the next bump, so we can decide if it's appropriate, and the kind of thing we'll accept a few minutes of Deck outage over, or if it's surprising churn. [1]: https://deck-internal-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/origin-ci-private/logs/periodic-ci-secret-bootstrap/2047432829557542912
|
Scheduling tests matching the |
|
@wking: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Today there was a Deck outage, when both of two Pods restarted their 'deck' containers simultaneously at 22:23, causing:
errors to https://prow.ci.openshift.org/ users. Timestamps and exit 0:
The container exits were because of Kubeconfig changes:
The Secret update was this ci-secret-bootstrapper:
This commit adds more logs to that
secret updatedentry, to make it easier for us to figure out which change triggered the next bump, so we can decide if it's appropriate, and the kind of thing we'll accept a few minutes of Deck outage over, or if it's surprising churn.Summary by CodeRabbit