Skip to content

Conversation

@jhadvig
Copy link
Member

@jhadvig jhadvig commented Jan 19, 2026

During cluster upgrades (4.20→4.21), the console and downloads PDBs block node draining when pods are unhealthy or stuck, causing MCO to timeout and fail:
Cannot evict pod as it would violate the pod's disruption budget

Solution:
Add unhealthyPodEvictionPolicy: AlwaysAllow to both PDBs.
This allows unhealthy/stuck pods to be evicted even if it violates the budget, while healthy pods remain protected by maxUnavailable: 1
Impact
✅ Unblocks cluster upgrades (fixes node drain failures)
✅ Maintains availability guarantee during normal operations
✅ Works for both SNO and HA topologies
✅ No behavior change for healthy pods

/assign @TheRealJon @cajieh

@openshift-ci openshift-ci bot requested review from TheRealJon and spadgett January 19, 2026 16:28
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 19, 2026
@jhadvig
Copy link
Member Author

jhadvig commented Jan 19, 2026

/cherry-pick release-4.21

@openshift-cherrypick-robot

@jhadvig: once the present PR merges, I will cherry-pick it on top of release-4.21 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@cajieh
Copy link
Contributor

cajieh commented Jan 19, 2026

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 19, 2026
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 19, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cajieh, jhadvig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cajieh
Copy link
Contributor

cajieh commented Jan 19, 2026

/verified

@openshift-ci-robot
Copy link
Contributor

@cajieh: The /verified command must be used with one of the following actions: by, later, remove, or bypass. See https://docs.ci.openshift.org/docs/architecture/jira/#premerge-verification for more information.

Details

In response to this:

/verified

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@cajieh
Copy link
Contributor

cajieh commented Jan 19, 2026

/verified later @yapei

@openshift-ci-robot
Copy link
Contributor

@cajieh: This PR has been marked to be verified later by @yapei.

Details

In response to this:

/verified later @yapei

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jan 19, 2026
@cajieh
Copy link
Contributor

cajieh commented Jan 19, 2026

/jira refresh

@openshift-ci-robot
Copy link
Contributor

@cajieh: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@cajieh
Copy link
Contributor

cajieh commented Jan 19, 2026

/retest

@cajieh
Copy link
Contributor

cajieh commented Jan 19, 2026

@jhadvig It looks like you need to update the title to include "OCPBUGS-72260".🙂

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 19, 2026

@jhadvig: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@jhadvig jhadvig changed the title OCPBUGS-72387 Fix PDB blocking node drains during cluster upgrades OCPBUGS-72387: Fix PDB blocking node drains during cluster upgrades Jan 20, 2026
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jan 20, 2026
@openshift-ci-robot
Copy link
Contributor

@jhadvig: This pull request references Jira Issue OCPBUGS-72387, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @yapei

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

During cluster upgrades (4.20→4.21), the console and downloads PDBs block node draining when pods are unhealthy or stuck, causing MCO to timeout and fail:
Cannot evict pod as it would violate the pod's disruption budget

Solution:
Add unhealthyPodEvictionPolicy: AlwaysAllow to both PDBs.
This allows unhealthy/stuck pods to be evicted even if it violates the budget, while healthy pods remain protected by maxUnavailable: 1
Impact
✅ Unblocks cluster upgrades (fixes node drain failures)
✅ Maintains availability guarantee during normal operations
✅ Works for both SNO and HA topologies
✅ No behavior change for healthy pods

/assign @TheRealJon @cajieh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested a review from yapei January 20, 2026 12:28
@jhadvig
Copy link
Member Author

jhadvig commented Jan 20, 2026

Thanks @cajieh 👍

@openshift-merge-bot openshift-merge-bot bot merged commit ce61af9 into openshift:main Jan 20, 2026
10 checks passed
@openshift-ci-robot
Copy link
Contributor

@jhadvig: Jira Issue OCPBUGS-72387: All pull requests linked via external trackers have merged:

This pull request has the verified-later tag and will need to be manually moved to VERIFIED after testing. Jira Issue OCPBUGS-72387 has been moved to the MODIFIED state.

Details

In response to this:

During cluster upgrades (4.20→4.21), the console and downloads PDBs block node draining when pods are unhealthy or stuck, causing MCO to timeout and fail:
Cannot evict pod as it would violate the pod's disruption budget

Solution:
Add unhealthyPodEvictionPolicy: AlwaysAllow to both PDBs.
This allows unhealthy/stuck pods to be evicted even if it violates the budget, while healthy pods remain protected by maxUnavailable: 1
Impact
✅ Unblocks cluster upgrades (fixes node drain failures)
✅ Maintains availability guarantee during normal operations
✅ Works for both SNO and HA topologies
✅ No behavior change for healthy pods

/assign @TheRealJon @cajieh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot

@jhadvig: new pull request created: #1100

Details

In response to this:

/cherry-pick release-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-robot
Copy link
Contributor

Fix included in accepted release 4.22.0-0.nightly-2026-01-21-055002

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria verified-later

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants