Fix issue where cancelling a job from grafana would prevent the worker from picking up new jobs #3896
+148
−34
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r from picking up new jobs
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes an issue where cancelling a running job from Grafana would prevent workers from picking up new jobs. The root cause was that only one job reservation was being marked as completed during cancellation, leaving other reservations incomplete and blocking the concurrency group.
Changes:
- Modified
forceJobCompletionto complete all uncompleted reservations for a job instead of just one - Updated tests to verify multiple reservations are properly completed
- Added a comprehensive integration test demonstrating the fix allows subsequent jobs to run after cancellation
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| packages/realm-server/handlers/handle-remove-job.ts | Updated forceJobCompletion to mark all uncompleted reservations as completed instead of just a specific one, and simplified the function signature |
| packages/realm-server/tests/server-endpoints/maintenance-endpoints-test.ts | Added test imports, updated existing tests to create multiple reservations, and added comprehensive integration test verifying workers can pick up new jobs after cancellation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
backspace
approved these changes
Jan 23, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue where when a running job was cancelled from grafana the worker would be tainted in such a way that it was unable to pick up new work.