Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Task tracker for addition of pod priority and preemption to Kubernetes #47604

Closed
13 tasks done
bsalamat opened this issue Jun 15, 2017 · 28 comments
Closed
13 tasks done

Task tracker for addition of pod priority and preemption to Kubernetes #47604

bsalamat opened this issue Jun 15, 2017 · 28 comments
Assignees
@bsalamat
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. milestone/removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.

Comments

@bsalamat
Copy link
Member

bsalamat commented Jun 15, 2017 •
edited
Loading

Here is a list of items we need to add pod priority and preemption. These items are not the same size and some may need more work. We can break those down later into smaller tasks if needed.

  • Add a design doc to add priority to Kubernetes API.
  • Add pod priority
    • Add priority fields to PodSpec.
    • Add PriorityClass API object.
    • Add an Admission Controller for priority.
  • Add a design doc for priority-based preemption.
  • Incorporate priority in pod scheduling logic (to fix the starvation problem)
  • Support PodDisruptionBudget in preemption.
  • Add priority-base preemption to scheduler.
  • Quota per priority
    • Add priority fields to ResourceQuota.
    • Incorporate priority in quota checks.
  • Add documentation on how to use pod priority and its effects on behavior of the system.
@bsalamat bsalamat mentioned this issue Jun 15, 2017
Merged
@k8s-github-robot
Copy link

@bsalamat There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

@k8s-github-robot k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jun 15, 2017
@bsalamat
Copy link
Member Author

@kubernetes/sig-scheduling-feature-requests
@kubernetes/sig-scheduling-misc

@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. kind/feature Categorizes issue or PR as related to a new feature. labels Jun 15, 2017
@k8s-github-robot k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jun 15, 2017
@timothysc timothysc added this to the v1.8 milestone Jun 16, 2017
@bsalamat bsalamat mentioned this issue Jun 16, 2017
Closed
@ravisantoshgudimetla
Copy link
Contributor

/cc me

@ravisantoshgudimetla
Copy link
Contributor

@bsalamat I see that you raised PR's for Add pod priority, #48377 - Can you please create individual issues against each task, so that it becomes clear. I want to work on some items and it will be helpful to have individual issues with clear description. Also, with these issues, they can be assigned to individuals. WDYT?

@ravisantoshgudimetla
Copy link
Contributor

/cc @derekwaynecarr

@davidopp
Copy link
Member

davidopp commented Jul 7, 2017

I think this could be broken down into a few separate issues. Here's one possibility. Note that I'm just listing the things that haven't been started yet (IIUC the goal is for people to see what work they could take, so things that are already started probably aren't relevant?)

  • implement priority/preemption in pod scheduling logic
  • documentation for priority/preemption
  • design for adding priority to ResourceQuota
  • implement adding priority to ResourceQuota
  • update ResourceQuota documentation to reflection priority

Does that look reasonable?

This was referenced Jul 7, 2017
Closed
Closed
Closed
Closed
Closed
@bsalamat
Copy link
Member Author

bsalamat commented Jul 7, 2017

Thanks, @davidopp!
I created the issues to track remaining work items.

@ravisantoshgudimetla
Copy link
Contributor

Thanks @davidopp and @bsalamat !!

@bsalamat bsalamat mentioned this issue Jul 20, 2017
Merged
k8s-github-robot pushed a commit that referenced this issue Jul 20, 2017
Merge pull request #48377 from bsalamat/priority_class
8d26afa 
Automatic merge from submit-queue

Add PriorityClass API object under new "scheduling" API group

**What this PR does / why we need it**: This PR is a part of a series of PRs to add pod priority to Kubernetes. This PR adds a new API group called "scheduling" with a new API object called "PriorityClass". PriorityClass maps the string value of priority to its integer value.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**: Given the size of this PR, I will add the admission controller for the PriorityClass in a separate PR.

**Release note**:

```release-note
Add PriorityClass API object under new "scheduling" API group
```

ref/ #47604
ref/ #48646
@resouer
Copy link
Contributor

resouer commented Jul 20, 2017 •
edited
Loading

@ravisantoshgudimetla have you begin design for adding priority to ResourceQuota? I think this is high priority in all separate remaining tasks.

@ravisantoshgudimetla
Copy link
Contributor

ravisantoshgudimetla commented Jul 20, 2017 •
edited
Loading

@resouer I had change of priorities at my workplace because of which I will not be able to work on this issue this quarter. I had an offline discussion with @bsalamat and updated him on this. Please feel free to pick this up in case you are interested.

@bsalamat bsalamat mentioned this issue Jul 20, 2017
Merged
@bsalamat bsalamat mentioned this issue Jul 27, 2017
Merged
@wanghaoran1988
Copy link
Contributor

/cc @mdshuai

@bsalamat bsalamat mentioned this issue Jul 31, 2017
Merged
@luxas
Copy link
Member

luxas commented Oct 20, 2017

@bsalamat @davidopp Can you confirm that this is on track for v1.9?

@bsalamat
Copy link
Member Author

@luxas Given that code freeze is only one month away, we are not very confident at this point, but I still call it on track for 1.9.

@bsalamat bsalamat mentioned this issue Oct 26, 2017
Closed
@dims
Copy link
Member

dims commented Nov 15, 2017

@bsalamat @luxas time to move this out to 1.10?

@bsalamat
Copy link
Member Author

I think some of the items will be ready for 1.9. Particularly, a solution for starvation will probably be added in 1.9.

This was referenced Nov 17, 2017
Merged
Merged
k8s-github-robot pushed a commit that referenced this issue Nov 19, 2017
Merge pull request #55109 from bsalamat/starvation2
6b97376 
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add a new scheduling queue based on priority queue.

**What this PR does / why we need it**:
This PR is a part of solution to fix potential starvation of pods in pod preemption. It adds a different scheduling queue to the scheduler that allows highest priority pods to be scheduled before other pods.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:
I tried enabling this queue and made sure that our existing tests pass with this queue enabled.

**Release note**:

```release-note
Add a new scheduling queue that helps schedule the highest priority pending pod first.
```

/sig scheduling

ref/ #54501
ref/ #47604
@bsalamat
Copy link
Member Author

/status in-progress

k8s-github-robot pushed a commit that referenced this issue Nov 21, 2017
Merge pull request #55933 from bsalamat/starvation3
34b258c 
Automatic merge from submit-queue (batch tested with PRs 54316, 53400, 55933, 55786, 55794). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add support to take nominated pods into account during scheduling to avoid starvation of higher priority pods

**What this PR does / why we need it**:
When a pod preempts lower priority pods, the preemptor gets a "nominated node name" annotation. We call such a pod a nominated pod. This PR adds the logic to take such nominated pods into account when scheduling other pods on the same node that the nominated pod is expected to run. This is needed to avoid starvation of preemptor pods. Otherwise, lower priority pods may fill up the space freed after preemption before the preemptor gets a chance to get scheduled.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #54501

**Special notes for your reviewer**: This PR is built on top of #55109 and includes all the changes there as well.

**Release note**:

```release-note
Add support to take nominated pods into account during scheduling to avoid starvation of higher priority pods.
```

/sig scheduling
ref/ #47604
@bsalamat bsalamat mentioned this issue Nov 22, 2017
Merged
@k8s-github-robot
Copy link

[MILESTONENOTIFIER] Milestone Removed From Issue

@bsalamat @kubernetes/sig-scheduling-misc

Important: Code freeze is in effect and only issues with priority/critical-urgent may remain in the v1.9 milestone.

Help

@k8s-github-robot k8s-github-robot removed this from the v1.9 milestone Nov 22, 2017
k8s-github-robot pushed a commit that referenced this issue Nov 23, 2017
Merge pull request #56178 from bsalamat/pdb
82c8898 
Automatic merge from submit-queue (batch tested with PRs 55952, 49112, 55450, 56178, 56151). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add PodDisruptionBudget support in pod preemption

**What this PR does / why we need it**:
This PR adds the logic to make scheduler preemption aware of PodDisruptionBudget. Preemption tries to avoid preempting pods whose PDBs are violated by preemption. If preemption does not find any other pods to preempt, it will preempt pods despite violating their PDBs.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #53913

**Special notes for your reviewer**:

**Release note**:

```release-note
Add PodDisruptionBudget support during pod preemption
```

ref/ #47604

/sig scheduling
@davidopp davidopp mentioned this issue Jan 5, 2018
Open
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 27, 2018
@bsalamat
Copy link
Member Author

/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Feb 27, 2018
@bsalamat bsalamat removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 27, 2018
@lukaszgryglicki lukaszgryglicki mentioned this issue Mar 1, 2018
Closed
@bsalamat
Copy link
Member Author

This feature is built and moved to beta in Kubernetes 1.11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bsalamat bsalamat

Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. milestone/removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests