Update huge pages KEP for container isolation of huge pages #1199

bg-chun · 2019-08-05T13:02:38Z

Propose new enhancement for hugepage.

add example of CRI update
Support container isolation of huge pages

More information is available in below issue.
kubernetes/kubernetes#80716

k8s-ci-robot · 2019-08-05T13:02:39Z

Welcome @bg-chun!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2019-08-05T13:02:46Z

Hi @bg-chun. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

bg-chun · 2019-09-02T11:05:00Z

/ok-to-test

k8s-ci-robot · 2019-09-02T11:05:14Z

@bg-chun: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

odinuge · 2019-08-24T15:11:12Z

keps/sig-node/20190129-hugepages.md

+    - [Phase 2](#Phase-2)
+      - [Support container isolation of huge pages](#support-container-isolation-of-huge-pages)
+      - [Enhance Node Allocatable feature to reserve huge pages for system](#enhance-node-allocatable-feature-to-reserve-huge-pages-for-system)
+      - [cAdviser changes(Phase 2)](#cAdviser-changes(Phase-2))


s/cAdviser/cAdvisor/g 😄

odinuge · 2019-09-02T11:04:56Z

keps/sig-node/20190129-hugepages.md

+To support NUMA, `cAdviser` should discover and store pre-allocated huge pages per NUMA node. The `v3` version of `MachineInfo` will be introduced.
+
+#### Enhance Node Allocatable feature to reserve huge pages for system
+Some system services like `OVS-DPDK` comsume huge pages per NUMA node, to determine the allocatalbe number of huge pages in `kubelet`, `Node Allocatable feature` should support to reserve huge pages per NUMA node.


Not sure if I understand this correct; do you mean that huge pages per NUMA node should be node resources in k8s in the same way as huge pages (and cpu, memory, and empherial storage) is today? Or do you mean that the data should be stored internally, so that it can be utilized by the topology manager?

At this point, the data which represents pre-allocated hugepages per NUMA node will be additional information for future usage(Memory Manager or maybe Topology Manager).
So, I mean the data should be stored internally.

And I believe that the node resources that you mentioned are tightly coupled with node scheduler.

bg-chun · 2019-09-03T07:11:59Z

@derekwaynecarr
May I ask you to trigger testing?

odinuge · 2019-09-03T10:40:47Z

/ok-to-test

bg-chun · 2019-09-03T12:05:46Z

Oh... it seems that CI does not allow updating the table of contents of existing KEP.

AlexeyPerevalov

Handle memory restriction per NUMA node it's necessary feature for DPDK like applications. But not so clear how to guarantee in common case the memory allocation from specific NUMA node (e.g. in case of anonymous hugepages). The parent process of the container can bind NUMA node for memory allocation (e.g. with libnuma), but process itself can rebind it.

AlexeyPerevalov · 2019-09-03T12:09:31Z

keps/sig-node/20190129-hugepages.md

+
+#### Support container isolation of huge pages
+
+Container isolation of huge pages should be supported to avoid competition between containers to consume huge pages. Currently, `kubelet` sets the agregated huge pages limits on pod's cgroup of hugetlb subsystem. This should be enhanced to set limits on container's cgroup.


What kind of cgroup are you going to use? Current memory cgroup doesn't have support of limiting NUMA nodes.

Container isolation of hugepages does not mean limiting NUMA nodes; it's a related issue but totally different.

At this point, Kubelet supports hugepages in pod isolation, not container isolation.
This is mentioned on an official website(see future section).
And this issue provides more details.

Supporting Pod isolation of hugepage means that the below situation that can happen.

The pod has two containers
(container A and B).

Each of the containers requests 2 of 1GB-Hugepages as a resource.
(container A: 2GB / container B : 2GB)

But Kubelet set a limitation as 4GB on pod's cgroup of hugetlb subsystem.

It means a single container in a pod can consume 4GB maximally.

If a container consumes 2GB maximally, there will be no issue.
But if a container consumes 4GB, another one cannot consume any hugepage.

What kind of cgroup are you going to use?
=> To limiting NUMA nodes, I am going to use cpuset subsystem(cpuset.mems).
But this KEP and updates will not cover it, Memory Manager KEP will cover it.

bg-chun · 2019-09-03T14:20:59Z

Below PR updates cAdvisor to discover and store pre-allocated huge pages per NUMA node as mentioned in KEP updates.
google/cadvisor#2304

bg-chun · 2019-09-04T12:41:09Z

It seems that to set a limit of hugepages over CRI message, LinuxContainerResources should be updated and the function WithResources in containerd should also be updated.

But, I'm not sure whether the additional update of containerd is required or not.
The additional update means updating the other part of containerd like a containerd-shim or runc.

This week, I will check it then I will leave the result of this work.

bg-chun · 2019-10-22T04:28:18Z

@derekwaynecarr , @bart0sh, @kad , @odinuge
Guys, I updated KEP to focus on only container isolation and spread the content to whole document.
May i ask you guys to review it again?

derekwaynecarr

thanks for the update. merging the kep update, and we can discuss implementation separately.

/lgtm
/approve

k8s-ci-robot · 2019-10-22T15:17:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bg-chun, derekwaynecarr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-node/OWNERS~~ [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

derekwaynecarr · 2019-10-22T15:19:28Z

@bg-chun for hugetlbfs, the writing container will be charged for its usage (similar to memory). assuming more than one container wants to write to hugetlbfs, requests alone should be sufficient, as limits should have been limiting the local container.

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

cbf123 · 2020-02-06T20:47:04Z

keps/sig-node/20190129-hugepages.md

 `hugepages-2Mi: 4Mi`, but invalid to request `hugepages-2Mi: 3Mi`.

 The request and limit for `hugepages-<hugepagesize>` must match.  Similar to
 memory, an application that requests `hugepages-<hugepagesize>` resource is at
 minimum in the `Burstable` QoS class.

+If multiple containers consume huge pages in the same pod, the request must be


I think we would want to differentiate between init containers and regular containers, since the init containers are all guaranteed to be done when the regular containers start up.

k8s-ci-robot requested review from dchen1107 and derekwaynecarr August 5, 2019 13:03

bg-chun mentioned this pull request Aug 5, 2019

Support Container Isolation of Hugepages kubernetes/kubernetes#80716

Closed

bg-chun force-pushed the hugepage-updates branch 2 times, most recently from 74028e4 to 2c9ee00 Compare August 5, 2019 14:11

bg-chun mentioned this pull request Aug 8, 2019

Propose Memory Manager for NUMA awareness #1203

Merged

bg-chun force-pushed the hugepage-updates branch from 2c9ee00 to 54911ed Compare September 2, 2019 11:00

odinuge reviewed Sep 2, 2019

View reviewed changes

bg-chun mentioned this pull request Sep 3, 2019

Add hugepage info to v1 node structure google/cadvisor#2304

Merged

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 3, 2019

AlexeyPerevalov reviewed Sep 3, 2019

View reviewed changes

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 3, 2019

bg-chun force-pushed the hugepage-updates branch from 255b299 to 294a606 Compare September 4, 2019 04:53

bg-chun mentioned this pull request Sep 4, 2019

cgroup(cpuset) allow a container to pin pthread to any logical CPUs on the machine kubernetes/kubernetes#82160

Closed

derekwaynecarr approved these changes Oct 22, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 22, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 22, 2019

k8s-ci-robot merged commit 2e968f3 into kubernetes:master Oct 22, 2019

k8s-ci-robot added this to the v1.17 milestone Oct 22, 2019

This was referenced Oct 29, 2019

Hugepages PRs Tracker kubernetes/kubernetes#84526

Closed

Add support of setting hugepage limit on container cgroup sandbox moby/moby#40160

Closed

This was referenced Nov 4, 2019

[WIP] Add hugepage info to dockercontainer's resource spec kubernetes/kubernetes#84701

Closed

[WIP] Add hugepage limit to specResource moby/moby#40171

Closed

bg-chun mentioned this pull request Dec 18, 2019

1.18.0-alpha.1 tag cutting request. kubernetes/sig-release#928

Closed

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Dec 18, 2019

Update cri-api vendor

ceaca65

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Dec 19, 2019

Update cri-api vendor

85fad2b

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

This was referenced Dec 21, 2019

update cri-plugin to parse hugepages limit containerd/cri#1332

Merged

update createSandboxContainer to parse hugepages limits from CRI message cri-o/cri-o#2940

Merged

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

f1217c6

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

3bf6076

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

31f780c

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

c146006

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

fb8c104

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

df695db

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

cda7f26

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun pushed a commit to bg-chun/cri that referenced this pull request Feb 4, 2020

Update cri-api vendor

b5db106

update cri-api vendor to include hugepages changes KEP: kubernetes/enhancements#1199 CRI: kubernetes/kubernetes#83614 Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

bg-chun mentioned this pull request Feb 5, 2020

Extending Hugepage Feature #1539

Closed

cbf123 reviewed Feb 6, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update huge pages KEP for container isolation of huge pages #1199

Update huge pages KEP for container isolation of huge pages #1199

bg-chun commented Aug 5, 2019 •

edited

Loading

k8s-ci-robot commented Aug 5, 2019

k8s-ci-robot commented Aug 5, 2019

bg-chun commented Sep 2, 2019

k8s-ci-robot commented Sep 2, 2019

odinuge Aug 24, 2019

bg-chun Sep 2, 2019

odinuge Sep 2, 2019

bg-chun Sep 2, 2019

bg-chun commented Sep 3, 2019

odinuge commented Sep 3, 2019

bg-chun commented Sep 3, 2019

AlexeyPerevalov left a comment

AlexeyPerevalov Sep 3, 2019

bg-chun Sep 3, 2019

bg-chun Sep 3, 2019 •

edited

Loading

bg-chun commented Sep 3, 2019

bg-chun commented Sep 4, 2019

bg-chun commented Oct 22, 2019

derekwaynecarr left a comment

k8s-ci-robot commented Oct 22, 2019

derekwaynecarr commented Oct 22, 2019

cbf123 Feb 6, 2020


		#### Support container isolation of huge pages

		Container isolation of huge pages should be supported to avoid competition between containers to consume huge pages. Currently, `kubelet` sets the agregated huge pages limits on pod's cgroup of hugetlb subsystem. This should be enhanced to set limits on container's cgroup.

Update huge pages KEP for container isolation of huge pages #1199

Update huge pages KEP for container isolation of huge pages #1199

Conversation

bg-chun commented Aug 5, 2019 • edited Loading

Propose new enhancement for hugepage.

k8s-ci-robot commented Aug 5, 2019

k8s-ci-robot commented Aug 5, 2019

bg-chun commented Sep 2, 2019

k8s-ci-robot commented Sep 2, 2019

odinuge Aug 24, 2019

Choose a reason for hiding this comment

bg-chun Sep 2, 2019

Choose a reason for hiding this comment

odinuge Sep 2, 2019

Choose a reason for hiding this comment

bg-chun Sep 2, 2019

Choose a reason for hiding this comment

bg-chun commented Sep 3, 2019

odinuge commented Sep 3, 2019

bg-chun commented Sep 3, 2019

AlexeyPerevalov left a comment

Choose a reason for hiding this comment

AlexeyPerevalov Sep 3, 2019

Choose a reason for hiding this comment

bg-chun Sep 3, 2019

Choose a reason for hiding this comment

bg-chun Sep 3, 2019 • edited Loading

Choose a reason for hiding this comment

bg-chun commented Sep 3, 2019

bg-chun commented Sep 4, 2019

bg-chun commented Oct 22, 2019

derekwaynecarr left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Oct 22, 2019

derekwaynecarr commented Oct 22, 2019

cbf123 Feb 6, 2020

Choose a reason for hiding this comment

bg-chun commented Aug 5, 2019 •

edited

Loading

bg-chun Sep 3, 2019 •

edited

Loading