libcontainer: add support for Intel RDT/CAT in runc #1279

xiaochenshen · 2017-01-17T19:49:31Z

v5 commits:
4d2756c libcontainer: add test cases for Intel RDT/CAT
692f6e1 libcontainer: add support for Intel RDT/CAT in runc
af3b0d9 libcontainer/SPEC.md: add documentation for Intel RDT/CAT

Changes in v5:
Reworked code according to @crosbymichael and @hqhq's comments:

Obsoleted the design of "refactor resource manager interface": removed generic resourceManagers, isolated IntelRdt functions without touching cgroups code, removed the static map with static keys on resourceManagers set.
Simplified IntelRdtManager implementation, removed "dead and empty" methods - GetPids(), GetAllPids(), Freeze(). Kept the required methods set - Apply(), GetStats(), Destroy(), GetPath(), and Set().
Re-implemented GetStats(), isolated cgroups.Stats and intelrdt.Stats and kept type safety on the Stats.
Changed test stuffs according to the functional rework.
Added read-only L3 cache info (CbmMask, MinCbmBits, and NumClosids) into Intelrdt.Stats. Runc users could run "runc events" to fetch the information, which helps know more about L3 cache capabilities and how to set correct L3 cache schema in different Intel Xeon platforms.

v4 commits:
b1c8366 libcontainer: add support for Intel RDT/CAT in runc
48d8ffe libcontainer/SPEC.md: add documentation for Intel RDT/CAT
8851a0d libcontainer: refactor resource manager interface
8064819 vendor: specs-go: update specs for Intel RDT/CAT

Changes in v4:
Rebased to latest master branch.

v3 commits:
be25a20 libcontainer: add support for Intel RDT/CAT in runc
96fd05f libcontainer/SPEC.md: add documentation for Intel RDT/CAT
d5ac70d libcontainer: refactor resource manager interface
c302b70 vendor: specs-go: update specs for Intel RDT/CAT

Changes in v3:
Addressed comment #1279 (comment) from @yummypeng :

Add parameter path to record IntelRdtPath in NewIntelRdtManager.

v2 commits (for keeping records in code review):
ab32d21 libcontainer: add support for Intel RDT/CAT in runc
5972fd5 libcontainer/SPEC.md: add documentation for Intel RDT/CAT
7f8b321 libcontainer: refactor resource manager interface
9b03a51 vendor: specs-go: update specs for Intel RDT/CAT

Changes in v2:
Addressed the comments from @hqhq and @yummypeng :

Added intelrdt config validator.
Removed resctrl filesystem re-mount logic in isIntelRdtMounted().
Optimized IsIntelRdtEnabled() to avoid unnecessary time-consuming mount points parsing.
Changed Intel RDT related json items to snake_case style.
Fixed some redundant codes with regard to intelRdtPath.
Fixed potential null pointer issue in IntelRdtManager.Set().
Fixed some typos (Sets, num_closids and etc.)
Improved libcontainer/SPEC.md: added minimal supported kernel version, provided a more sophisticated config example.

v1 commits (for keeping records in code review):
4752fd2 libcontainer: add support for Intel RDT/CAT in runc
c8486a7 libcontainer/SPEC.md: add documentation for Intel RDT/CAT
aefd02a libcontainer: refactor resource manager interface
8e8e5d9 vendor: specs-go: update specs for Intel RDT/CAT

This PR fixes issue #433

About Intel RDT/CAT feature:
Intel platforms with new Xeon CPU support Intel Resource Director Technology
(RDT). Cache Allocation Technology (CAT) is a sub-feature of RDT, which
currently supports L3 cache resource allocation.

This feature provides a way for the software to restrict cache allocation to a
defined 'subset' of L3 cache which may be overlapping with other 'subsets'.
The different subsets are identified by class of service (CLOS) and each CLOS
has a capacity bitmask (CBM).

For more information about Intel RDT/CAT can be found in the section 17.17
of Intel Software Developer Manual.

About Intel RDT/CAT kernel interface:
In Linux 4.10 kernel or newer, the interface is defined and exposed via
"resource control" filesystem, which is a "cgroup-like" interface.

Comparing with cgroups, it has similar process management lifecycle and
interfaces in a container. But unlike cgroups' hierarchy, it has single level
filesystem layout.

Intel RDT "resource control" filesystem hierarchy:

mount -t resctrl resctrl /sys/fs/resctrl
tree /sys/fs/resctrl
/sys/fs/resctrl/
|-- info
|   |-- L3
|       |-- cbm_mask
|       |-- min_cbm_bits
|       |-- num_closids
|-- cpus
|-- schemata
|-- tasks
|-- <container_id>
    |-- cpus
    |-- schemata
    |-- tasks

For runc, we can make use of tasks and schemata configuration for L3 cache
resource constraints.

The file tasks has a list of tasks that belongs to this group (e.g.,
<container_id>" group). Tasks can be added to a group by writing the task ID
to the "tasks" file (which will automatically remove them from the previous
group to which they belonged). New tasks created by fork(2) and clone(2) are
added to the same group as their parent. If a pid is not in any sub group, it
Is in root group.

The file schemata has allocation bitmasks/values for L3 cache on each socket,
which contains L3 cache id and capacity bitmask (CBM).

	Format: "L3:<cache_id0>=<cbm0>;<cache_id1>=<cbm1>;..."

For example, on a two-socket machine, L3's schema line could be L3:0=ff;1=c0
which means L3 cache id 0's CBM is 0xff, and L3 cache id 1's CBM is 0xc0.

The valid L3 cache CBM is a contiguous bits set and number of bits that can
be set is less than the max bit. The max bits in the CBM is varied among
supported Intel Xeon platforms. In Intel RDT "resource control" filesystem
layout, the CBM in a group should be a subset of the CBM in root. Kernel will
check if it is valid when writing. e.g., 0xfffff in root indicates the max bits
of CBM is 20 bits, which mapping to entire L3 cache capacity. Some valid CBM
values to set in a group: 0xf, 0xf0, 0x3ff, 0x1f00 and etc.

For more information about Intel RDT/CAT kernel interface:
https://www.kernel.org/doc/Documentation/x86/intel_rdt_ui.txt

An example for runc:

Consider a two-socket machine with two L3 caches where the default CBM is
0xfffff and the max CBM length is 20 bits. With this configuration, tasks
inside the container only have access to the "upper" 80% of L3 cache id 0 and
the "lower" 50% L3 cache id 1:

"linux": {
	"intelRdt": {
		"l3CacheSchema": "L3:0=ffff0;1=3ff"
	}
}

Signed-off-by: Xiaochen Shen xiaochen.shen@intel.com

xiaochenshen · 2017-01-17T21:36:06Z

@cyphar @crosbymichael @hqhq @mrunalp @vishh
/cc @opencontainers/runc-maintainers

This PR will obsolete #1198

To address @crosbymichael and @cyphar 's comments #1198 (comment) and #1198 (comment), the design is updated:

It adds a new "ResourceManager" structure as the base interface for all resource managers, such as cgroups manager and incoming IntelRdt manager.

All registered resource managers are consolidated in linuxContainer structure. We can apply to unified operations (e.g., Apply(), Set(), Destroy()) using all of the registered resource managers.

Currently, cgroups manager is the single resource manager in libcontainer. Linux kernel 4.10 will introduce Intel RDT/CAT feature, the kernel interface is exposed via "resource control" filesystem, which is a cgroup-like interface. In order to support Intel RDT/CAT in libcontainer, we need a new resource manager (IntelRdt manager) outside cgroups.

crosbymichael · 2017-02-06T21:53:28Z

Thanks. Design looks much better than before.

xiaochenshen · 2017-02-09T04:55:37Z

@opencontainers/runc-maintainers
Any comments for this PR? Thank you.

xiaochenshen · 2017-03-06T10:17:15Z

@LK4D4 @cyphar @crosbymichael @hqhq @mrunalp @vishh
/cc @opencontainers/runc-maintainers

Who can help forward me the simplest steps to build runc in my forked repo (github.com/xiaochenshen/runc)? Thank you.

When I tried to rebase the code against latest runc master branch to resolve the conflict, I found the Godeps is replaced by vndr tool recently. I have tried to build with "make" in github.com/opencontainers/runc, it works. But in my forked dir, I got following errors. It looks like vndr related issues.

# make
go build -i -ldflags "-X main.gitCommit="291bf601105c97dc1aa631fdfb7fca63c947319b" -X main.version=1.0.0-rc2" -tags "seccomp" -o runc .
# github.com/xiaochenshen/runc
./restore.go:111: cannot use spec (type *"github.com/xiaochenshen/runc/vendor/github.com/opencontainers/runtime-spec/specs-go".Spec) as type *"github.com/opencontainers/runc/vendor/github.com/opencontainers/runtime-spec/specs-go".Spec in field value
./utils_linux.go:154: cannot use spec (type *"github.com/xiaochenshen/runc/vendor/github.com/opencontainers/runtime-spec/specs-go".Spec) as type *"github.com/opencontainers/runc/vendor/github.com/opencontainers/runtime-spec/specs-go".Spec in field value
make: *** [runc] Error 2

cyphar · 2017-03-07T14:16:08Z

@xiaochenshen Make sure that this project is cloned properly inside your GOPATH. In particular you should have something like this set up:

% mkdir ~/yourgopath
% export GOPATH=~/yourgopath
% mkdir -p $GOPATH/src/github.com/opencontainers/runc
% git clone https://github.com/opencontainers/runc $GOPATH/src/github.com/opencontainers/runc
# ...
% cd $GOPATH/src/github.com/opencontainers/runc
% make
# should work

xiaochenshen · 2017-03-08T02:56:27Z

@cyphar Thank you for your guide. It works with github.com/opencontainers/runc.
But this issue happens only in my forked repo github.com/xiaochenshen/runc which is also under $GOPATH ($GOPATH/src/github.com/xiaochenshen/runc).
Should I modify vendor.conf file?

Or git clone my repo https://github.com/xiaochen/runc into the directory $GOPATH/src/github.com/opencontainers/runc? It looks weird, to some extent:

git clone https://github.com/xiaochenshen/runc $GOPATH/src/github.com/opencontainers/runc

xiaochenshen · 2017-03-10T10:11:54Z

@crosbymichael /cc @opencontainers/runc-maintainers

To address your suggestion in opencontainers/runtime-spec#630 (review):
IntelRdt struct has been moved out of Resources and inside Linux now.

This PR has been updated accordingly.

rh-atomic-bot · 2017-04-05T18:40:47Z

269/277 passed on RHEL - Failed.
261/275 passed on CentOS - Failed.
275/276 passed on Fedora - Failed.
Log - https://aos-ci.s3.amazonaws.com/opencontainers/runc/runc-integration-tests-prs/314/fullresults.xml

xiaochenshen · 2017-04-05T18:54:49Z

ping @crosbymichael @cyphar @mrunalp @hqhq
/cc @rjnagal @vmarmol @dqminh @avagin

opencontainers/runtime-spec#630 + this PR to fix issue #433. And opencontainers/runtime-spec#630 has been merged.

Could you help code review this PR at your convenience? Thank you!
BTW, I just rebased the code to fix conflict with rootless cgroup manager.

yummypeng · 2017-04-19T12:09:31Z

libcontainer/resourcemanager/resource_manager_linux.go

+	// restore the object later
+	GetPaths() map[string]string
+
+	// Set the resource as configured


typo: Set->Sets

Thanks for finding the typo.

yummypeng · 2017-04-19T13:23:39Z

libcontainer/container_linux.go

@@ -432,9 +451,14 @@ func (c *linuxContainer) newSetnsProcess(p *Process, cmd *exec.Cmd, parentPipe,
 	if err != nil {
 		return nil, err
 	}
+	intelRdtPath, err := intelrdt.GetIntelRdtPath(c.ID())


Since you have already done this in the above state, err := c.currentState(), there is no need to do it again. Just reference state.IntelRdtPath in the following return &setnsProcess{...}.

Yes, you are right. I will fix it.

yummypeng · 2017-04-20T02:35:47Z

libcontainer/container_linux_test.go

+		if path == "" {
+			t.Fatal("intel rdt path should not be empty")
+		}
+		if intelRdtPath := path; intelRdtPath != expectedIntelRdtPath {


I guess that maybe you want to be consistent with the previous code, but I still think there is no need to define a new variable named intelRdtPath here. Maybe you can change path to intelRdtPath for clarity.

Yes, I will fix it.

yummypeng · 2017-04-20T06:30:54Z

libcontainer/intelrdt/intelrdt.go

+			return false
+		}
+
+		// If not mounted, we try to mount again:


Why? I think it's users' responsibility to guarantee that resctrl is mounted before they use it.

I agree with you now. I will remove this.

I was thinking that it is no harm for it. Unlike cgroups, Intel RDT resource control filesystem will likely not be mounted during boot-up by default in popular Linux OS distributions.

Shouldn't this be part of default mount info as we did for cgroups in https://github.com/opencontainers/runc/blob/master/libcontainer/specconv/example.go#L111-L116 and be handled as bind mount in mountToRootfs?

@hqhq
Unlike cgroup and cgroup filesystem, Intel RDT and resource control (resctrl) filesystem heavily depend on hardware/CPU support.

Likely, resctrl filesystem will not be mounted to rootfs either (1) h/w doesn't support Intel RDT or (2) kernel version older than 4.10, doesn't support Intel RDT.

I mean resctrl should also be bind mounted to container if we mount it on host by default, I didn't notice you're going to remove this, I'm OK with that, let users mount this on host and add that mount info to config.json specifically would be better.

@hqhq

let users mount this on host and add that mount info to config.json specifically would be better.

I agree that, if Intel RDT is supported by h/w and kernel, it is user's responsibility to mount the resctrl filesystem in host, and then to add mount info to config.json accordingly.

In most cases (e.g., non-Intel Xeon platforms), Intel RDT is not supported and resctrl filesystem is not enabled. In my opinion, we may not add "fixed" rectrl filesystem mount info into specs.Mount in libcontainer/specconv/example.go by default.

yummypeng · 2017-04-20T06:36:55Z

libcontainer/intelrdt/intelrdt.go

+	if err != nil {
+		return nil, err
+	}
+	numClosid, err := getIntelRdtParamUint(path, "num_closid")


typo: num_closid -> num_closids

Thanks for finding this typo.

yummypeng · 2017-04-20T07:40:03Z

libcontainer/intelrdt/intelrdt.go

+
+	// Returns Intel RDT "resource control" filesystem path to save in
+	// a state file and to be able to restore the object later
+	GetPath() string


If GetPaths() is not a suitable method in IntelRdtManager, is it possible that we delete it from Manager interface and just add it to cgroups.Manager ?

Currently I kept GetPaths() here intentionally. In future, we may add more Intel RDT resource constrains besides L3 cache. If it is really not needed at that time, I will reorg the Manager interface just like you suggested.

yummypeng · 2017-04-20T09:39:32Z

libcontainer/intelrdt/intelrdt.go

+
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	path, err := d.join(m.Id)


Maybe I am wrong, but have you ever thought about the scenario of docker-in-docker?

@yummypeng

For the nested containers case, I added comments in:
#1279 (comment)

xiaochenshen · 2017-04-21T11:58:24Z

@yummypeng Thank you for your help on code review. I will check up the issues one by one later.

yummypeng · 2017-04-24T08:33:50Z

I tried to add the configuration of intelRdt in my config.json, then run a container in an environment without support of intelRdt, and unexpectedly, the container was started successfully. Seems there needs an validation of Spec maybe in https://github.com/opencontainers/runc/blob/master/libcontainer/configs/validate/validator.go#L24.

yummypeng · 2017-04-27T09:26:39Z

libcontainer/intelrdt/intelrdt.go

+	// e.g., 0xfffff in root indicates the max bits of CBM is 20 bits,
+	// which mapping to entire L3 cache capacity. Some valid CBM values
+	// to set in a group: 0xf, 0xf0, 0x3ff, 0x1f00 and etc.
+	l3CacheSchema := container.IntelRdt.L3CacheSchema


This requires a check if container.IntelRdt == nil, otherwise this will panic.

Yes, you are right. I will add check in case of null pointer.

yummypeng · 2017-04-27T09:42:15Z

Since resctrl is a "cgroup-like" interface, why not mount it into container like cgroup does?

xiaochenshen · 2017-04-27T10:49:13Z

@yummypeng

Since resctrl is a "cgroup-like" interface, why not mount it into container like cgroup does?

There are discussions about that in obsolete #1198. You can refer to @crosbymichael and @cyphar 's comments: #1198 (comment) and #1198 (comment).

hqhq · 2017-05-02T06:55:34Z

libcontainer/intelrdt/intelrdt.go

+	isFlagSet, err := parseCpuInfoFile("/proc/cpuinfo")
+	if err != nil {
+		return false
+	}


Parsing mount points is time consuming, I think we can return if isFlagSet is false in most cases.

Thanks, I agree with you.

hqhq · 2017-05-02T07:08:48Z

libcontainer/resourcemanager/resource_manager_linux.go

@@ -0,0 +1,34 @@
+// +build linux


Is this gonna break Solaris and Windows?

@hqhq Which is better in your opinion? Thank you.
(1) Just remove this line (+// +build linux) in resource_manager_linux.go.
(2) Add new file resource_manager_unsupported.go for !linux cases:

// +build !linux package resourcemanager

After a second check, I think current implementation is fine, it's only imported by packages which are wrapped by linux build flag, sorry for the noise.

hqhq · 2017-05-02T07:15:56Z

libcontainer/intelrdt/intelrdt.go

+}
+
+func (raw *intelRdtData) join(id string) (string, error) {
+	path := filepath.Join(raw.root, id)


What about nested container cases? You can refer to the cgroup join function to see how cgroup handle this.

@hqhq

I was thinking if Intel RDT/CAT is possible to support nested containers case. Any more suggestions or comments?

Unlike cgroups' hierarchy, Intel RDT resource control filesystem supports only single level filesystem layout (see details in section "Intel RDT "resource control" filesystem hierarchy" in this PR).

The other limitation is that, Intel RDT/CAT only supports limited number of groups (directories), which is indicated in info/L3/num_closids. It is h/w limitation by nature.

hqhq · 2017-05-02T07:29:12Z

events.go

+type intelRdt struct {
+	// The read-only default "schema" in root, for reference
+	L3CacheSchemaRoot string `json:"l3CacheSchemaRoot,omitempty"`
+	L3CacheSchema     string `json:"l3CacheSchema,omitempty"`


json variables should be snake_case.

Got it. I will fix it.

hqhq · 2017-05-02T07:30:39Z

libcontainer/SPEC.md

+hardware and kernel support Intel RDT/CAT.
+
+In Linux kernel, it is exposed via "resource control" filesystem, which is a
+"cgroup-like" interface.


Can you specify the minimum linux kernel version that supports intelrdt feature?

No problem - 4.10 kernel.

+Linux kernel 4.10 introduces Intel RDT/CAT support, it is exposed via "resource control" filesystem...

crosbymichael · 2017-09-06T21:30:55Z

libcontainer/intelrdt/intelrdt.go

+// Check if Intel RDT is enabled
+func IsIntelRdtEnabled() bool {
+	// We have checked the flag before
+	if isIntelRdtEnabled {


isn't this racy? you should probably use a sync.Once for initialization of this bool

@crosbymichael
Thank you for good suggestion. I will fix it in follow-up PR.

I opened a new PR in #1589 to fix this issue.

crosbymichael · 2017-09-06T21:34:00Z

LGTM

Can you fix my comment in a follow up PR?

mrunalp · 2017-09-06T21:35:48Z

LGTM. We can fix up smaller issues in follow-on PRs. Thanks!

mrunalp · 2017-09-06T21:35:57Z

LGTM

xiaochenshen · 2017-09-07T01:58:26Z

@crosbymichael @mrunalp

Thank you for your code review.
I will open a new PR to fix remaining issues.

This is the follow-up PR of opencontainers#1279 to fix remaining issues: Use init() to avoid race condition in IsIntelRdtEnabled(). Add also rename some variables and functions. Signed-off-by: Xiaochen Shen <xiaochen.shen@intel.com>

xiaochenshen · 2017-09-13T04:47:12Z

@crosbymichael @hqhq @cyphar @mrunalp
/cc @runc-maintainers @yummypeng

I planned to add Intel RDT/CAT support for Docker. More information can be found in #433 .
But I am confused with the "vendor" configuration among Docker related projects (containerd, moby and docker-cli). I found a lot of dependency issues because recent runc changes have not been merged into these projects yet.

Who could share me the tips or documents about "vendor" tools?
Thank you in advance.

cyphar · 2017-09-13T18:09:04Z

There is a vendor.conf in the root of Docker's source tree. You need to just update the commit ID in that file, and then run vndr github.com/opencontainers/runc in the source tree (the tool is available at https://github.com/lk4d4/vndr).

xiaochenshen · 2017-09-14T05:52:22Z

@cyphar
Got it. Thank you.

Memory Bandwidth Allocation (MBA) is a resource allocation sub-feature of Intel Resource Director Technology (RDT) which is supported on some Intel Xeon platforms. Intel RDT/MBA provides indirect and approximate throttle over memory bandwidth for the software. A user controls the resource by indicating the percentage of maximum memory bandwidth. Hardware details of Intel RDT/MBA can be found in section 17.18 of Intel Software Developer Manual: https://software.intel.com/en-us/articles/intel-sdm In Linux 4.12 kernel and newer, Intel RDT/MBA is enabled by kernel config CONFIG_INTEL_RDT. If hardware support, CPU flags 'rdt_a' and 'mba' will be set in /proc/cpuinfo. Intel RDT "resource control" filesystem hierarchy: mount -t resctrl resctrl /sys/fs/resctrl tree /sys/fs/resctrl /sys/fs/resctrl/ |-- info | |-- L3 | | |-- cbm_mask | | |-- min_cbm_bits | | |-- num_closids | |-- MB | |-- bandwidth_gran | |-- delay_linear | |-- min_bandwidth | |-- num_closids |-- ... |-- schemata |-- tasks |-- <container_id> |-- ... |-- schemata |-- tasks For MBA support for `runc`, we will reuse the infrastructure and code base of Intel RDT/CAT which implemented in opencontainers#1279. We could also make use of `tasks` and `schemata` configuration for memory bandwidth resource constraints. The file `tasks` has a list of tasks that belongs to this group (e.g., <container_id>" group). Tasks can be added to a group by writing the task ID to the "tasks" file (which will automatically remove them from the previous group to which they belonged). New tasks created by fork(2) and clone(2) are added to the same group as their parent. The file `schemata` has a list of all the resources available to this group. Each resource (L3 cache, memory bandwidth) has its own line and format. Memory bandwidth schema: It has allocation values for memory bandwidth on each socket, which contains L3 cache id and memory bandwidth percentage. Format: "MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;..." The minimum bandwidth percentage value for each CPU model is predefined and can be looked up through "info/MB/min_bandwidth". The bandwidth granularity that is allocated is also dependent on the CPU model and can be looked up at "info/MB/bandwidth_gran". The available bandwidth control steps are: min_bw + N * bw_gran. Intermediate values are rounded to the next control step available on the hardware. For more information about Intel RDT kernel interface: https://www.kernel.org/doc/Documentation/x86/intel_rdt_ui.txt An example for runc: Consider a two-socket machine with two L3 caches where the minimum memory bandwidth of 10% with a memory bandwidth granularity of 10%. Tasks inside the container may use a maximum memory bandwidth of 20% on socket 0 and 70% on socket 1. "linux": { "intelRdt": { "memBwSchema": "MB:0=20;1=70" } } Signed-off-by: Xiaochen Shen <xiaochen.shen@intel.com>

Memory Bandwidth Allocation (MBA) is a resource allocation sub-feature of Intel Resource Director Technology (RDT) which is supported on some Intel Xeon platforms. Intel RDT/MBA provides indirect and approximate throttle over memory bandwidth for the software. A user controls the resource by indicating the percentage of maximum memory bandwidth. Hardware details of Intel RDT/MBA can be found in section 17.18 of Intel Software Developer Manual: https://software.intel.com/en-us/articles/intel-sdm In Linux 4.12 kernel and newer, Intel RDT/MBA is enabled by kernel config CONFIG_INTEL_RDT. If hardware support, CPU flags `rdt_a` and `mba` will be set in /proc/cpuinfo. Intel RDT "resource control" filesystem hierarchy: mount -t resctrl resctrl /sys/fs/resctrl tree /sys/fs/resctrl /sys/fs/resctrl/ |-- info | |-- L3 | | |-- cbm_mask | | |-- min_cbm_bits | | |-- num_closids | |-- MB | |-- bandwidth_gran | |-- delay_linear | |-- min_bandwidth | |-- num_closids |-- ... |-- schemata |-- tasks |-- <container_id> |-- ... |-- schemata |-- tasks For MBA support for `runc`, we will reuse the infrastructure and code base of Intel RDT/CAT which implemented in opencontainers#1279. We could also make use of `tasks` and `schemata` configuration for memory bandwidth resource constraints. The file `tasks` has a list of tasks that belongs to this group (e.g., <container_id>" group). Tasks can be added to a group by writing the task ID to the "tasks" file (which will automatically remove them from the previous group to which they belonged). New tasks created by fork(2) and clone(2) are added to the same group as their parent. The file `schemata` has a list of all the resources available to this group. Each resource (L3 cache, memory bandwidth) has its own line and format. Memory bandwidth schema: It has allocation values for memory bandwidth on each socket, which contains L3 cache id and memory bandwidth percentage. Format: "MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;..." The minimum bandwidth percentage value for each CPU model is predefined and can be looked up through "info/MB/min_bandwidth". The bandwidth granularity that is allocated is also dependent on the CPU model and can be looked up at "info/MB/bandwidth_gran". The available bandwidth control steps are: min_bw + N * bw_gran. Intermediate values are rounded to the next control step available on the hardware. For more information about Intel RDT kernel interface: https://www.kernel.org/doc/Documentation/x86/intel_rdt_ui.txt An example for runc: Consider a two-socket machine with two L3 caches where the minimum memory bandwidth of 10% with a memory bandwidth granularity of 10%. Tasks inside the container may use a maximum memory bandwidth of 20% on socket 0 and 70% on socket 1. "linux": { "intelRdt": { "memBwSchema": "MB:0=20;1=70" } } Signed-off-by: Xiaochen Shen <xiaochen.shen@intel.com>

This was referenced Jan 17, 2017

libcontainer: add support for Intel RDT/CAT in runc #1198

Closed

Proposal: Intel RDT/CAT support for OCI/runc and Docker #433

Closed

xiaochenshen force-pushed the rdt-cat-resource-manager-v1 branch from fccc503 to f0a0fe2 Compare January 17, 2017 21:18

xiaochenshen force-pushed the rdt-cat-resource-manager-v1 branch 2 times, most recently from 04e7cec to aaa98cb Compare February 9, 2017 04:47

xiaochenshen force-pushed the rdt-cat-resource-manager-v1 branch from aaa98cb to abf51f3 Compare February 9, 2017 05:00

xiaochenshen force-pushed the rdt-cat-resource-manager-v1 branch 2 times, most recently from 2d4d43f to e414983 Compare March 8, 2017 08:06

xiaochenshen mentioned this pull request Mar 9, 2017

specs-go/config: add Intel RDT/CAT Linux support opencontainers/runtime-spec#630

Merged

xiaochenshen force-pushed the rdt-cat-resource-manager-v1 branch from e414983 to d1b743f Compare March 10, 2017 10:08

xiaochenshen force-pushed the rdt-cat-resource-manager-v1 branch from d1b743f to f9f1347 Compare March 10, 2017 10:18

xiaochenshen force-pushed the rdt-cat-resource-manager-v1 branch from f9f1347 to 4752fd2 Compare April 5, 2017 18:24

xiaochenshen mentioned this pull request Apr 11, 2017

image-index: Document Linux cpuinfo flags (on x86) for platform.features opencontainers/image-spec#631

Closed

yummypeng reviewed Apr 20, 2017

View reviewed changes

yummypeng reviewed Apr 27, 2017

View reviewed changes

wking mentioned this pull request May 1, 2017

config-linux: RFC 2119 wording for intelRdt opencontainers/runtime-spec#787

Merged

hqhq requested changes May 2, 2017

View reviewed changes

crosbymichael reviewed Sep 6, 2017

View reviewed changes

mrunalp merged commit 5274430 into opencontainers:master Sep 6, 2017

xiaochenshen mentioned this pull request Sep 8, 2017

libcontainer: intelrdt: use init() to avoid race condition #1589

Merged

xiaochenshen mentioned this pull request Sep 9, 2017

libcontainer: intelrdt: add update command support #1590

Merged

xiaochenshen mentioned this pull request Sep 18, 2017

Proposal: Intel RDT/MBA support for OCI/runc and Docker #1596

Open

xiaochenshen mentioned this pull request Oct 31, 2017

libcontainer: intelrdt: add support for Intel RDT/MBA in runc #1632

Merged

This was referenced Dec 5, 2017

phrase in github CodeJuan/blog#52

Open

[feature request] support Intel RDT AliyunContainerService/pouch#269

Closed

cyphar mentioned this pull request Feb 24, 2018

VERSION: release v1.0.0-rc5 #1739

Merged

xiaochenshen mentioned this pull request Sep 5, 2018

config-linux: add Intel RDT CLOS name sharing support opencontainers/runtime-spec#988

Merged

libcontainer: add support for Intel RDT/CAT in runc #1279

libcontainer: add support for Intel RDT/CAT in runc #1279

Conversation

xiaochenshen commented Jan 17, 2017 • edited Loading

xiaochenshen commented Jan 17, 2017

crosbymichael commented Feb 6, 2017

xiaochenshen commented Feb 9, 2017

xiaochenshen commented Mar 6, 2017

cyphar commented Mar 7, 2017

xiaochenshen commented Mar 8, 2017 • edited Loading

xiaochenshen commented Mar 10, 2017

rh-atomic-bot commented Apr 5, 2017

xiaochenshen commented Apr 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaochenshen commented Apr 21, 2017

yummypeng commented Apr 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yummypeng commented Apr 27, 2017

xiaochenshen commented Apr 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaochenshen May 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crosbymichael commented Sep 6, 2017 • edited by caniszczyk Loading

mrunalp commented Sep 6, 2017 • edited by caniszczyk Loading

mrunalp commented Sep 6, 2017

xiaochenshen commented Sep 7, 2017

xiaochenshen commented Sep 13, 2017 • edited Loading

cyphar commented Sep 13, 2017

xiaochenshen commented Sep 14, 2017

xiaochenshen commented Jan 17, 2017 •

edited

Loading

xiaochenshen commented Mar 8, 2017 •

edited

Loading

xiaochenshen commented Apr 5, 2017 •

edited

Loading

xiaochenshen May 3, 2017 •

edited

Loading

crosbymichael commented Sep 6, 2017 •

edited by caniszczyk

Loading

mrunalp commented Sep 6, 2017 •

edited by caniszczyk

Loading

xiaochenshen commented Sep 13, 2017 •

edited

Loading