cpu: Support processor-less (memory-only) NUMA nodes #734

knweiss · 2017-11-09T11:32:46Z

This PR fixes the following crash on systems with processor-less (memory-only) NUMA node(s):

panic: runtime error: index out of range
goroutine 40 [running]:
github.com/prometheus/node_exporter/collector.(*cpuCollector).updateCPUfreq(0xc420214690, 0xc4202f0900, 0x0, 0x0)
.../go/packages/src/github.com/prometheus/node_exporter/collector/cpu_linux.go:164 +0x13ad
github.com/prometheus/node_exporter/collector.(*cpuCollector).Update(0xc420214690, 0xc4202f0900, 0x106da40, 0x0)
.../go/packages/src/github.com/prometheus/node_exporter/collector/cpu_linux.go:94 +0x60
github.com/prometheus/node_exporter/collector.execute(0xad6ab8, 0x3, 0x1035e40, 0xc420214690, 0xc4202f0900)
.../go/packages/src/github.com/prometheus/node_exporter/collector/collector.go:131 +0x7e
github.com/prometheus/node_exporter/collector.nodeCollector.Collect.func1(0xc4202f0900, 0xc4202f2190, 0xad6ab8, 0x3, 0x1035e40, 0xc420214690)
.../go/packages/src/github.com/prometheus/node_exporter/collector/collector.go:122 +0x53
created by github.com/prometheus/node_exporter/collector.nodeCollector.Collect
.../go/packages/src/github.com/prometheus/node_exporter/collector/collector.go:121 +0x109

There are also two additional cleanup commits.

Processor-less (memory-only) NUMA nodes exist e.g. in systems that use Intel Optane drives for RAM expansion using Intel Memory Drive Technology (IMDT).

IMDT RAM expansion supports two modes:

Unify Remote Memory domains : present a processor-less (memory-only)
NUMA domain, which is the default
Expand local memory domains : to expand each processor’s memory domain with a portion of the memory made available by Optane and IMDT

This commit fixes a crash in the first case (when cpulist is empty).

Here's an example of such a system:

$ numastat -m|head -n5

Per-node system memory usage (in MBs):
                          Node 0          Node 1          Node 2           Total
                 --------------- --------------- --------------- ---------------
MemTotal               118239.56       130816.00       464384.00       713439.56

$ for i in {0..2}; do echo -n "$i: " ; cat /sys/bus/node/devices/node$i/cpulist ; done
0: 0-7,16-23
1: 8-15,24-31
2:

$ /opt/vsmp/bin/vsmpversion -vvv
Memory Drive Technology: 8.2.1455.74 (Sep 28 2017 13:09:59)
System configuration:
    Boards:      3
       1 x Proc. + I/O + Memory
       2 x NVM devices (Intel SSDPED1K375GAQ)
    Processors:  2, Cores: 16, Threads: 32
        Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz Stepping 01
    Memory (MB): 713472 (of 977450), Cache: 251416, Private: 12562
       1 x 249088MB   [262036/   678/12270]
       1 x 232192MB   [357707/125369/  146]  82:00.0#1
       1 x 232192MB   [357707/125369/  146]  83:00.0#1

Processor-less (memory-only) NUMA nodes exist e.g. in systems that use Intel Optane drives for RAM expansion using Intel Memory Drive Technology (IMDT). IMDT RAM expansion supports two modes: * "Unify Remote Memory domains": present a processor-less (memory-only) NUMA domain, which is the default * "Expand local memory domains": to expand each processor’s memory domain with a portion of the memory made available by Optane and IMDT This commit fixes a crash in the first case (when "cpulist" is empty). Here's an example of such a system: $ numastat -m|head -n5 Per-node system memory usage (in MBs): Node 0 Node 1 Node 2 Total --------------- --------------- --------------- --------------- MemTotal 118239.56 130816.00 464384.00 713439.56 $ for i in {0..2}; do echo -n "$i: " ; cat /sys/bus/node/devices/node$i/cpulist ; done 0: 0-7,16-23 1: 8-15,24-31 2: $ /opt/vsmp/bin/vsmpversion -vvv Memory Drive Technology: 8.2.1455.74 (Sep 28 2017 13:09:59) System configuration: Boards: 3 1 x Proc. + I/O + Memory 2 x NVM devices (Intel SSDPED1K375GAQ) Processors: 2, Cores: 16, Threads: 32 Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz Stepping 01 Memory (MB): 713472 (of 977450), Cache: 251416, Private: 12562 1 x 249088MB [262036/ 678/12270] 1 x 232192MB [357707/125369/ 146] 82:00.0#1 1 x 232192MB [357707/125369/ 146] 83:00.0#1

SuperQ · 2017-11-09T12:29:13Z

I think this is missing an update to the end-to-end test output fixture.

knweiss · 2017-11-09T14:09:50Z

@SuperQ I'm obviously missing something. I don't expect any changes in the output fixture as the memory-only NUMA node should be skipped and only leave an debug output message in the log.

Unfortunately, I can't see the buildkite error output. However, the end-to-end test works on my machine.

Looking the node_exporter project page it seems that the master branch is broken, isn't it? The buildkite badge has status "build: failing"

SuperQ · 2017-11-09T15:27:56Z

Ahh, sorry, I wasn't looking at the change that closely, I thought there would be some new metrics output.

I'll look into the buildkite issues.

SuperQ

LGTM

SuperQ · 2017-11-10T10:02:55Z

The buildkite thing seems to be related to the build agent being broken.

* cpu: Support processor-less (memory-only) NUMA nodes Processor-less (memory-only) NUMA nodes exist e.g. in systems that use Intel Optane drives for RAM expansion using Intel Memory Drive Technology (IMDT). IMDT RAM expansion supports two modes: * "Unify Remote Memory domains": present a processor-less (memory-only) NUMA domain, which is the default * "Expand local memory domains": to expand each processor’s memory domain with a portion of the memory made available by Optane and IMDT This commit fixes a crash in the first case (when "cpulist" is empty). Here's an example of such a system: $ numastat -m|head -n5 Per-node system memory usage (in MBs): Node 0 Node 1 Node 2 Total --------------- --------------- --------------- --------------- MemTotal 118239.56 130816.00 464384.00 713439.56 $ for i in {0..2}; do echo -n "$i: " ; cat /sys/bus/node/devices/node$i/cpulist ; done 0: 0-7,16-23 1: 8-15,24-31 2: $ /opt/vsmp/bin/vsmpversion -vvv Memory Drive Technology: 8.2.1455.74 (Sep 28 2017 13:09:59) System configuration: Boards: 3 1 x Proc. + I/O + Memory 2 x NVM devices (Intel SSDPED1K375GAQ) Processors: 2, Cores: 16, Threads: 32 Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz Stepping 01 Memory (MB): 713472 (of 977450), Cache: 251416, Private: 12562 1 x 249088MB [262036/ 678/12270] 1 x 232192MB [357707/125369/ 146] 82:00.0#1 1 x 232192MB [357707/125369/ 146] 83:00.0#1 * cpu: rename some variables (pkg => node) * cpu: Use %v not %q in log.Debugf() format strings

* cpu: Support processor-less (memory-only) NUMA nodes (#734) * cpu: Support processor-less (memory-only) NUMA nodes Processor-less (memory-only) NUMA nodes exist e.g. in systems that use Intel Optane drives for RAM expansion using Intel Memory Drive Technology (IMDT). IMDT RAM expansion supports two modes: * "Unify Remote Memory domains": present a processor-less (memory-only) NUMA domain, which is the default * "Expand local memory domains": to expand each processor’s memory domain with a portion of the memory made available by Optane and IMDT This commit fixes a crash in the first case (when "cpulist" is empty). Here's an example of such a system: $ numastat -m|head -n5 Per-node system memory usage (in MBs): Node 0 Node 1 Node 2 Total --------------- --------------- --------------- --------------- MemTotal 118239.56 130816.00 464384.00 713439.56 $ for i in {0..2}; do echo -n "$i: " ; cat /sys/bus/node/devices/node$i/cpulist ; done 0: 0-7,16-23 1: 8-15,24-31 2: $ /opt/vsmp/bin/vsmpversion -vvv Memory Drive Technology: 8.2.1455.74 (Sep 28 2017 13:09:59) System configuration: Boards: 3 1 x Proc. + I/O + Memory 2 x NVM devices (Intel SSDPED1K375GAQ) Processors: 2, Cores: 16, Threads: 32 Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz Stepping 01 Memory (MB): 713472 (of 977450), Cache: 251416, Private: 12562 1 x 249088MB [262036/ 678/12270] 1 x 232192MB [357707/125369/ 146] 82:00.0#1 1 x 232192MB [357707/125369/ 146] 83:00.0#1 * cpu: rename some variables (pkg => node) * cpu: Use %v not %q in log.Debugf() format strings * Release v0.15.2

* cpu: Support processor-less (memory-only) NUMA nodes Processor-less (memory-only) NUMA nodes exist e.g. in systems that use Intel Optane drives for RAM expansion using Intel Memory Drive Technology (IMDT). IMDT RAM expansion supports two modes: * "Unify Remote Memory domains": present a processor-less (memory-only) NUMA domain, which is the default * "Expand local memory domains": to expand each processor’s memory domain with a portion of the memory made available by Optane and IMDT This commit fixes a crash in the first case (when "cpulist" is empty). Here's an example of such a system: $ numastat -m|head -n5 Per-node system memory usage (in MBs): Node 0 Node 1 Node 2 Total --------------- --------------- --------------- --------------- MemTotal 118239.56 130816.00 464384.00 713439.56 $ for i in {0..2}; do echo -n "$i: " ; cat /sys/bus/node/devices/node$i/cpulist ; done 0: 0-7,16-23 1: 8-15,24-31 2: $ /opt/vsmp/bin/vsmpversion -vvv Memory Drive Technology: 8.2.1455.74 (Sep 28 2017 13:09:59) System configuration: Boards: 3 1 x Proc. + I/O + Memory 2 x NVM devices (Intel SSDPED1K375GAQ) Processors: 2, Cores: 16, Threads: 32 Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz Stepping 01 Memory (MB): 713472 (of 977450), Cache: 251416, Private: 12562 1 x 249088MB [262036/ 678/12270] 1 x 232192MB [357707/125369/ 146] 82:00.0#1 1 x 232192MB [357707/125369/ 146] 83:00.0#1 * cpu: rename some variables (pkg => node) * cpu: Use %v not %q in log.Debugf() format strings

knweiss added 3 commits November 9, 2017 12:26

cpu: rename some variables (pkg => node)

0dc9160

cpu: Use %v not %q in log.Debugf() format strings

3da7c0f

SuperQ self-requested a review November 9, 2017 12:29

SuperQ approved these changes Nov 10, 2017

View reviewed changes

SuperQ merged commit a8d7d11 into prometheus:master Nov 10, 2017

knweiss deleted the processorless_numa_node branch November 12, 2017 11:37

knweiss mentioned this pull request Nov 27, 2017

Package "/sys/bus/node/devices/node0" CPU "0" is missing package_throttle / panic: runtime error: index out of range #745

Closed

SuperQ mentioned this pull request Dec 5, 2017

Release 0.15.2 #751

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cpu: Support processor-less (memory-only) NUMA nodes #734

cpu: Support processor-less (memory-only) NUMA nodes #734

knweiss commented Nov 9, 2017

SuperQ commented Nov 9, 2017

knweiss commented Nov 9, 2017 •

edited

Loading

SuperQ commented Nov 9, 2017

SuperQ left a comment

SuperQ commented Nov 10, 2017

cpu: Support processor-less (memory-only) NUMA nodes #734

cpu: Support processor-less (memory-only) NUMA nodes #734

Conversation

knweiss commented Nov 9, 2017

SuperQ commented Nov 9, 2017

knweiss commented Nov 9, 2017 • edited Loading

SuperQ commented Nov 9, 2017

SuperQ left a comment

Choose a reason for hiding this comment

SuperQ commented Nov 10, 2017

knweiss commented Nov 9, 2017 •

edited

Loading