build: update minimum kernel version to 3.10.0

[gibfahn]

Also updates glibc version to 2.17. This means the minimum supported
distros are RHEL/CentOS 7 and Ubuntu 14.04.

Distro	Kernel	glibc
Current	2.6.32	2.12
New	3.10.0	2.17
RHEL6	2.6.32	2.12
Ubuntu 12.04	3.2	2.15
RHEL7	3.10.0	2.17
Ubuntu 14.04	3.13	2.19
Ubuntu 16.04	4.4	2.23
Latest	4.12.5	2.26

Refs: nodejs/build#809

Checklist

• documentation is changed or added
• commit message follows commit guidelines

Affected core subsystem(s)

build

cc/ @nodejs/build @nodejs/release @bnoordhuis @seishun @rvagg

[gibfahn]

This will be for v9.x onwards. v10.x goes EoL in April 2021.

It should be possible to work around the Node 8.x glibc issues in nodejs/build#809, if so this won't need to be backported to 8.x.

[seishun]

Is it really necessary to bump the minimum kernel version?

+1 otherwise.

[gibfahn]

Is it really necessary to bump the minimum kernel version?

Not sure, we can't bump after we ship, so it's hard to say in advance. I think the real question is what the reasonable minimum supported level is.

Looking at the Red Hat lifecycle docs, it looks like RHEL 6.7's extended support ends December 31, 2018. Node 10.0.0 will ship in October 2018. EDIT: Looks like I got that wrong, that's the end of support for 6.7. As @cosinusoidally points out, it should be November 30, 2020.

The other issue is that the glibc bump means that people probably won't want to be using RHEL/CentOS 6. However I expect that many people will simply check the kernel version, which may lead to people thinking that RHEL 6 is still supported.

Also we don't actually have to move up our build/test machines straight away, we can continue to ship binaries that run on RHEL 6 until something forces us to upgrade. This is what we did with RHEL 5, we stopped officially supporting it in v4.x, but still build v6.x binaries on it (see nodejs/build#718).

[mscdex]

Can't we just use the V8 patch by @bnoordhuis instead? Unfortunately some of us are stuck with older systems that cannot be immediately upgraded for a variety of reasons. For example. right now the devtoolset packages for CentOS/RHEL allow us to compile the most recent node versions on CentOS/RHEL 5.

[gibfahn]

Can't we just use the V8 patch by @bnoordhuis instead?

For Node 8.x sure, for Node 10.x it's an open question (which is why I added the discuss label).

One thing that would be relevant is a link to a V8 supported platforms page, I can't find one. All I can find is this (which doesn't list a kernel/distro version) and this, which says that support ended for Ubuntu 12.04 ended in April 2016 (Ubuntu 12.04 has a later kernel version than RHEL 6).

Unfortunately some of us are stuck with older systems that cannot be immediately upgraded for a variety of reasons.

If we're talking about production systems, is running Node 8.x till April 2021 not enough time to upgrade (not a rhetorical question, that's what we need to discuss)?

If we're talking about Node development then I can see that that doesn't give much time, but I'd assume most developers can update their dev machines past RHEL 6.

It'd be great if we had some metrics about what percentage of customers were on RHEL 6.

[addaleax]

@gibfahn Sorry if this a question with an obvious answer, but is the kernel version actually relevant here or is it just used as a proxy for OS versions?

[gibfahn]

@gibfahn Sorry if this a question with an obvious answer, but is the kernel version actually relevant here or is it just used as a proxy for OS versions?

Not obvious to me! From a practical perspective most people aren't building or updating their kernels, so Distro versions seem more important. I think (winging it here) that as long as Node doesn't directly make syscalls, and does everything through glibc, kernel compatibility doesn't matter if you have the right glibc. EDIT: I'm wrong (didn't know about libuv), see #14795 (comment).

[bnoordhuis]

The kernel version is relevant to libuv, it makes direct syscalls. (And usually has fallbacks for kernels that don't support them but not always.)

[Trott]

This needs two CTC reviews 'cause semver-major. (Whee! A doc-only semver-major change!)

@nodejs/ctc

[mscdex]

If we're talking about production systems, is running Node 8.x till April 2021 not enough time to upgrade (not a rhetorical question, that's what we need to discuss)?

It's one thing if you're in direct control of such systems, but it's another if you're not. It's also a lot easier if you are dealing with virtualized systems vs. physical systems spread out everywhere (as far as troubleshooting goes if upgrades go awry). Additionally there are some cases where older hardware needs to be supported and whose drivers are only supported for older kernels (whether they technically work with newer kernels does not really matter), etc.

With all of those things in mind, it can be a long process when moving off of such older systems...

[rvagg]

Are we working on concrete concerns here or just predicting future problems based on our current experience with the nodejs/build#809 problem?

Re kernel version in particular I'd like to hear from libuv what their minimum is and just use that rather than referencing known operating systems. It seems like libuv is the only reason we mention kernel version in there at all anyway.

For both kernel and libc versions, my preference is to be as conservative as possible here and I'd rather not bump at all until we know there's going to be a problem. Since we don't currently don't major bump V8 or libuv in LTS releases we have a whole Current cycle to determine what the minimums should be. Perhaps we just need to loosen our Current policy so that we could change minimums pre-LTS if absolutely required? That way we could know for sure what the minimum is for a given LTS and stick to that when it happens. Being pre-emptive without any reason is going to knock out huge numbers of users that are forced to deploy on ancient systems for various reasons—e.g. there seems to be a lot of this in China, a lot of IoT and other limited devices have custom operating systems that are near impossible to change or become unsupported by their vendors.

So -1 in general but I don't want to be a blocker if I can't convince anyone else.

[seishun]

• LTS for node 8.x ends December 2019.
• RHEL 6.7's extended support ends December 31, 2018

I'm not sure what the holdup is. Surely we don't want to support an EOL distro in node 10.x?

Being pre-emptive without any reason

CentOS 6 is already causing issues, see nodejs/build#809. Who knows how many more patches it will take to keep supporting it? And it's clear that V8 maintainers aren't too keen on working around bugs that were fixed years ago.

[bnoordhuis]

Re kernel version in particular I'd like to hear from libuv what their minimum is and just use that rather than referencing known operating systems.

https://github.com/libuv/libuv/blob/v1.14.0/SUPPORTED_PLATFORMS.md - for Linux, it's kernel >= 2.6.32 + glibc >= 2.12 at the time of writing.

[rvagg]

CentOS 6 is already causing issues, see nodejs/build#809. Who knows how many more patches it will take to keep supporting it? And it's clear that V8 maintainers aren't too keen on working around bugs that were fixed years ago.

Because 8.x is still Current, we're not going to be upgrading V8 soon (at all? is 6.1 even going to be able to make it in?). When 8.x goes LTS it'll be stable V8 and libuv and those problems won't persist.

I'm suggesting we take the same approach with 9.x and 10.x, deal with specifics as they come, adapt if we can, perhaps it'll take a patch or two. V8 doesn't have to support older kernels because almost nobody does desktop on ancient linuxes. Our deploy base is completely different and we ought not confuse our audience with theirs. But since we also have V8 representatives amongst us and a lot of interest on their side in supporting Node.js then I imagine there's a good possibility that our patches will make it upstream—they're just not aware of the breakage because they don't test as far back, it doesn't mean they don't want to support as far back.

[seishun]

they're just not aware of the breakage because they don't test as far back, it doesn't mean they don't want to support as far back.

Quoting their response from https://chromium-review.googlesource.com/c/612351:

In general, I'm a little uncomfortable with working around bug that was fixed in glibc four years ago, given that the trap handler code is so security-sensitive. Just changing the data type seems safe enough though; whether it's one byte, four or eight is pretty inconsequential to the overall design.

It might have been different if the change were less trivial.

[gibfahn]

Since we don't currently don't major bump V8 or libuv in LTS releases we have a whole Current cycle to determine what the minimums should be. Perhaps we just need to loosen our Current policy so that we could change minimums pre-LTS if absolutely required? That way we could know for sure what the minimum is for a given LTS and stick to that when it happens.

I don't see an issue with waiting till next March (given that v10.x goes Current in April). At the same time I'm not sure what difference it makes to do that. If 9.x doesn't theoretically support these platforms, then that gives people a nudge to start upgrading. We don't actually have to break builds on those platforms (we can still include them in CI).

I'm suggesting we take the same approach with 9.x and 10.x, deal with specifics as they come, adapt if we can, perhaps it'll take a patch or two.

This seems like a different approach to what you proposed earlier. I don't think we should only bump supported platform levels when forced to, we should look at lifecycles and make a decision. Yes we could bump in mid-stream if we have to, but that's worse from a user perspective than just stating that we no longer support it from the start of that LTS version.

Being pre-emptive without any reason is going to knock out huge numbers of users that are forced to deploy on ancient systems for various reasons.

To be quite honest, I think all this is largely academic, as I don't think people will be able to find our supported platforms list. It should be front and centre on the download page. I've yet to meet anyone not associated with node core who has found it.

[gibfahn]

@rvagg how about this for a proposal:

Before each release we look at platform support, and drop things that are EoL (if we want to). We don't drop them from CI until absolutely necessary, but the fact that we don't state support for them means that people know that they should be upgrading if possible.

Being pre-emptive without any reason is going to knock out huge numbers of users that are forced to deploy on ancient systems for various reasons—

I don't think knock out is the right word. Node 8.x should continue to work for existing apps for a very long time, and if you're not upgrading your kernel then you're probably not that worried about security patches. We're freezing users at a particular LTS version.

[refack]

Three point IMHO should be considered:

1. The reverse rug-pulling - platforms that will go EOL within an LTS's lifetime. AFAICT that the main motivation for this PR.
2. 6cc6379 - Effectively disavowing support for running node on EOL platforms. As far as I remember, this is introduced to be our escape hatch to stop supporting EOL platforms even mid LTS period.
   So maybe we don't need to be preemptive?
3. My (new) understanding of the target user space -
   1. Systems that "just work" and are highly unlikely to upgrade (at most only security patches)
   2. "Blessed" systems, that can't upgrade the OS, but do want latest node - The most relevant part of this discussion.

[seishun]

We don't drop them from CI until absolutely necessary

I find this proposition slightly puzzling. Let's say someone submits a PR that works on all platforms except one EoL platform. What happens next?

• If we block the PR because of it, then the supported platform list is misleading. Presumably it would be quite frustrating for the contributor.
• If we finally drop the platform from CI, then what was the point in keeping it there in the first place?

[gibfahn]

I find this proposition slightly puzzling. Let's say someone submits a PR that works on all platforms except one EoL platform. What happens next?

We make a decision based on the contents of the PR. If it's easy to work around we can work around it, otherwise we should consider whether it's worth dropping the platform. I assume this will be pretty rare in any case.

If we finally drop the platform from CI, then what was the point in keeping it there in the first place?

You might have given someone on an EoL platform another year of working node updates, during which time they have hopefully seen the deprecation notice and started trying to upgrade (or prepared to EoL their hardware). The question is why not support something if it's low-maintenance. Dropping support for it allows us to flexibly drop it when it becomes high-maintenance.

Instead of "dropping support", I guess we could move those platforms to Experimental instead. It has the same result (don't rely on this continuing to work), but maybe more accurately reflects the platforms' status.

I'd note that "dropping support but actually still supporting" is what we did for CentOS 5 levels for Node 4.x and 6.x, seemed okay there.

[gibfahn]

It'd be a good idea to make a formal part of the semver-major release process to get Build to weigh in on minimum platform support.

@rvagg this sounds like a great idea, but how do you suggest we actually do that? Create an issue and assign to the relevant milestone?

[rvagg]

@gibfahn how about we amend doc/releases.md and put a note in there about it being a necessary part of semver-major bumps. There's already a bit in there about bumping NODE_MODULE_VERSION, that could be refactored into a new section about "New semver-major releases" and also include a step that requires the releaser to open an issue against the nodejs/build repo to get them to sign off on the minimum build requirements for that new branch. It's going to add overhead and delay new semver-major releases but if the person doing it knows it's part of the step then they can get it started when they're doing release candidates.
Someone want to volunteer for updating that doc?

[BridgeAR]

@rvagg I somewhat do not understand why this could not happen proactively by the build-wg. It should be clear to everyone when a new semver-major is going to be released or planned for and in that case the build-wg could just provide the info for the minimum support and open a PR for that semver-major step to also document this in case something changed.
In that case we are somewhat back to the beginning of this PR though.

[mhdawson]

@gibfahn did you check that the power/zlinux versions of ubuntu 14.04 have the same kernel/glibc levels ?

[gibfahn]

@gibfahn did you check that the power/zlinux versions of ubuntu 14.04 have the same kernel/glibc levels ?

I have now, they're the same.

[BridgeAR]

Is there any progress here? I would like to go ahead and close this otherwise.

[gibfahn]

Is there any progress here? I would like to go ahead and close this otherwise.

This needs to land at some point. I'd rather leave it open as we should be reevaluating it for each major version (even if it doesn't land).

[seishun]

Are there any hard objections to this landing in 10.x?

[BridgeAR]

I put it on the TSC agenda to decide if it should land on 10.x or later.

[gibfahn]

Related to discussion in nodejs/build#961 (comment).

[cosinusoidally]

Also Centos 6 will be around until the and of 2020.

[ChALkeR]

Debian oldstable (jessie) has 3.16.0.

[Trott]

I removed the tsc-* labels. The consensus at the meeting today was that the Build WG is empowered to make this decision. If consensus (or whatever the Build WG decision-making process is) does not result in a decision, it can always be put back to the TSC. But Build does not need TSC approval to make this change.

(And if I'm summarizing incorrectly, please set the record straight. /cc @gibfahn )

[seishun]

Kernel page-table isolation (which mitigates Meltdown) has been backported to 4.14.11. It seems unlikely that it's going to be backported to any earlier branches. We probably don't want to run CI on insecure kernel versions, so the minimum kernel version should be bumped to 4.14.11.

[gibfahn]

the minimum kernel version should be bumped to 4.14.11.

Isn't Ubuntu 17.10 on 4.13? That would mean dropping support for all released versions of RHEL and Ubuntu right? Seems unlikely to me. Are you saying you don't think Red Hat are going to backport the security fixes to RHEL 6 and 7?

We probably don't want to run CI on insecure kernel versions

Why not? All code is reviewed by collaborators before being run through CI. I understand wanting to drop EoL platforms (as you mentioned in nodejs/build#1070), but the worst that happens if rogue code gets into ci is that we have to wipe and restart everything. ci-release is completely isolated, with different machines, keys etc.

[seishun]

Looks like the fix was backported to 3.10.0 in CentOS 7 and to 4.9.65 in Debian. So I guess it's impossible to tell whether KPTI is present from kernel version alone. Never mind then.

All code is reviewed by collaborators before being run through CI.

Correct, but the risk (and therefore the responsibility) is higher when the rogue code can read arbitrary memory. I assume that was the motivation for nodejs/build#1070.

[BridgeAR]

Closing due to the stale progress. I guess in case there is a conclusion in the build WG a new PR is created almost faster than rebasing.

If anyone feels differently: please reopen.