deps: upgrade npm to 4.0.1

[zkat]

Checklist

• make -j8 test (UNIX), or vcbuild test nosign (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

Affected core subsystem(s)

• deps

Description of change

Hey y'all! #8977 was created a while back to track stuff surrounding this release, and I think it's a good decision to wait to do a minor bump later, rather than rush this release out in any way. Thanks a bunch for your patience!

As you can see, this is a semver-major bump for npm. That said, the actual impact of the release to the ecosystem is relatively small. Most certainly much smaller in impact than npm@3 was. We're still sniffing out a couple of issues and might even put together an npm@4.0.2 soon to iron that out before npm@4 becomes latest.

THIS IS NOT A FINAL RELEASE -- as mentioned in #8977, this is a preview of npm@4 so y'all can start testing and taking a look at changes and make sure everything looks good before considering including it with official node releases.

One more note: With the release of npm@4, we've decided to move both npm@2 and npm@3 to maintenance: that means we're unlikely to continue adding features to either of those, but will include critical patches, specially security patches, if something comes up. Regular bugfixes will not be done, as it's much better for our team to be able to focus on the latest release which we're standing behind. As far as Node Core is concerned, though, npm@2 continues what we consider to be a sort of LTS release for y'all's sake. Maintenance doesn't mean it's no longer maintained, and users should continue to have a working experience with npm@2 until node@4 leaves LTS. npm@3.10.9, which I'll be downstreaming soon, might end up being our last npm@3 release.

So without further ado, npm@4!

Breaking Changes

• npm search rewritten to stream results, and no longer supports sorting.
• npm scripts no longer prepend the path of the node executable used to run npm before running scripts. A --scripts-prepend-node-path option has been added to configure this behavior. (/cc @addaleax)
• prepublish has been deprecated, replaced by prepare. A prepublishOnly script has been temporarily added, which will only run on npm publish. NOTE: This change only affects users publishing new versions of their packages and developers installing through git and local deps. Registry installs should be (mostly) unaffected by this change.
• npm outdated exits with exit code 1 if it finds any outdated packages. (/cc @watilde)
• npm tag has been removed after a deprecation cycle. Use npm dist-tag.
• Partial shrinkwraps are no longer supported. npm-shrinkwrap.json is considered a complete installation manifest except for devDependencies. This will affect certain projects that relied heavily on this feature, most notably hapi.
• devDependencies are now included in npm-shrinkwrap.json by default. This should make the transition to npm@5 easier.

Other Notable Changes

• npm now sends two extra http headers to the registry (npm/npm#14129):
  • Npm-In-CI - whether npm was run within a Continuous Integration environment.
  • Npm-Scope - the scope of the toplevel package this installation is for. For example, if you have a @node/foo package, all dependency requests for that package will include @node in the header, even if those dependencies themselves are not scoped or are for a different scope.

Changelogs

• v3.10.9
• v4.0.0
• v4.0.1

r: @Fishrock123
r: @addaleax
r: @jasnell

[Fishrock123]

npm@3.10.9, which I'll be downstreaming soon, might end up being our last npm@3 release.

You must be joking.

Users of v6 which just went into LTS are now (by default) stuck with an npm that may never receive any updates??

[Fishrock123]

Is this intended to be a patch-level release on our end? We'd need to have some discussions about that and I'm not sure how likely we are to accept it as such.

I realize you say npm@3 has gone into maintenance and will include critical patches, specially security patches, if something comes up. but it doesn't sound particularly promising that npm@3 will have security patches or the like from the rest of the messaging.

[zkat]

@Fishrock123 #8977 (comment) is what specifically calls out that npm@4 is being considered as potentially a minor-level bump for node@7, which is why this PR is up for consideration. It's up so that y'all can look at it and decide.

As far as npm@3 goes, that's a limitation that we're working with -- we care about Node.js LTS having a version of npm that works well. It is not a priority for that version to be the latest and greatest in features and overall patch-attention. It's intended to be something that can be installed by default for automated builds/installation, and if developers want the latest and greatest for npm, we encourage them to npm i -g npm@latest.

I know this is a conversation that has happened in disconnected ways between multiple parties (I've had various versions of it, myself, with you, @thealphanerd, and @jasnell, for example, and I think some of y'all had also had your own conversations with @othiym23). It seems clear from your reaction that this was surprising, and I did not expect it to be.

We're unlikely to be completely rigid on stuff, and yes, the conversation matters. Perhaps #8977 is the place to center the specifics about it. Communication about LTS issues has been a bit difficult as of later.

[Florian-R]

(...) npm@4 is being considered as potentially a minor-level bump for node@7

Just curious, how could this be a minor bump if there's (even small) breaking changes in npm?

[jasnell]

I think handling this as a semver-major for now is the right thing to do until we can get a better sense of the impact of the changes. The only real impact that would have is that v7, v6 and v4 would each continue with npm3 for the foreseeable future. If there's no significant breakage, then we can revisit the decision later on.

Doing a feature freeze on npm3 seems appropriate. I think the main concern we have is that we'll be able to keep getting security and critical issue updates for the npm3 line, at least through the lifetime of the v6 LTS or until if/when a decision is made to pull npm4 into the LTS lines. @zkat the concerning (and surprising) part of your original post above is the statement "might end up being our last npm@3 release". That would imply that we wouldn't be seeing any updates at all, even for security, but I don't believe that is what you intended.

[zkat]

@jasnell oh. no. I didn't mean to imply that. That was purely meant to emphasize that we're treating it as a maintenance thing: our stance about making sure there is a version of npm that works reliably for Node's LTS hasn't changed. There are, of course, possible exceptions to this: if something fundamental involving major changes that were made in a later version is what's "broken", we won't fix it (example: we're not going to patch npm@2 to prevent the Windows longpath issues because npm@3 is our patch for it). I believe this is considered an acceptable caveat all-around, and is the exception, not the rule, and has worked out well in the past in this respect. (see also: nodejs/Release#37)

Please let me know if I can help clarify anything about npm@4, its impact, etc, to help fit it into Node's dev plans :)

[addaleax]

Replaced by #9848