npm in 0.10 LTS

[othiym23]

As a followup to a discussion I had with @jasnell and @Fishrock123 on Twitter, I'd like to help this WG and the TSC figure out how the 0.10 release series deals with npm. For the most part, @zkat is the npm CLI team member responsible for our LTS strategy, so this is mostly just to capture the discussion I was having with James and Jeremiah on Twitter, and maybe give Kat some additional context.

The npm CLI team would like to officially discontinue support for npm@1. There hasn't been a new release of npm@1.4 since September 13th, 2014, and at the time that npm@1.4.28 was released, the npm team didn't plan on continuing to support npm@1 development beyond the release of npm@2.0.0 on that same day. Our policy as a support organization has just been to ask people who encounter one of the many issues with npm@1.4.28 to upgrade to npm@latest as a first step, so npm actually stopped supporting npm@1 a long time ago.

npm@1.4.28 is very familiar to many, if not most, Node users, because it's been bundled with Node.js 0.10 since 0.10.32 was released on September 16th, 2014. At the time npm@2.0.0 was released, there were concerns expressed about including an npm including breaking changes in npm in the middle of a stable Node release series. This frustrated me at the time, but in retrospect it looks pretty reasonable.

A lot of changes have been made to npm since then (including several security fixes that affect users of npm@1.4.28), so as we think about moving Node 0.10 into LTS, it's worth taking another look at this and figuring out how to handle npm LTS in a way that works well for the LTS WG and for users depending on stability in 0.10 LTS.

Concerns

The main areas of concern are npm@1's lack of forward compatibility with npm@>=2 features and the primary npm registry, fixes to security flaws in npm@2 that also affect npm@1, and changes included in npm@2 that might have an adverse impact on LTS users of the version of npm bundled with Node 0.10.

Forwards incompatibility of npm@1 with scoped packages

One of the most significant changes in npm@2 is its support for scoped packages. There was a lot of deliberation around the design of scoped packages, and the choice of the format @scope/name for scoped package names was designed to cause as little hassle for users as it possibly could.

Unfortunately, since we didn't plan for npm@1 lasting beyond the rollout of scoped packages, we didn't do much testing of what happened when users tried to install scoped packages with npm@1.4.28. It turns out that npm@1 ignores the leading @ in scoped package names and treats the rest as a GitHub shortcut, which can result in the very confusing situation of a user thinking they've installed an npm private package they shouldn't have access to, when in fact they've installed a package from GitHub (which might not be the package they were expecting).

In addition, the registry changes to support private packages include a switch from using HTTP Basic authentication to bearer tokens (and also changes when authentication information is sent with npm requests – the registry needs to know who's making a request to decide if they're entitled to view any information on private packages). npm@1 is increasingly ill-equipped to deal with current registry architectures, which includes not just npm, Inc's registry (and its npmE on-premise registry product), but also third-party registry projects that have added their own support for scoped and private packages, like sinopia, cnpmjs, and (I believe?) Artifactory.

The support for scoped packages is itself not a breaking change, but backporting all of the necessary code from npm@2 to npm@1 (including all the fixes made related to scopes since npm@2.0.0 went out) is enough work to render supporting scopes in npm@1 practically infeasible.

Security fixes in npm@2 affecting npm@1 users

As I mentioned above, there are a few security fixes in npm@2 that affect npm@1 users. I think / hope this is everything, but it's possible I overlooked something:

• b9474a8 fstream-npm@1.0.5: Stop publishing build cruft (config.gypi) and per-project .npmrc files to keep local configuration out of published packages.
• 300834e tar@2.0.0: Normalize symbolic links that point to targets outside the extraction root. This prevents packages containing symbolic links from overwriting targets outside the expected paths for a package.
• 0dc6875 semver@4.3.2: Package versions can be no more than 256 characters long. This prevents a situation in which parsing the version number can use exponentially more time and memory to parse, leading to a potential denial of service.

Since the fix to semver was made to a version of semver that includes one of the breaking changes included in npm@2, addressing it in npm@1 wouldn't be as simple as updating npm to use a newer version of semver – it would be necessary to fork semver@2 and backport the fix to it.

Breaking changes in npm@2

The changes most pertinent to LTS discussions are in bold.

• df4b0e7 #5518: support passing arguments to run scripts.
• 4378a17 semver@4.0.0: prerelease versions no longer show up in ranges; ^0.x.y behaves the way it did in semver@2 rather than semver@3 (ea547e2 in semver@3: ^0.x.y was functionally the same as =0.x.y).
• c2cccd4 / npm/npm-registry-client@ba6b73e npm-registry-client@5.0.0: npm/npm-registry-client#92: Move /whoami endpoint out of the package namespace (to /-/whoami).
• c6ddb64 npm requires Node >= 0.8.
• 021770b lifecycle: do not add the directory containing node executable.

Possible solutions

Here are two ways we could fix this problem for 0.10 LTS:

Include npm@2 in 0.10 LTS

The npm CLI team is getting close to making npm@3 the version of npm intended for general use. It's had a lengthy development cycle and a pretty extensive beta, but since it includes a nearly complete rewrite of the installer, we've planned from the beginning of the npm@3 beta to support npm@2 and npm@3 in parallel, probably for at least six months. This makes it a much easier thing for us to support going forward (in fact, right now I'm planning on npm@2 being the npm major version included in Node 0.12 and 4.0 LTS, assuming both of those have LTS support). In addition, it includes all the security fixes and also works much better with current npm registry architectures.

That said, it's up to the LTS WG and the TSC to decide if the breaking changes enumerated above are low-risk enough to justify inclusion in LTS. I like this solution because it requires much less work from the CLI team, and we've actually planned to support npm@2 for a while anyway.

Release an npm@1.4.29 that encourages users to upgrade to npm@2

As I hope I've made clear, backporting the security fixes and forward compatibility changes necessary to make npm@1 usable without including any breaking changes is more than the CLI team is able to take on. However, it would not be very difficult to put together a new version of npm@1.4 that includes two changes:

1. Exit with an early error and a helpful message if users attempt to install scoped packages.
2. Every time the CLI is run, print out a deprecation warning strongly encouraging users to upgrade to at least npm@2.

That would address the most significant forwards compatibility issue, along with making it fairly easy for users to figure out what they need to do in circumstances where they're blocked.

These are just ideas I've come up with, and the team is open to other suggestions.

So that's the story with 0.10. If the WG decides that some form of including npm@2 is the way to go, @zkat, as npm@2 maintainer, will be the npm point of contact. If the WG decides to stick with npm@1.4, I'm probably the best person on the CLI team to handle things, as I'm the member of the team who's got the most familiarity with its code. Kat and I are both happy to participate in discussions related to this and show up for LTS WG and TSC meetings discussing them. Just let us know where we're expected and when and we'll be there.

[jasnell]

@nodejs/lts @rvagg ... need to make sure this is on the agenda for the next LTS WG call.
@nodejs/tsc ... fyi

[jasnell]

We only had three on the call today so we're going to defer this discussion to the next LTS call.

[jasnell]

After discussing it on the LTS WG call today, the current plan is to include a clear warning in the upcoming v0.10.41 release that npm will be updated to the latest npm@2 version in v0.10.42. Additionally, a blog post on nodejs.org would communicate the plan to update npm in v0.10.42, detail the breaking changes and rationale for the update, and give users plenty of notice that the update is coming (and give them time to push back should anything truly critical break).

The actual npm update would not land until v0.10.42.

@nodejs/tsc and @nodejs/LTS WG members... we need your input on this plan.

[rvagg]

include a clear warning in the upcoming v0.10.41 release

what is the nature of this proposed warning? will npm be spitting out a console message about it or just a changelog notice?

[jasnell]

I believe that's still to be determined. @zkat @othiym23 ?

[othiym23]

I can go ahead an put together an npm@1.4.29 this week that will barf on attempts to install scoped packages and print a message with whatever text the WG wants indicating that 0.10.42 will be including npm@2. I could, among other things, link to this issue. ;)

[piscisaureus]

Instead of giving a version number, it'd be better to provide a date. If any security issue comes up that forces us to release 0.10.42, we'd have to make the breaking change sooner than expected (or postpone to 0.10.43 and create a lot of confusion).

Conversely, nobody has any idea when 0.10.42 might be released, and if it'll ever happen at all, so it also doesn't create a great sense of urgency.

[ChALkeR]

@piscisaureus I believe that the abovementioned issue fixed by npm/npm@b9474a8 is worth a release in 0.10 branch, the earlier the better.

Note that is has been already two months since that fix.

[othiym23]

A candidate for the transitional npm@1.4 release is available as npm/npm#10152. Feel free to comment on the messaging of the deprecation banner, but be aware that my preference is that it be somewhat long and ugly – the whole goal of this is to encourage everyone to transition away from npm@1 as expeditiously as possible. I'd like to land this in the first 0.10 LTS release, so it would be good to know where and how I should submit the PR to get this reviewed and incorporated into the 0.10 LTS release-candidate branch.

Please note that this release includes no security fixes, and is intended as a transitional release on the path to landing npm@2 in 0.10 LTS. The npm@2 version submitted for 0.10 LTS will contain all of the extant security patches available as of the time that it's downstreamed to the LTS branch.

[othiym23]

npm@1.4.29 is now a thing (those release notes are worth reviewing carefully). @chrisdickinson tells me that https://github.com/nodejs/node/tree/v0.10 is the base against which I should land the PR including it in the tree, but I'll hold off on assembling that PR until somebody (preferably @rvagg or @jasnell) tells me that's the proper place to do it.

[jasnell]

@othiym23 ... Yes, but I'm considering creating v0.10.x-staging and v0.12.x-staging branches to parallel v4.x-staging. We'll put this on the LTS WG meeting agenda for monday.

[jasnell]

@othiym23 ... based on the LTS WG discussion today, I've created the v0.10-staging and v0.12-staging branches. Please open a PR for the npm update against the v0.10-staging branch.

[ChALkeR]

Related: nodejs/node#5570