[BUG] Fails to install npm packages from private github repos in docker since v6.11.0

[simonkotwicz]

What / Why / When

npm fails to install npm packages from private github repositories in docker since v6.11.0

Where

packages from private github repositories in docker

How

Current Behavior

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/user/private-repo
npm ERR!
npm ERR! remote: Password authentication is not available for Git operations.
npm ERR! remote: You must use a personal access token or SSH key.
npm ERR! remote: See https://github.com/settings/tokens or https://github.com/settings/ssh
npm ERR! fatal: unable to access 'https://github.com/user/private-repo': The requested URL returned error: 403
npm ERR!
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-02-19T02_07_01_401Z-debug.log

Steps to Reproduce

1. Install an npm package from a private git repository you have access to (eg. npm i git+https://github.com/user/private-repo)

2. Write up a simple docker file like...

FROM node:12.16.0-alpine

WORKDIR /home/node

ARG GITHUB_READ_TOKEN
ARG NPM_VERSION

COPY package.json package-lock.json ./

RUN npm install -g npm@$NPM_VERSION && \
    apk add --no-cache --virtual temp-build-deps git && \
    echo -e "machine github.com\n  login $GITHUB_READ_TOKEN" > ~/.netrc && \
    npm install && \
    apk del temp-build-deps && \
    rm ~/.netrc

COPY . .

ENTRYPOINT ["npm", "test"]

run docker build . --build-arg NPM_VERSION=6.13.6 --build-arg GITHUB_READ_TOKEN=***

Expected Behavior

docker build succeeds like it does for docker build . --build-arg NPM_VERSION=6.10.0 --build-arg GITHUB_READ_TOKEN=***

[isaacs]

Does it work when you run the git commands directly, not through npm?

Can you try executing git ls-remote -h -t https://github.com/user/private-repo with that ~/.netrc file in place?

What is the user account that this is running as? Is it possible that the echo command is adding things to a home directory that does not match the owner of the cwd where git is running? (You could figure this out by adding id and ls -l . to the RUN command.)

If the first command works fine, but the user running all this is root, and the owner of the directory is something other than root, then you'll have to put the .netrc in that user's home directory instead of in root's, because the git command won't run as root, I'd expect it'd be running as the owner of the directory (the better to avoid root-owned files and folders in a local installation).

[simonkotwicz]

git ls-remote -h -t https://github.com/user/private-repo with that ~/.netrc in place succeeds.

Like you said, the user running npm install is root but the owner of the directory is node. Running npm install as the node user does work (and the right thing to do usually) but putting the .netrc file in node's home directory (home/node) and running npm install as root does not fix the issue.

Shouldn't running npm install as root also cause the underlying git commands to run as root? I would have expected since a ~/.npmrc file works for the npm install so should a ~/.netrc or ~/.git-credentials file in the same location.

[isaacs]

When npm is running as root, any git commands are run with an effective UID and GID matching the owner of the working directory in which that git command is run.

This prevents doing a git clone as root and creating root-owned files in a user-owned location which cannot be easily removed or modified by subsequent npm commands. Of course, while this is the right thing to do in most cases that developers touch directly on their local machines, it can present some additional complexity in Docker environments, where it's common to run a command as root, with a root-owned cache folder, but a non-root-owned install target location.

When you placed the config file at ~node/.netrc, did you also leave a copy at ~root/.netrc? It may be that you're running one command in a root-owned directory (my guess would be, /root/.npm/_cacache/tmp), and possibly another git command in a node-owned directory (eg, ~node/project/node_modules/foo).

[simonkotwicz]

Even in both locations it still fails.

Interestingly, without a package-lock.json file, npm install works fine and happily clones the private git repository using the ~root/.netrc file

[Tacklebox]

In a similar situation, I haven't looked into how npm does it's inference/user switching, but it seems it fails to set the home directory of the user it switches to.

If you run npm ci as root in a directory owned by user node it will fail on ls-remote as above even if both /root/.netrc and /home/node/.netrc exist.

Running HOME=/home/node npm ci as root works though.

[mjpelmear]

I am experiencing a similar issue related to the .ssh/known_hosts file. I believe it has the same root cause.

It works fine for us on windows/macos hosts, but fails on linux hosts, starting with npm 6.11.0 (in docker containers) during npm install with a prompt to accept bitbucket's ssh fingerprint (which we store in /root/.ssh/known_hosts).

Copying known_hosts (and our ssh deploy key) to the node user's .ssh folder (with appropriate permissions) and re-running npm install works fine, as long as I leave the files in root's .ssh folder as well (so, with the files in both places).

Also, similarly to the above comment, npm install works fine without putting these files into the node user's folder if I delete package-lock.json. (Signs of a different bug?)

Tested on the node:10.16.3-stretch and node:14.4.0-stretch docker images (with npm 6.11.0 and 6.14.5).

It would be nice to have a way to override the user inference behavior.

[darcyclarke]

npm v6 is no longer in active development; We will continue to push security releases to v6 at our team's discretion as-per our Support Policy.

If your bug is preproducible on v7, please re-file this issue using our new issue template.

If your issue was a feature request, please consider opening a new RRFC or RFC. If your issue was a question or other idea that was not CLI-specific, consider opening a discussion on our feedback repo

Closing: This is an automated message.

[kgagnon]

Sorry to necro this but a specific google search leads to here. (I know for a fact haha)

I just wanted to share my fix in case it can help anyone. I had user:root defined in my docker-compose.yml on my frontend service that was doing the npm install on my project. npm (I tried with node:16 and node:17 docker images) needs to be the node user to fetch private SSH keys.

So make sure that you run your container as node and that you drop your ssh key in /home/node/.shh NOT in /root/.ssh like you could easily think.

To the devs: can this be that npm install in the node docker image assumes the ssh configs will be in that exact path and doesn't look it up in the current userspace?

[viniciusfersil123]

@kgagnon You saved my life bro ! It worked like a charm ! ❤️