Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTPS URL for GitLab subgroups is incorrect #46

Closed
mterrel opened this issue Jul 30, 2019 · 1 comment
Closed

HTTPS URL for GitLab subgroups is incorrect #46

mterrel opened this issue Jul 30, 2019 · 1 comment

Comments

@mterrel
Copy link
Contributor

mterrel commented Jul 30, 2019

I am unable to download (via https) & install a package from a public repo on GitLab that is in a subgroup when using a shortcut package specifier. I'm using the pacote library directly to do the download and install, but I can reproduce the issue with the npm CLI as well.

To repro:

  1. First, ensure you have no SSH keys present (e.g. in $HOME/.ssh). Because pacote will try all available transport types, having SSH keys may mask the issue.
  2. Create a new directory and use npm init to create a new package.json file.
  3. In that newly created directory, run npm install gitlab:adpt/starters/hello-node

Expected behavior:

The requested package installs into the node_modules directory without error.

Actual behavior:

I get the following error message:

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@gitlab.com/adpt/starters%2Fhello-node.git
npm ERR! 
npm ERR! Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-07-30T20_28_02_210Z-debug.log

Analysis & debugging:

Although the error that is shown is for accessing the repo via the ssh: protocol, that error is actually not surprising because there are specifically no SSH keys set up.

The real issue is that pacote first tried accessing the GitLab repo via the https: protocol, which should have succeeded, but did not. Unfortunately, the npm CLI and pacote don't make it easy to see this process happening.

By adding a console.log statement in the error case logic of function revs in pacote/lib/util/git.js (line 164 in my current version):

console.log(err.message)

You can see that pacote first tried https, then ssh:

Error while executing:
/usr/bin/git ls-remote -h -t https://gitlab.com/adpt/starters%2Fhello-node.git

fatal: https://gitlab.com/adpt/starters%2Fhello-node.git/info/refs not valid: is this a git repository?

exited with error code: 128
Error while executing:
/usr/bin/git ls-remote -h -t ssh://git@gitlab.com/adpt/starters%2Fhello-node.git

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

exited with error code: 128

If I run the following git command manually on the command line, I get the same error that pacote did:

/usr/bin/git ls-remote -h -t https://gitlab.com/adpt/starters%2Fhello-node.git

fatal: https://gitlab.com/adpt/starters%2Fhello-node.git/info/refs not valid: is this a git repository?

However, if I change the URL-encoded slash character (%2F) back to an actual slash (/), the command completes successfully:

/usr/bin/git ls-remote -h -t https://gitlab.com/adpt/starters/hello-node.git

323e15dc83860abe2ae634c7f866df2e91cc4531	refs/heads/master

Conclusion & Suggested Solution:

GitLab does not appear to support URL-encoded slashes as separators for subgroups when accessing via https.

So I suggest that the https template that hosted-git-info provides for GitLab should not URL encode those slashes as it is currently.

I'll send a PR later for feedback on that approach.

@mterrel
Copy link
Contributor Author

mterrel commented Jul 30, 2019

I forgot to mention that this is related to #43. But in order to fully support GitLab subgroups, it's possible that more work than just fixing the HTTPS URL may be required, so I opened a separate issue.

@isaacs isaacs closed this as completed in bbcf7b2 Aug 4, 2019
isaacs added a commit to npm/cli that referenced this issue Aug 5, 2019
FEATURES

* [bbcf7b2](npm/hosted-git-info@bbcf7b2)
  [#46](npm/hosted-git-info#46)
  [#43](npm/hosted-git-info#43)
  [#47](npm/hosted-git-info#47)
  [#44](npm/hosted-git-info#44) Add support for
  GitLab groups and subgroups ([@mterrel](https://github.com/mterrel),
  [@isaacs](https://github.com/isaacs),
  [@ybiquitous](https://github.com/ybiquitous))

BUGFIXES

* ([3b1d629](npm/hosted-git-info@3b1d629))
  [#48](npm/hosted-git-info#48) fix http protocol
  using sshurl by default ([@fengmk2](https://github.com/fengmk2))
* [5d4a8d7](npm/hosted-git-info@5d4a8d7) ignore
  noCommittish on tarball url generation
  ([@isaacs](https://github.com/isaacs))
* [1692435](npm/hosted-git-info@1692435) use gist
  tarball url that works for anonymous gists
  ([@isaacs](https://github.com/isaacs))
* [d5cf830](npm/hosted-git-info@d5cf830)
* Do not allow invalid gist urls ([@isaacs](https://github.com/isaacs))
* [e518222](npm/hosted-git-info@e518222)
  Use LRU cache to prevent unbounded memory consumption
  ([@iarna](https://github.com/iarna))
@isaacs isaacs mentioned this issue Aug 5, 2019
isaacs added a commit to npm/node that referenced this issue Aug 6, 2019
BUGFIXES

* [`27cccfbda`](npm/cli@27cccfb)
  [nodejs#223](npm/cli#223) vulns → vulnerabilities in
  npm audit output ([@sapegin](https://github.com/sapegin))
* [`d5e865eb7`](npm/cli@d5e865e)
  [nodejs#222](npm/cli#222)
  [nodejs#226](npm/cli#226) install, doctor: don't crash
  if registry unset ([@dmitrydvorkin](https://github.com/dmitrydvorkin),
  [@isaacs](https://github.com/isaacs))
* [`5b3890226`](npm/cli@5b38902)
  [nodejs#227](npm/cli#227)
  [npm.community#9167](https://npm.community/t/npm-err-cb-never-called-permission-denied/9167/5)
  Handle unhandledRejections, tell user what to do when encountering an
  `EACCES` error in the cache.  ([@isaacs](https://github.com/isaacs))

DEPENDENCIES

* [`77516df6e`](npm/cli@77516df)
  `licensee@7.0.3` ([@isaacs](https://github.com/isaacs))
* [`ceb993590`](npm/cli@ceb9935)
  `query-string@6.8.2` ([@isaacs](https://github.com/isaacs))
* [`4050b9189`](npm/cli@4050b91)
  `hosted-git-info@2.8.2`
    * [nodejs#46](npm/hosted-git-info#46)
      [nodejs#43](npm/hosted-git-info#43)
      [nodejs#47](npm/hosted-git-info#47)
      [nodejs#44](npm/hosted-git-info#44) Add support for
      GitLab subgroups ([@mterrel](https://github.com/mterrel),
      [@isaacs](https://github.com/isaacs),
      [@ybiquitous](https://github.com/ybiquitous))
    * [`3b1d629`](npm/hosted-git-info@3b1d629)
      [nodejs#48](npm/hosted-git-info#48) fix http
      protocol using sshurl by default
      ([@fengmk2](https://github.com/fengmk2))
    * [`5d4a8d7`](npm/hosted-git-info@5d4a8d7)
      ignore noCommittish on tarball url generation
      ([@isaacs](https://github.com/isaacs))
    * [`1692435`](npm/hosted-git-info@1692435)
      use gist tarball url that works for anonymous gists
      ([@isaacs](https://github.com/isaacs))
    * [`d5cf830`](npm/hosted-git-info@d5cf830)
      Do not allow invalid gist urls ([@isaacs](https://github.com/isaacs))
    * [`e518222`](npm/hosted-git-info@e518222)
      Use LRU cache to prevent unbounded memory consumption
      ([@iarna](https://github.com/iarna))
Trott pushed a commit to nodejs/node that referenced this issue Aug 20, 2019
BUGFIXES

* [`27cccfbda`](npm/cli@27cccfb)
  [#223](npm/cli#223) vulns → vulnerabilities in
  npm audit output ([@sapegin](https://github.com/sapegin))
* [`d5e865eb7`](npm/cli@d5e865e)
  [#222](npm/cli#222)
  [#226](npm/cli#226) install, doctor: don't crash
  if registry unset ([@dmitrydvorkin](https://github.com/dmitrydvorkin),
  [@isaacs](https://github.com/isaacs))
* [`5b3890226`](npm/cli@5b38902)
  [#227](npm/cli#227)
  [npm.community#9167](https://npm.community/t/npm-err-cb-never-called-permission-denied/9167/5)
  Handle unhandledRejections, tell user what to do when encountering an
  `EACCES` error in the cache.  ([@isaacs](https://github.com/isaacs))

DEPENDENCIES

* [`77516df6e`](npm/cli@77516df)
  `licensee@7.0.3` ([@isaacs](https://github.com/isaacs))
* [`ceb993590`](npm/cli@ceb9935)
  `query-string@6.8.2` ([@isaacs](https://github.com/isaacs))
* [`4050b9189`](npm/cli@4050b91)
  `hosted-git-info@2.8.2`
    * [#46](npm/hosted-git-info#46)
      [#43](npm/hosted-git-info#43)
      [#47](npm/hosted-git-info#47)
      [#44](npm/hosted-git-info#44) Add support for
      GitLab subgroups ([@mterrel](https://github.com/mterrel),
      [@isaacs](https://github.com/isaacs),
      [@ybiquitous](https://github.com/ybiquitous))
    * [`3b1d629`](npm/hosted-git-info@3b1d629)
      [#48](npm/hosted-git-info#48) fix http
      protocol using sshurl by default
      ([@fengmk2](https://github.com/fengmk2))
    * [`5d4a8d7`](npm/hosted-git-info@5d4a8d7)
      ignore noCommittish on tarball url generation
      ([@isaacs](https://github.com/isaacs))
    * [`1692435`](npm/hosted-git-info@1692435)
      use gist tarball url that works for anonymous gists
      ([@isaacs](https://github.com/isaacs))
    * [`d5cf830`](npm/hosted-git-info@d5cf830)
      Do not allow invalid gist urls ([@isaacs](https://github.com/isaacs))
    * [`e518222`](npm/hosted-git-info@e518222)
      Use LRU cache to prevent unbounded memory consumption
      ([@iarna](https://github.com/iarna))

PR-URL: #29023
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
BridgeAR pushed a commit to nodejs/node that referenced this issue Sep 3, 2019
BUGFIXES

* [`27cccfbda`](npm/cli@27cccfb)
  [#223](npm/cli#223) vulns → vulnerabilities in
  npm audit output ([@sapegin](https://github.com/sapegin))
* [`d5e865eb7`](npm/cli@d5e865e)
  [#222](npm/cli#222)
  [#226](npm/cli#226) install, doctor: don't crash
  if registry unset ([@dmitrydvorkin](https://github.com/dmitrydvorkin),
  [@isaacs](https://github.com/isaacs))
* [`5b3890226`](npm/cli@5b38902)
  [#227](npm/cli#227)
  [npm.community#9167](https://npm.community/t/npm-err-cb-never-called-permission-denied/9167/5)
  Handle unhandledRejections, tell user what to do when encountering an
  `EACCES` error in the cache.  ([@isaacs](https://github.com/isaacs))

DEPENDENCIES

* [`77516df6e`](npm/cli@77516df)
  `licensee@7.0.3` ([@isaacs](https://github.com/isaacs))
* [`ceb993590`](npm/cli@ceb9935)
  `query-string@6.8.2` ([@isaacs](https://github.com/isaacs))
* [`4050b9189`](npm/cli@4050b91)
  `hosted-git-info@2.8.2`
    * [#46](npm/hosted-git-info#46)
      [#43](npm/hosted-git-info#43)
      [#47](npm/hosted-git-info#47)
      [#44](npm/hosted-git-info#44) Add support for
      GitLab subgroups ([@mterrel](https://github.com/mterrel),
      [@isaacs](https://github.com/isaacs),
      [@ybiquitous](https://github.com/ybiquitous))
    * [`3b1d629`](npm/hosted-git-info@3b1d629)
      [#48](npm/hosted-git-info#48) fix http
      protocol using sshurl by default
      ([@fengmk2](https://github.com/fengmk2))
    * [`5d4a8d7`](npm/hosted-git-info@5d4a8d7)
      ignore noCommittish on tarball url generation
      ([@isaacs](https://github.com/isaacs))
    * [`1692435`](npm/hosted-git-info@1692435)
      use gist tarball url that works for anonymous gists
      ([@isaacs](https://github.com/isaacs))
    * [`d5cf830`](npm/hosted-git-info@d5cf830)
      Do not allow invalid gist urls ([@isaacs](https://github.com/isaacs))
    * [`e518222`](npm/hosted-git-info@e518222)
      Use LRU cache to prevent unbounded memory consumption
      ([@iarna](https://github.com/iarna))

PR-URL: #29023
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
BethGriggs pushed a commit to nodejs/node that referenced this issue Oct 19, 2019
BUGFIXES

* [`27cccfbda`](npm/cli@27cccfb)
  [#223](npm/cli#223) vulns → vulnerabilities in
  npm audit output ([@sapegin](https://github.com/sapegin))
* [`d5e865eb7`](npm/cli@d5e865e)
  [#222](npm/cli#222)
  [#226](npm/cli#226) install, doctor: don't crash
  if registry unset ([@dmitrydvorkin](https://github.com/dmitrydvorkin),
  [@isaacs](https://github.com/isaacs))
* [`5b3890226`](npm/cli@5b38902)
  [#227](npm/cli#227)
  [npm.community#9167](https://npm.community/t/npm-err-cb-never-called-permission-denied/9167/5)
  Handle unhandledRejections, tell user what to do when encountering an
  `EACCES` error in the cache.  ([@isaacs](https://github.com/isaacs))

DEPENDENCIES

* [`77516df6e`](npm/cli@77516df)
  `licensee@7.0.3` ([@isaacs](https://github.com/isaacs))
* [`ceb993590`](npm/cli@ceb9935)
  `query-string@6.8.2` ([@isaacs](https://github.com/isaacs))
* [`4050b9189`](npm/cli@4050b91)
  `hosted-git-info@2.8.2`
    * [#46](npm/hosted-git-info#46)
      [#43](npm/hosted-git-info#43)
      [#47](npm/hosted-git-info#47)
      [#44](npm/hosted-git-info#44) Add support for
      GitLab subgroups ([@mterrel](https://github.com/mterrel),
      [@isaacs](https://github.com/isaacs),
      [@ybiquitous](https://github.com/ybiquitous))
    * [`3b1d629`](npm/hosted-git-info@3b1d629)
      [#48](npm/hosted-git-info#48) fix http
      protocol using sshurl by default
      ([@fengmk2](https://github.com/fengmk2))
    * [`5d4a8d7`](npm/hosted-git-info@5d4a8d7)
      ignore noCommittish on tarball url generation
      ([@isaacs](https://github.com/isaacs))
    * [`1692435`](npm/hosted-git-info@1692435)
      use gist tarball url that works for anonymous gists
      ([@isaacs](https://github.com/isaacs))
    * [`d5cf830`](npm/hosted-git-info@d5cf830)
      Do not allow invalid gist urls ([@isaacs](https://github.com/isaacs))
    * [`e518222`](npm/hosted-git-info@e518222)
      Use LRU cache to prevent unbounded memory consumption
      ([@iarna](https://github.com/iarna))

PR-URL: #29023
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant