Dependabot w/ pnpm and private feed returns ERR_PNPM_FETCH_401

[AndrewCraswell]

Is there an existing issue for this?

• I have searched the existing issues

Package ecosystem

npm (pnpm)

Package manager version

8.6.11

Language version

Node v18.15.0

dependabot.yml content

version: 2
registries:
  upstreams:
    type: npm-registry
    url: https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/
    token: "PAT:${{AZURE_DEVOPS_PAT}}" # this is the non-base64 encoded PAT with Packaging read/write scope
updates:
  - package-ecosystem: npm
    directory: /
    registries:
      - upstreams
    schedule:
      interval: daily
    target-branch: main
    open-pull-requests-limit: 3
    milestone: 163501 # The work item Id to link to the PR
    versioning-strategy: increase

What you expected to see, versus what you actually saw

We have a repository in Azure Devops which uses a single private feed. When running our dependabot pipeline it successfully scans the first package, detects the update, and opens a PR. Then starting with the second package we begin seeing failures. In the logs below @1js/react-ecs-client is scanned and finds no update (because it already opened the PR in a previous run). But @azure/storage-file-datalake detects that an update is required, but fails before opening the PR.

Checking if @1js/react-ecs-client 2.0.4 needs updating
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client
🌍 <-- 200 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client
🌍 <-- 200 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client/2.0.4
🌍 <-- 404 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client/2.0.4
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js/react-ecs-client/2.0.4
🌍 <-- 404 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js/react-ecs-client/2.0.4
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client/latest
🌍 <-- 404 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@1js%2Freact-ecs-client/latest
No update needed for @1js/react-ecs-client 2.0.4

Checking if @azure/storage-file-datalake 12.13.0 needs updating
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake
🌍 <-- 200 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake
🌍 <-- 200 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake/12.14.0
🌍 <-- 404 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake/12.14.0
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure/storage-file-datalake/12.14.0
🌍 <-- 404 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure/storage-file-datalake/12.14.0
🌍 --> GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake/latest
🌍 <-- 404 https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fstorage-file-datalake/latest
Requirements to unlock own
Requirements update strategy bump_versions
Updating @azure/storage-file-datalake from 12.13.0 to 12.14.0
/home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.224.0/lib/dependabot/shared_helpers.rb:345:in `run_shell_command': Progress: resolved 0, reused 1, downloaded 0, added 0 (Dependabot::SharedHelpers::HelperSubprocessFailed)
 ERR_PNPM_FETCH_401  GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/@azure%2Fabort-controller: Unauthorized - 401

No authorization header was set for the request.

No authorization settings were found in the configs.
Try to log in to the registry by running "pnpm login"
or add the auth tokens manually to the ~/.npmrc file.
 ERR_PNPM_FETCH_401  GET https://pkgs.dev.azure.com/1edu/_packaging/Upstreams/npm/registry/events: Unauthorized - 401
...

I'm not understanding how the first request could succeed, yet subsequent request would suddenly receive a 401. The PAT token I've used is for my personal account, and I'm able to perform a pnpm install locally just fine.

Images of the diff or a link to the PR, issue, or logs

Full dependabot pipeline logs

Smallest manifest that reproduces the issue

No response

[AndrewCraswell]

I did some more testing. If I delete my pnpm-lock.yaml file Dependabot starts working flawlessly. I also tried switching to yarn with a yarn.lock file and that worked as well. So, I suspect there's some bug that is being encountered with my lockfile.

pnpm-lock.yaml

[mburumaxwell]

@deivid-rodriguez could you possibly offer some insight here?

[deivid-rodriguez]

Thanks for the ping @mburumaxwell!

I'm currently working on this. Your patch at #8091 does address part of the problem, but there are some other issues where Dependabot is unable to write a proper npmrc file in some situations (like here). Do you mind if I use your PR as a base for my changes?

[mburumaxwell]

@deivid-rodriguez sure you can use it

[deivid-rodriguez]

@AndrewCraswell Would you be able to check whether #8094 fixes your issue? I think it's going to be a bit of a mess to verify but maybe @mburumaxwell can help?

I think you'll need to clone the PR, rebuild the npm updater (script/build npm_and_yarn), then rebuild the updater wrapping it at https://github.com/tinglesoftware/dependabot-azure-devops, and then make sure your pipeline pulls that custom image?

[AndrewCraswell]

@mburumaxwell Any suggestions for how we could run the patch against our Azure DevOps pipeline to verify the changeset?

[deivid-rodriguez]

I merged my PR so it should be easier to try it now. Please let me know if it indeed worked!

[mburumaxwell]

I did my test and found that it takes quite a lot of time to move through one dependency update as per #8094 (comment).

@AndrewCraswell , you can test this using a temporary image from tinglesoftware/dependabot-azure-devops#829. The image tag is 1.21.1-pullrequest0829-0034.

If you are using the extension:

steps:
- task: dependabot@1
  inputs:
    dockerImageTag: '1.21.1-pullrequest0829-0034'

[AndrewCraswell]

I've run with the image tag and it worked! It took 16 minutes to open 3 PRs which is fine for us since we only open 3 at a time. :D

[mburumaxwell]

@AndrewCraswell using 1.21.1-pullrequest0829-0052 may be faster.
It is based on

• Test for PNPM fix tinglesoftware/dependabot-azure-devops#829
• https://github.com/mburumaxwell/dependabot-core/tree/pnpm-speed-fix

This is only temporary as we await an official fix.