7z binary (7zip) is not present on windows-latest runner beginning 2020-05-19

[lopopolo]

Describe the bug

Invoking 7z with shell: bash on windows-latest os produces this error:

/c/msys64/usr/bin/7z: line 2: /usr/lib/p7zip/7z: No such file or directory
##[error]Process completed with exit code 127.

#898 mentions an msys upgrade. Is this possibly related?

Failing run: https://github.com/artichoke/nightly/runs/691297319

This is the first failing run of a job that has not changed. This failing run has the following environment:

Current runner version: '2.262.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20200517.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20200517.1/images/win/Windows2019-Readme.md

Yesterday's successful run: https://github.com/artichoke/nightly/runs/687237362?check_suite_focus=true

This run used the following environment:

Current runner version: '2.262.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20200430.2
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20200430.2/images/win/Windows2019-Readme.md

Area for Triage:

Question, Bug, or Feature?: Bug

Virtual environments affected

• macOS 10.15
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

Expected behavior
I expect invoking 7z to produce a zip file.

Actual behavior
A description with steps to reproduce the issue. If your have a public example or repo to share,
please provide the link.

  staging="artichoke-nightly-x86_64-pc-windows-msvc"
  mkdir -p "$staging"/
  cp artichoke/{README.md,LICENSE} "$staging/"
  if [ "windows-latest" = "windows-latest" ]; then
    cp "artichoke/target/x86_64-pc-windows-msvc/release/artichoke.exe" "$staging/"
    cp "artichoke/target/x86_64-pc-windows-msvc/release/airb.exe" "$staging/"
    7z a "$staging.zip" "$staging"
    echo "::set-env name=ASSET::$staging.zip"
  else
    cp "artichoke/target/x86_64-pc-windows-msvc/release/artichoke" "$staging/"
    cp "artichoke/target/x86_64-pc-windows-msvc/release/airb" "$staging/"
    tar czf "$staging.tar.gz" "$staging"
    echo "::set-env name=ASSET::$staging.tar.gz"
  fi

outputs:

/c/msys64/usr/bin/7z: line 2: /usr/lib/p7zip/7z: No such file or directory
##[error]Process completed with exit code 127.

[Darleev]

Hello @lopopolo,
I tried to reproduce the issue on my side, using windows-latest and the same build version, but with no luck. Could you please try to simply restart github-actions job and provide us with a result?
We are looking forward to your reply.

[lopopolo]

I get these messages every time I enqueue a build:

There was a failure in sending the provision message: A timeout occurred while sending request to the remote provider.

[lopopolo]

@Darleev I was able to run the job again and it still failed with the same error.

https://github.com/artichoke/nightly/runs/693976956?check_suite_focus=true

This is the fourth time I've rerun the workflow and it has failed in this way.

/c/msys64/usr/bin/7z: line 2: /usr/lib/p7zip/7z: No such file or directory
##[error]Process completed with exit code 127.

This workflow runs daily and the repo hasn't seen any commits in 23 days. The only change is the new builder environment, which in this latest run was:

Current runner version: '2.262.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20200517.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20200517.1/images/win/Windows2019-Readme.md

What are my next steps?

[al-cheb]

Hello, @lopopolo

I can reproduce the issue. The root cause is the https://github.com/ruby/setup-ruby (For Ruby ≥ 2.4, MSYS2 is prepended to the Path, similar to what RubyInstaller2 does.) task which adds additional D:\rubyinstaller-2.6.3-1-x64\bin;C:\msys64\mingw64\bin;C:\msys64\usr\bin; items to the PATH. Previous working directory was - 7z -> /c/ProgramData/Chocolatey/bin/7z now it is changed to the new location 7z -> /c/msys64/usr/bin/7z which caused issue with /c/msys64/usr/bin/7z: line 2: /usr/lib/p7zip/7z: No such file or directory due to unable find dlls.

Temporary workaround to unblock builds, rename msys64 folder:

mv /c/msys64 /c/bakmsys64
7z a "$staging.zip" "$staging"

[al-cheb]

@MSP-Greg, Could you please assist?

[MSP-Greg]

@al-cheb @lopopolo reviewing now

[MSP-Greg]

@lopopolo

EDIT: Skip the pacman command to install p7zip, it is part of the compression package, which is installed.

If you need 7z available, install the MSYS2 version with:

pacman -Sy --noconfirm --noprogressbar --needed p7zip

There are a lot of conflicting tools in the WIndows image, ruby/setup-ruby uses the MSYS2 tools. I believe some of the items you're installing are now available in the MSYS2 install.

I've used 7z in my workflows, I always used a full path to 'C:\Program Files\7-Zip'...

[MSP-Greg]

@lopopolo

I believe 7z.exe exists in the following:

C:/Program Files/7-Zip
C:/ProgramData/Chocolatey/bin

You can probably replace 7z with "/c/Program Files/7-Zip/7z.exe" in your 'Build archive' step. I tried it in a Git Bash shell, and it worked fine

[lopopolo]

Wow thank you so much @al-cheb and @MSP-Greg! 💪

I applied the workaround to use the full path to 7-Zip in artichoke/nightly#1 and was able to successfully release: https://github.com/artichoke/nightly/releases/tag/test-GH-1-7zip-win.

From what I can see, ruby/setup-ruby including a bundled MSYS2 installation was temporary and expected to be rolled back once the windows runner included one by default. I've filed ruby/setup-ruby#57 to investigate cleaning that up.

[MSP-Greg]

@lopopolo

Glad that fixed your problem.

This is messy. As noted above, there are already two versions of 7z installed, the MSYS2 version is the 3rd.

The MSYS2 version is 16.02, the native Windows version I have installed is 19.00. I might consider that an indication that MSYS2 7z is not used a lot. I work with MSYS2 tools both locally and in CI, and I've never installed the p7zip package. It's part of the pacman 'compression' package, and I haven't installed that, but I've installed many of its packages individually...

Hence, I think the best thing to do is remove it from the MSYS2 install.

One thing somewhat separate from this discussion is what bash shell to use. The current windows bash shell command is:

C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}

Note that C:\Program Files\Git\bin is already in PATH.

If the path was removed, users could switch to the MSYS2 bash shell by just changing the PATH, as is done in ruby/setup-ruby. That code is in actions/runner, but I thought I'd mention it, as people have already asked about using the MSYS2 bash shell. See actions/runner#497

@al-cheb - any thoughts?

[MSP-Greg]

@al-cheb

I ran a list of the embedded MSYS2 install that ruby/setup-ruby uses. See the MSYS2 Packages step:
https://github.com/MSP-Greg/actions-image-testing/runs/695152533?check_suite_focus=true

The p7zip package is not installed, so the issue is not ruby/setup-ruby, but that the current MSYS2 install script is installing it as part of the compression package group. As mentioned above, it is an old package, and there are two other 7z installs (one in path) that any shell used in Windows should run.

Hence, see PR #916 which removes the package.

I don't know of any other similar conflicts, I'm hoping this is the only one.

Maybe a note in ruby/setup-ruby's README could make clear that MSYS2/MinGW tools are enabled and at the 'front of the path', so there may be issues if people want to use tools located elsewhere...