Add MSYS2 to Windows

[kaylangan]

Tool information

• Tool name: MSYS2
• Add or update? Add
• Desired version: Latest
• Approximate size:
• If this is an add request:
  • Brief description of tool: Providing MSYS2 instead of MinGW would enable a way easier package management for MinGW (via Pacman) and also faster builds.
  • URL for tool's homepage: https://www.msys2.org/

Virtual environments affected

• macOS 10.14
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

Can this tool be installed during the build?
Yes, via choco install msys2.

Are you willing to submit a PR?

[eine]

Ref actions/starter-workflows#95 (/cc @krcroft @andymckay)

Ref numworks/setup-msys2 1138-4EB/setup-msys2 (/cc @Ecco)

[eine]

I suggest to download and extract the available MSYS2 tarball, instead of using chocolatey. This is because chocolatey requires ~5min but the tarball is extracted in 30s. Please, see
actions/toolkit#165 (comment) for other references related to current workarounds for this issue.

[rgl]

@eine that time difference is probably because one is updating the base tarball binaries and the other is not.

[eine]

@rgl, although it might be the case, I don't think it is the only reason. Precisely, an option is available in setup-msys2 which allows to not only extract the tarball, but also execute pacman -Syu. Extraction time is ~40s (https://github.com/numworks/setup-msys2/runs/348237664) and extraction plus update requires ~70s (https://github.com/numworks/setup-msys2/runs/348238118). Conversely, choco install msys2 takes 270s (https://github.com/eine/ghdl-packaging/commit/069ae680c38a1832950a377618dc4a1798d6ccf3/checks?check_suite_id=231637124) at least.

Nevertheless, I think that installation time is not revelant in this specific request. Chocolatey package mingw is outdated (frozen to say the least) and msys2 should be installed instead. It is not reasonable to expect users to install MSYS2 through an action (which is already available), because it has many limitations. Provided that a proper MSYS2 installation is included in windows-latest, I don't really care how GitHub decides to do it.

[MSP-Greg]

The base MSYS2 install is mostly just command line tools, which are very similar to the tools provided with Git for Windows.

I think most users are looking for the main install along with the groups 'base' and 'base-devel', and probably mingw-w64-x86_64-gcc and mingw-w64-x86_64-cmake. Since the tarballs are not updated frequently, pacman -Syu should be done at some point.

Beyond that, packages like OpenSSL, libyaml, etc, but they can installed in the workflows...

[dreamer]

I think most users are looking for the main install along with the groups 'base' and 'base-devel', and probably mingw-w64-x86_64-gcc and mingw-w64-x86_64-cmake.

There's no reason to presume that. We are using MSYS2 to test Windows compilation of our project using GCC and Clang (in parallel to MSVC) and depend on following packages: pkg-config libtool libpng zlib ncurses SDL2 SDL2_net opusfile autogen autoconf automake-wrapper.

[MSP-Greg]

Many of the packages you mentioned are part of base-devel. pacman -Qg base-devel...

CLang/LLVM could be added.

We can't and shouldn't expect Actions to have every package. What I'm proposing is what I mentioned and any addition compiler toolsets (as you've mentioned CLang).

[MSP-Greg]

@kaylangan

1. Is C:/msys64 ok for location? This is the standard location.

2. What language would be best for an install script?

[1480c1]

1. Is C:/msys64 ok for location? This is the standard location.

That is the default on a lot of systems

2. What language would be best for an install script?

Since this for windows, I would assume powershell

[MSP-Greg]

I created an Install-MSYS2.ps1 file similar to the files in images/win/scripts/Installers. I did not create a Validate-MSYS2.ps1 file, but I suppose one could check bash, make, gcc, pacman, etc for output with a version argument.

See the output in the 'Install MSYS2' step in https://github.com/MSP-Greg/msys2-action/runs/396221571. It took 6 minutes. The bottom of the output contains lists of MSYS2 & MinGW64 packages.

The script only installs both 32-bit and 64-bit tools.

Note that if this is added to Windows images, the toolcache Ruby installs will no longer require embedded MSYS2 installs, which reduces their size considerably, and also allows install from a signed *.7z file.

As to whether to install any other packages, that's another issue.

[dreamer]

As to whether to install any other packages, that's another issue.

I would suggest adding clang; it can coexist with mingw.

[MSP-Greg]

@dreamer

How do you install from pacman? As below? It installs llvm as a dependent package...

pacman -S mingw-w64-x86_64-clang

Running pacman -Ss clang shows a few packages. I just want to make sure to install what's needed...

[dreamer]

In dosbox-staging we use mingw-w64-x86_64-clang and mingw-w64-i686-clang.

[MSP-Greg]

I updated the file mentioned above and the link to a workflow run. It now installs both 32 & 64 bit tools.

[MSP-Greg]

@eine @lazka

Do you have any suggestions for what packages should be installed in the MSYS2 install?

The script I provided installs base-devel and compression groups on MSYS2, and clang, cmake, gcc, and ragel on mingw64 / mingw32.

There's a lot of packages, Appveyor currently has these installed. Any suggestions for the most popular packages that could be included to the above?

[Ecco]

@MSP-Greg Why install any package by default anyway? Just do the base install and let developers install whatever package they want. Otherwise you'll uselessly slow things down for a lot of people…

[MSP-Greg]

@Ecco

Sorry you disagree.

It depends on your reference.

My reference is CI that also runs on Ubuntu & macOS, where the equivalent of MSYS2, compiler tools, and many packages are already installed.

Between GitHub issues and GitHub Community, there seem ot be many people asking for MSYS2, and many people trying to do their own install and unhappy with time it takes.

Otherwise you'll uselessly slow things down

I suspect having the packages installed on the 'virtual-environment' image is much faster than downloading them and installing. Again, I'm not suggesting adding a lot of packages, just what would be commonly used.

What have you used MSYS2 for?

[1480c1]

@MSP-Greg Why install any package by default anyway? Just do the base install and let developers install whatever package they want. Otherwise you'll uselessly slow things down for a lot of people…

@Ecco By not providing any default installs, the amount of time spend per person needing to install the whole entire gcc toolchain and clang+llvm etc would be huge and cost extra time for each VM needing to be on. Plus the trade off of initial time spend installing and size increase is worth the amount of time saved later on when you don't need to host the vm for a few more minutes and time spend downloading the packages again.

Finally, if everyone who wanted to test mingw-w64 on windows needed to download every single package every time, I'm afraid that msys2.org will probably not be able to handle it.

That would essentially be handing a person a clean arch install and expecting them to install everything needed to run a ci every single time they want to test their software

[eine]

@MSP-Greg, thanks for providing an example installation script! I hope that employees pick it ASAP.

Regarding the discussion about which specific packages should or should not be installed, IMHO it is an interesting but absolutely irrelevant discussion in the current context. I don't mean to cut the discussion, but to bring the attention so that we can focus on the features that are necessary and useful for all of us.

Precisely, I think that installing MSYS2 is not enough and it can break the environment, as it would conflict with mingw. We need GitHub employees to confirm that MSYS2 is going to replace MINGW, and that the default bash will be the one provided by MSYS2. This has several effects:

• Multi-line scripts in YAML should be supported. See Set custom default shell runner#240 and MSY2/MINGW example workflow starter-workflows#95.
• Features from setup-msys2, such as update, MSYSTEM or MSYS2_PATH_TYPE should be documented.

---

I think most users are looking for the main install along with the groups 'base' and 'base-devel'

I do think that all users that want MSYS2 are looking for pacman. That's the only certainty we have. I agree with you that having base, base-devel, etc. can be useful for a majority of users, and I would not oppose to having them installed by default. Nevertheless, as said, I think this details are not relevant. Specially for open-source projects, where those extra minutes are not charged. That's something GitHub needs to worry about, to optimize their own infrastructure. Given the current performance penalty of Windows environments, that seems not to be something prioritary for them.

pacman -Syu should be done at some point

I think this is a different issue from installating default packages. This is about providing a "stable" environment or an "edge" set of packages. A first execution of the shell is recommended anyway. Hence, it's a matter of deciding which that first command should be. We had this discussion in setup-msys2 before: numworks/setup-msys2#7.

Overall, as long as a base MSYS2 is provided, all other features (updating, installing (cached) packages, etc.) can be managed by an Action such as setup-msys2. That's why I think we should focus on the features that are not easily achievable with an action right now.

Do you have any suggestions for what packages should be installed in the MSYS2 install?

Personally, I want mingw-w64-x86_64-llvm, mingw-w64-x86_64-clang, mingw-w64-x86_64-z3, mingw-w64-*-toolchain, since I'm building PKGBUILD files in an MSYS shell, before testing them in MINGW32 and MINGW64 shells. Anyway, once again, this should not be about what each of us in this thread wants. None of us has any data/statistics to justify the inclusion of any specific package.

Otherwise you'll uselessly slow things down for a lot of people…

This installation script is for the base image that GitHub has cached in their servers. Theoretically, when a workflow/job is started, all it needs to do is copy it to memory. That's the whole point of supporting a single Windows environment. Therefore, the penalty of having more packages installed by default should not be relevant at all. OTOH, not installing them is slow for sure, because they need to be retrieved each time. As a result, I think that GitHub is the one that should be worried about their HD/RAM footprint and network quote. That's something defined by their product strategy: do they want to take care of larger storage costs or do they want to charge paid users by forcing an slower environment?

Finally, if everyone who wanted to test mingw-w64 on windows needed to download every single package every time, I'm afraid that msys2.org will probably not be able to handle it.

I assume that GitHub has some internal caching that would prevent msys2.org from being hit too hard.

That would essentially be handing a person a clean arch install and expecting them to install everything needed to run a ci every single time they want to test their software

Well, this is the state of the art, since that's the essence of containers. Sure, you can have your own image built on top of a clean arch. However, pulling a large docker image in a GNU/Linux job is not very different from running a pre-install script in MSYS2. Most, if not all, practical use cases will need some kind of pre-install configuration, no matter how complete the default environment tries to be.

[AlenaSviridenko]

Hi @eine, we are going to update docs with a proper note on a next image deployment, which is approximately next week. But we don't have plans to add MSYS2 to PATH in future.

[vadi2]

Qt can be installed with https://github.com/jurplel/install-qt-action.

[1480c1]

As msys2 was added under C:\msys64 now with windows-2019 does this issue still need to stay open?

[maxim-lobanov]

Yep, we have one more opened PR to improve installation stability: #928 .
We are planning to close issue after that

[1480c1]

I see, it's probably best to link those prs and stuff to here so it's more visible without needing to go through the PR page,

[MSP-Greg]

Install script in master Install-Msys2.ps1

[andreasdotorg]

I'm missing pacman in the MSYS2 install on windows-2019. Is that a deliberate omission or an oversight?

[1480c1]

Where are you looking for pacman?

[MSP-Greg]

@andreasdotorg

Git's bash shell has been the default on Win images, so MSYS2 is not included in Path. Pacman is located at C:\msys64\usr\bin. You may need to add other locations to Path to use the MSYS2 MinGW tools.

[1480c1]

In appveyor, I used

$path = ([System.Environment]::GetEnvironmentVariable('PATH', 'Machine').Split(';') | Where-Object { $_ -notmatch 'Git' }) -join ';'
[System.Environment]::SetEnvironmentVariable('PATH', $path, 'Machine')

to remove git from the path so I could use msys2's C:\msys64\usr\bin;C:\msys64\mingw64\bin in the path without any conflicts, you might be able to modify it to match github actions better

[nulano]

I'm using the following to run commands in MSYS/MinGW32 or MSYS/MinGW64 bash shell:

name: Test Windows
on: [push, pull_request]
jobs:
  msys:
    runs-on: windows-2019
    strategy:
      matrix:
        mingw: ["MINGW32", "MINGW64"]
        include:
          - mingw: "MINGW32"
            package: "mingw-w64-i686"
          - mingw: "MINGW64"
            package: "mingw-w64-x86_64"
    defaults:
      run:
        shell: bash.exe --login -eo pipefail "{0}"
    env:
      MSYSTEM: ${{ matrix.mingw }}
      CHERE_INVOKING: 1
    name: MSYS2 ${{ matrix.mingw }}
    steps:
      - name: Set up shell
        run: echo ::add-path::C:\msys64\usr\bin\
        shell: pwsh
      - name: Print system version
        run: |
          uname
      - name: Install package
        run: |
          # install python
          pacman -S --noconfirm ${{ matrix.package }}-python3-setuptools
          # build the project
          python3 setup.py install

The first step adds msys at the start of the path, and the MSYSTEM and CHERE_INVOKING envvars tell MSYS bash to run on the given platform (MinGW32 or MinGW64) and in the project directory. I'm setting a default custom shell to run multiple steps in the MSYS shell, but you can explicitly set it for each step.

Edit: added @1480c1's suggestion

[1480c1]

You might want to add -eo pipefail to your shell: line to emulate what github actions uses for their bash

[eine]

@nulano, I updated the README in eine/setup-msys2 to show how to set a default shell, as you did. The main differences between your approach and setup-msys2 are the following:

• setup-msys2 is an Action. Hence, setting MSYSTEM, modifying the PATH, updating or preinstalling some packages, etc. is done in a single step.
• The equivalent to CHERE_INVOKING: 1 is the default in setup-msys2: bash.exe --norc -ilceo pipefail "cd $OLDPWD && %args%".
• With setup-msys2 the PATH is NOT inherited by default, it's an option. See Inconsistencies using git describe --dirty checkout#250 for possible conflicts when trying to use git from the host and the one in MSYS2.

You can find multiple workflow setups in https://github.com/eine/setup-msys2/blob/master/.github/workflows/action.yml.

Ref #916 #908 #906 #899 #355 #342

[vsafonkin]

Msys2 was added to Windows images, this issue is closed.