Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Brew conflict on Python versions #16

Closed
GlassOfWhiskey opened this issue Aug 28, 2023 · 18 comments
Closed

Brew conflict on Python versions #16

GlassOfWhiskey opened this issue Aug 28, 2023 · 18 comments
Labels
bug Something isn't working dependencies Pull requests that update a dependency file

Comments

@GlassOfWhiskey
Copy link

Since Aug 25 2023, when I try to run this action I receive the following error

Run brew install --only-dependencies colima
  brew install --only-dependencies colima
  shell: /bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    TOXENV: py3.11-unit
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    PKG_CONFIG_PATH: /Users/runner/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig
    Python_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    Python2_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    Python3_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    BREW_CELLAR: /usr/local/Cellar
==> Fetching dependencies for colima: capstone, ca-certificates, libslirp, libssh, ncurses, snappy, vde and qemu
==> Fetching capstone
==> Downloading https://ghcr.io/v2/homebrew/core/capstone/manifests/4.0.2
==> Downloading https://ghcr.io/v2/homebrew/core/capstone/blobs/sha256:b617967c6e67a7be0d466fb6df8f8777a23a6bf73fe712220d656ea5d2ce527a
==> Fetching ca-certificates
==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/manifests/2023-08-22
==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/blobs/sha256:a331e92e7a759571296581f029e5cc2ec7cee70cd92dc0b5f8eb76095f94a21a
==> Fetching libslirp
==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/manifests/4.7.0
==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/blobs/sha256:e7e395084378af6bbc0582807938b023cb513cb5c99673b59c462429d077c25f
==> Fetching libssh
==> Downloading https://ghcr.io/v2/homebrew/core/libssh/manifests/0.10.5_1
==> Downloading https://ghcr.io/v2/homebrew/core/libssh/blobs/sha256:d3c33e1e13a0ce529aa1cb61242357f8c2ae90931402e9248f86178aab458bb3
==> Fetching ncurses
==> Downloading https://ghcr.io/v2/homebrew/core/ncurses/manifests/6.4-1
==> Downloading https://ghcr.io/v2/homebrew/core/ncurses/blobs/sha256:f31d17ab8166110d3dd0337b240c616acf33a3582c0ded2fbe4db61fbdda1b5d
==> Fetching snappy
==> Downloading https://ghcr.io/v2/homebrew/core/snappy/manifests/1.1.10
==> Downloading https://ghcr.io/v2/homebrew/core/snappy/blobs/sha256:1e9238c5f3f100b635ca74a17b3441d5f5f9c23007537107340d5397bcbd483d
==> Fetching vde
==> Downloading https://ghcr.io/v2/homebrew/core/vde/manifests/2.3.3
==> Downloading https://ghcr.io/v2/homebrew/core/vde/blobs/sha256:e203e8f3933c5dcdc45cdaae85f63b31ecd38a86d90eb5f4f0c1fd7825ad2347
==> Fetching qemu
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.1.0_1
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:9082a0867fb6fad925bf1b76f90670ef8e47ab8d26a66d1d883810c1e8a26479
==> Installing dependencies for colima: capstone, ca-certificates, libslirp, libssh, ncurses, snappy, vde and qemu
==> Installing colima dependency: capstone
==> Pouring capstone--4.0.2.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/capstone/4.0.2: 25 files, 14.6MB
==> Installing colima dependency: ca-certificates
==> Pouring ca-certificates--2023-08-22.ventura.bottle.tar.gz
==> Regenerating CA certificate bundle from keychain, this may take a while...
🍺  /usr/local/Cellar/ca-certificates/2023-08-22: 3 files, 221.7KB
==> Installing colima dependency: libslirp
==> Pouring libslirp--4.7.0.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/libslirp/4.7.0: 11 files, 346.9KB
==> Installing colima dependency: libssh
==> Pouring libssh--0.10.5_1.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/libssh/0.10.5_1: 23 files, 1.2MB
==> Installing colima dependency: ncurses
==> Pouring ncurses--6.4.ventura.bottle.1.tar.gz
🍺  /usr/local/Cellar/ncurses/6.4: 4,001 files, 9.4MB
==> Installing colima dependency: snappy
==> Pouring snappy--1.1.10.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/snappy/1.1.10: 18 files, 153.8KB
==> Installing colima dependency: vde
==> Pouring vde--2.3.3.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/vde/2.3.3: 66 files, 1MB
==> Installing colima dependency: qemu
==> Pouring qemu--8.1.0_1.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/qemu/8.1.0_1: 167 files, 529.3MB
==> Upgrading 4 dependents of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
freetds 1.3.19 -> 1.3.20, krb5 1.21.1 -> 1.21.2, python@3.10 3.10.12_1 -> 3.10.13, python@3.11 3.11.4_1 -> 3.11.5
==> Fetching krb5
==> Downloading https://ghcr.io/v2/homebrew/core/krb5/manifests/1.21.2
==> Downloading https://ghcr.io/v2/homebrew/core/krb5/blobs/sha256:3d61bf09ad35a994a36390723f15d2be2be9969a980884a45941300a8c9b33cf
==> Fetching freetds
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/manifests/1.3.20
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/blobs/sha256:417dbf14ad640826c8cadbbb089b1437a37058ef6c0b295ff1ec30e3e8f07761
==> Fetching dependencies for python@3.10: sqlite
==> Fetching sqlite
==> Downloading https://ghcr.io/v2/homebrew/core/sqlite/manifests/3.43.0
==> Downloading https://ghcr.io/v2/homebrew/core/sqlite/blobs/sha256:d2d619eaa851c53646f5e056733e467cd777e1d753b616c2437a876f9e02b13d
==> Fetching python@3.10
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.10/manifests/3.10.13
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.10/blobs/sha256:0c09cdc5a91d619fcbd6476d292e7c50e1dd2aa4688fcd10cf71296c2c358ccb
==> Fetching python@3.11
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/manifests/3.11.5
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/blobs/sha256:c87f0729bff2c3ab0cb3a66f7187ff0c621eed150573506c88a9e51597c4e732
==> Upgrading krb5
  1.21.1 -> 1.21.2 

==> Pouring krb5--1.21.2.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/krb5/1.21.2: 162 files, 4.9MB
==> Upgrading freetds
  1.3.19 -> 1.3.20 

==> Pouring freetds--1.3.20.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/freetds/1.3.20: 1,262 files, 13.7MB
==> Upgrading python@3.10
  3.10.12_1 -> 3.10.13 

==> Installing dependencies for python@3.10: sqlite
==> Installing python@3.10 dependency: sqlite
==> Pouring sqlite--3.43.0.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/sqlite/3.43.0: 11 files, 4.5MB
==> Installing python@3.10
==> Pouring python@3.10--3.10.13.ventura.bottle.tar.gz
==> /usr/local/Cellar/python@3.10/3.10.13/bin/python3.10 -Im ensurepip
==> /usr/local/Cellar/python@3.10/3.10.13/bin/python3.10 -Im pip install -v --no
🍺  /usr/local/Cellar/python@3.10/3.10.13: 3,096 files, 55.9MB
==> Upgrading python@3.11
  3.11.4_1 -> 3.11.5 

==> Pouring python@3.11--3.11.5.ventura.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.11

To list all files that would be deleted:
  brew link --overwrite --dry-run python@3.11

Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3-config
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config
==> /usr/local/Cellar/python@3.11/3.11.5/bin/python3.11 -Im ensurepip
==> /usr/local/Cellar/python@3.11/3.11.5/bin/python3.11 -Im pip install -v --no-
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.11/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.11/site-packages

tkinter is no longer included with this formula, but it is available separately:
  brew install python-tk@3.11

gdbm (`dbm.gnu`) is no longer included in this formula, but it is available separately:
  brew install python-gdbm@3.11
`dbm.ndbm` changed database backends in Homebrew Python 3.11.
If you need to read a database from a previous Homebrew Python created via `dbm.ndbm`,
you'll need to read your database using the older version of Homebrew Python and convert to another format.
`dbm` still defaults to `dbm.gnu` when it is installed.

For more information about Homebrew and Python, see: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python@3.11/3.11.5: 3,287 files, 61MB
==> Checking for dependents of upgraded formulae...
==> No broken dependents found!
==> Caveats
==> python@3.11
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.11/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.11/site-packages

tkinter is no longer included with this formula, but it is available separately:
  brew install python-tk@3.11

gdbm (`dbm.gnu`) is no longer included in this formula, but it is available separately:
  brew install python-gdbm@3.11
`dbm.ndbm` changed database backends in Homebrew Python 3.11.
If you need to read a database from a previous Homebrew Python created via `dbm.ndbm`,
you'll need to read your database using the older version of Homebrew Python and convert to another format.
`dbm` still defaults to `dbm.gnu` when it is installed.

For more information about Homebrew and Python, see: https://docs.brew.sh/Homebrew-and-Python
Error: Process completed with exit code 1.

There is a conflict between upgraded Python versions related to the /usr/local/bin/2to3 file. There is no difference in the environment (runner version, OS version, image provisioner, etc.) between the working and the broken runs. The only difference is that before Aug 25 the only indirect dependency loaded by Homebrew was krb

==> Upgrading 1 dependent of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
krb5 1.21.1 -> 1.21.2

Conversely, starting form Aug 25, the two Python versions started to appear, probably due to some update in the Homebrew package dependencies

==> Upgrading 3 dependents of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
krb5 1.21.1 -> 1.21.2, python@3.10 3.10.12_1 -> 3.10.13, python@3.11 3.11.4_1 -> 3.11.5

As a workaround, I solved the issue by explicitly specifying HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK in the step's environment variables:

name: "Install Docker (MacOs X)"
uses: douglascamata/setup-docker-macos-action@v1-alpha.9
env:
  HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
@GlassOfWhiskey
Copy link
Author

You can verify yourself by exploring the workflow runs on the streamflow repository.

@douglascamata
Copy link
Owner

douglascamata commented Aug 28, 2023 •
edited
Loading

@GlassOfWhiskey I recommend that you first run this action before setting up Python. This seems to be a conflict between the Python version installed by action/setup-python and the one from Homebrew. Ideally, action/setup-python should find the one installed by Homebrew and use it.

Unfortunately I cannot remove HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK as it can cause newly installed/upgrade "main packages" (docker, lima, colima, qemu) to use old versions of their dependencies, causing all sorts of weird issues.

@douglascamata douglascamata added bug Something isn't working dependencies Pull requests that update a dependency file labels Aug 28, 2023
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 28, 2023
@GlassOfWhiskey
Fix Python dependencies conflict in Mac OS CI
c9785d1 
This commit modifies the Mac OS X CI workflow as suggested in
douglascamata/setup-docker-macos-action#16 to prevent conflicting
symlink error due to cuncurrent Python updates.
@GlassOfWhiskey GlassOfWhiskey mentioned this issue Aug 28, 2023
Merged
@GlassOfWhiskey
Copy link
Author

@douglascamata I tried your suggestion in alpha-unito/streamflow#214 but unfortunately it didn't work. Even when running as the first action in a workflow, this action at the moment updates both Python 3.10 an Python 3.11, causing the conflict.

@douglascamata
Copy link
Owner

Hummm, I see. 🤔

I think I can remove some "old" workarounds that should allow us to set HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=true again without breaking things. I'll try it out.

@douglascamata douglascamata mentioned this issue Aug 28, 2023
Closed
@douglascamata
Copy link
Owner

Strongly considering to use nix here to avoid having to deal with Homebrew's conflicts with system-wide packages.

@commjoen
Copy link

commjoen commented Aug 30, 2023 •
edited
Loading

Having a similar issue here, though setting the workaround as described above

- uses: douglascamata/setup-docker-macos-action@v1-alpha.9
        env:
          HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1

Did not work for me at https://github.com/OWASP/wrongsecrets/pull/945/files with output at https://github.com/OWASP/wrongsecrets/actions/runs/6020963466/job/16333057066?pr=945 . Do you know what I can best do to still be able to use your awesome workflow @douglascamata ? Can you help me please?

@douglascamata
Copy link
Owner

@commjoen your issue now is very different though and it's a new issue we all will have. The dtc link error happens even in my two macOS computers. Probably something's broken in Homebrew. 🤷

@douglascamata
Copy link
Owner

douglascamata commented Aug 30, 2023 •
edited
Loading

And it seems that I am correct: Homebrew/homebrew-core#140677.

@commjoen
Copy link

@commjoen your issue now is very different though and it's a new issue we all will have. The dtc link error happens even in my two macOS computers. Probably something's broken in Homebrew. 🤷

Thanks! anything i can do to fix my own issue?

@douglascamata
Copy link
Owner

douglascamata commented Aug 30, 2023 •
edited
Loading

@commjoen I don't know... I didn't even have time yet to fix the Python version issue. I think you can manually instal dtc and do a brew link --force dtc. Your action clearly downloaded a bottle that was already rebuilt, given the file name and sha256. We need to complain in the Homebrew issue.

To me this is one more reason to leave Homebrew. Broken bottle builds and other brew issues are the source of all problems this action has. 😓

@douglascamata
Copy link
Owner

I'm trying different approaches to avoid the conflicts altogether, give me some time.

@douglascamata
Copy link
Owner

I got a fix almost there in #19. Should be able to finish and merge it later today.

@douglascamata douglascamata mentioned this issue Sep 4, 2023
Closed
@douglascamata
Copy link
Owner

Hey folks, new release's out!

Tags v1-alpha and v1-alpha.10 are updated.

Please try it out and let me know how it goes.

@douglascamata
Copy link
Owner

The action will now try to avoid upgrading/reinstalling QEMU if it's already present in the system. If you want to force a QEMU upgrade, use the new input for it (see https://github.com/douglascamata/setup-docker-macos-action?tab=readme-ov-file#inputsupgrade-qemu-defaults-to-false).

@commjoen
Copy link

commjoen commented Sep 4, 2023

OWASP/wrongsecrets#976 Seems to work

@douglascamata
Copy link
Owner

Thanks for the report, @commjoen. 👍

I'll give the other participants here some time to try it out and reply. Will close the issue if the feedback is positive or there's no more activity in between a week to ten days from now.

GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Sep 5, 2023
@GlassOfWhiskey
Fix Python dependencies conflict in Mac OS CI
f3d83b7 
This commit modifies the Mac OS X CI workflow as suggested in
douglascamata/setup-docker-macos-action#16 to prevent conflicting
symlink error due to cuncurrent Python updates.
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Sep 5, 2023
@GlassOfWhiskey
Fix Python dependencies conflict in Mac OS CI
3b564aa 
This commit updates the douglascamata/setup-docker-macos-action to
version `v1-alpha.10` to prevent conflicting symlink error due to
concurrent Python updates (see
douglascamata/setup-docker-macos-action#16)
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Sep 5, 2023
@GlassOfWhiskey
Fix Python dependencies conflict in Mac OS CI
21ac9d2 
This commit updates the `douglascamata/setup-docker-macos-action` to
version `v1-alpha.10` to prevent conflicting symlink error due to
concurrent Python updates (see
douglascamata/setup-docker-macos-action#16)
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Sep 5, 2023
@GlassOfWhiskey
Fix Python dependencies conflict in Mac OS CI (#214)
49368d2 
This commit updates the `douglascamata/setup-docker-macos-action` to
version `v1-alpha.10` to prevent conflicting symlink error due to
concurrent Python updates (see
douglascamata/setup-docker-macos-action#16)
@GlassOfWhiskey
Copy link
Author

Version v1-alpha.10 solved Python conflict in StreamFlow. Thank you very much for your help

@douglascamata
Copy link
Owner

My pleasure, @GlassOfWhiskey. I'm closing this for now. Thanks for the report and testing. Hope this solution will last for a while (forever ideally). 🙇

GlassOfWhiskey added a commit to alpha-unito/streamflow-postgresql that referenced this issue Sep 11, 2023
@GlassOfWhiskey
Fix Python dependencies conflict in Mac OS CI
4716ce6 
This commit updates the `douglascamata/setup-docker-macos-action`
to version `v1-alpha.10` to prevent conflicting symlink error due
to concurrent Python updates (see
douglascamata/setup-docker-macos-action#16).
@GlassOfWhiskey GlassOfWhiskey mentioned this issue Sep 11, 2023
Merged
GlassOfWhiskey added a commit to alpha-unito/streamflow-postgresql that referenced this issue Sep 11, 2023
@GlassOfWhiskey
Fix Python dependencies conflict in Mac OS CI (#34)
f4f8000 
This commit updates the `douglascamata/setup-docker-macos-action`
to version `v1-alpha.10` to prevent conflicting symlink error due
to concurrent Python updates (see
douglascamata/setup-docker-macos-action#16).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working dependencies Pull requests that update a dependency file
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@douglascamata @commjoen @GlassOfWhiskey