Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setuptools changes underscore to dash for package name #2522

Closed
LorenLiu opened this issue Jan 6, 2021 · 8 comments · Fixed by #4159
Closed

setuptools changes underscore to dash for package name #2522

LorenLiu opened this issue Jan 6, 2021 · 8 comments · Fixed by #4159

Comments

@LorenLiu
Copy link

LorenLiu commented Jan 6, 2021

After I pip install cx_oracle, I use python -m pip show cx_oracle command and it shows the package name is cx-Oracle
After discussed with cx_Oracle's author, it shows setuptools changed the underscore to dash for the package name. Is that an issue for setuptools or is there any way to stop this auto-conversion? Thanks.

setup.cfg for cx_Oracle
[metadata]
name = cx_Oracle

PKG-INFO for cx_Oracle
Metadata-Version: 2.1
Name: cx-Oracle

setup.py used in cx_Oracel

setup the extension

extension = setuptools.Extension(
name = "cx_Oracle",
include_dirs = includeDirs,
extra_compile_args = extraCompileArgs,
define_macros = [("CXO_BUILD_VERSION", BUILD_VERSION)],
extra_link_args = extraLinkArgs,
sources = sources + dpiSources,
depends = depends,
libraries = libraries,
library_dirs = libraryDirs)

@jaraco
Copy link
Member

jaraco commented Jan 17, 2021

It looks like when Setuptools resolves the name for the metadata, it uses pkg_resources.safe_name, referencing the "[unspecified] standard distribution name" that replaces underscores with dashes.

But looking at recent PEPs like PEP 508, it does look like Names are allowed to contain underscores.

Probably egg_info needs to be updated to allow for underscores (and anything else allowed be the new standard). Who knows what havoc that will wreak.

bugfood added a commit to bugfood/fpm that referenced this issue Jul 28, 2021
This works around:
pypa/setuptools#2522

For example, when building a module which depends on 'backports_abc',
fpm will currently translate that into a dependency on 'backports-abc'.
When building the 'backports_abc' module, fpm will currently retain the
original name. Thus, it is not possible to use fpm to meet some
dependencies.

Underscores should be allowed in package names, but the most important
thing is to be internally consistent. If and when setuptools fixes
safe_name(), then the behavior will automatically change and remain
consistent.

Note that this change matches both methods ok loading requirements:
* for setup.py, setuptools internally calls pkg_resources.safe_name()
* for requirements.txt, pkg_resources uses its own safe_anem()
bugfood added a commit to bugfood/fpm that referenced this issue Jul 28, 2021
This works around:
pypa/setuptools#2522

For example, when building a module which depends on 'backports_abc',
fpm will currently translate that into a dependency on 'backports-abc'.
When building the 'backports_abc' module, fpm will currently retain the
original name. Thus, it is not possible to use fpm to meet some
dependencies.

Underscores should be allowed in package names, but the most important
thing is to be internally consistent. If and when pkg_resources fixes
safe_name(), then the behavior will automatically change and remain
consistent.

Note that this change matches both methods ok loading requirements:
* for setup.py, setuptools internally calls pkg_resources.safe_name()
* for requirements.txt, pkg_resources uses its own safe_name()
bugfood added a commit to bugfood/fpm that referenced this issue Jul 28, 2021
This works around:
pypa/setuptools#2522

For example, when building a module which depends on 'backports_abc',
fpm will currently translate that into a dependency on 'backports-abc'.
When building the 'backports_abc' module, fpm will currently retain the
original name. Thus, it is not possible to use fpm to meet some
dependencies.

Underscores should be allowed in package names, but the most important
thing is to be internally consistent. If and when pkg_resources fixes
safe_name(), then the behavior will automatically change and remain
consistent.

Note that this change matches both methods ok loading requirements:
* for setup.py, setuptools internally calls pkg_resources.safe_name()
* for requirements.txt, pkg_resources uses its own safe_name()
bugfood added a commit to bugfood/fpm that referenced this issue Jul 29, 2021
This works around:
pypa/setuptools#2522

For example, when building a module which depends on 'backports_abc',
fpm will currently translate that into a dependency on 'backports-abc'.
When building the 'backports_abc' module, fpm will currently retain the
original name. Thus, it is not possible to use fpm to meet some
dependencies.

Underscores should be allowed in package names, but the most important
thing is to be internally consistent. Monkey-patch
pkg_resources.safe_name() in order to allow underscores (default for fpm)
or disallow underscores, according to user specification.

If and when pkg_resources fixes safe_name(), then the behavior of fpm
will remain the same.

Note that this change matches both methods of loading requirements:
* for setup.py, setuptools internally calls pkg_resources.safe_name()
* for requirements.txt, pkg_resources uses its own safe_name()
bugfood added a commit to bugfood/fpm that referenced this issue Nov 11, 2021
This works around:
pypa/setuptools#2522

For example, when building a module which depends on 'backports_abc',
fpm will currently translate that into a dependency on 'backports-abc'.
When building the 'backports_abc' module, fpm will currently retain the
original name. Thus, it is not possible to use fpm to meet some
dependencies.

Underscores should be allowed in package names, but the most important
thing is to be internally consistent. Monkey-patch
pkg_resources.safe_name() in order to allow underscores (default for fpm)
or disallow underscores, according to user specification.

If and when pkg_resources fixes safe_name(), then the behavior of fpm
will remain the same.

Note that this change matches both methods of loading requirements:
* for setup.py, setuptools internally calls pkg_resources.safe_name()
* for requirements.txt, pkg_resources uses its own safe_name()
bugfood added a commit to bugfood/fpm that referenced this issue Mar 10, 2023
This works around:
pypa/setuptools#2522

For example, when building a module which depends on 'backports_abc',
fpm will currently translate that into a dependency on 'backports-abc'.
When building the 'backports_abc' module, fpm will currently retain the
original name. Thus, it is not possible to use fpm to meet some
dependencies.

Underscores should be allowed in package names, but the most important
thing is to be internally consistent. Monkey-patch
pkg_resources.safe_name() in order to allow underscores (default for fpm)
or disallow underscores, according to user specification.

If and when pkg_resources fixes safe_name(), then the behavior of fpm
will remain the same.

Note that this change matches both methods of loading requirements:
* for setup.py, setuptools internally calls pkg_resources.safe_name()
* for requirements.txt, pkg_resources uses its own safe_name()
@sztomi
Copy link

sztomi commented Sep 6, 2023

This is still an issue.

@livein21st
Copy link

This is still an issue for me as well

@jaraco
Copy link
Member

jaraco commented Dec 14, 2023

I can confirm that with #4159, the issue is addressed:

 setuptools bugfix/2522-retain-valid-name @ .tox/py/bin/pip install --no-cache --no-build-isolation cx-oracle
Collecting cx-oracle
  Downloading cx_Oracle-8.3.0.tar.gz (363 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 363.9/363.9 kB 8.8 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cx-oracle
  Building wheel for cx-oracle (pyproject.toml) ... done
  Created wheel for cx-oracle: filename=cx_Oracle-8.3.0-cp312-cp312-macosx_14_0_arm64.whl size=178447 sha256=36d93f88ded91ec3726325bc9bb1c2de6493af624c9acecefbe22fadd5de3554
  Stored in directory: /private/var/folders/sx/n5gkrgfx6zd91ymxr2sr9wvw00n8zm/T/pip-ephem-wheel-cache-5slq_i5v/wheels/c2/36/74/ca03900db8246dfd872cb23f51299a034ff843b7c15e87b5dd
Successfully built cx-oracle
Installing collected packages: cx-oracle
Successfully installed cx-oracle-8.3.0
 setuptools bugfix/2522-retain-valid-name @ head -n 3 .tox/py/lib/python3.12/site-packages/cx_Oracle-8.3.0.dist-info/METADATA
Metadata-Version: 2.1
Name: cx_Oracle
Version: 8.3.0

@jaraco
Copy link
Member

jaraco commented Dec 14, 2023

Releasing as v69.0.3.

@sztomi
Copy link

sztomi commented Dec 15, 2023

Thank you!

toofar added a commit to qutebrowser/qutebrowser that referenced this issue Dec 26, 2023
In `pylint_checkers/setup.py` we define the package name as `qute_pylint`. When
setuptools was creating the egg (or wheel?) it was sanitizing the package name
replacing all problematic chars, including underscore, with a hyphen (-). So
the output of `pip freeze`, among other things, have the name with a hyphen:
`qute-pylint`. This was fixed in setuptools so underscores are no longer
sanitized: pypa/setuptools#2522

Change the regex to include both just in case someone is running it with the
old setuptools or something. Also because I'm not able to reproduce the
hyphen version of the name locally, not sure why. Maybe it depends on your
python or importlib version too?
@johannkm
Copy link

Just a heads up: we were accidentally relying on underscores becoming hyphens, I suspect other projects will hit this as well 🙂

johannkm added a commit to dagster-io/dagster that referenced this issue Dec 27, 2023
Open questions:

- ~~why did tests start failing this weekend? (best guess is setuptools
69.03 on Dec 23rd)~~
- ~~why does this fix the failures?~~
- would this be a breaking change? I'm hopeful that it's not, because
the package is already published as dagster-webserver
https://pypi.org/project/dagster-webserver/

Edit: the first two are answered by
pypa/setuptools#2522 (comment):
setuptools change how they handle hyphens vs underscores

This makes the two required changes to get the build green (these two
are special because of the underscore packages, they're the ones that
are relied on in tests). In general we should make it a pattern to make
package names use hyphens and match their directory name to avoid
confusion.
johannkm added a commit to dagster-io/dagster that referenced this issue Dec 27, 2023
Open questions:

- ~~why did tests start failing this weekend? (best guess is setuptools
69.03 on Dec 23rd)~~
- ~~why does this fix the failures?~~
- would this be a breaking change? I'm hopeful that it's not, because
the package is already published as dagster-webserver
https://pypi.org/project/dagster-webserver/

Edit: the first two are answered by
pypa/setuptools#2522 (comment):
setuptools change how they handle hyphens vs underscores

This makes the two required changes to get the build green (these two
are special because of the underscore packages, they're the ones that
are relied on in tests). In general we should make it a pattern to make
package names use hyphens and match their directory name to avoid
confusion.
@henryiii
Copy link
Contributor

henryiii commented Jan 16, 2024

Build was checking metadata in its test suite, but happy to adapt to this.

By the way, I think this affects SDist naming (SDist names should be normalized, but setuptools doesn't seem to do that). We got a report of an SDist being made with spaces in the name, which is invalid. I'm suspecting this change might be related.

# setup.cfg
[metadata]
name = This is not cool
version = 0.0.1
$ touch pyproject.toml
$ pipx run build .
...
Successfully built This is not cool-0.0.1.tar.gz and This_is_not_cool-0.0.1-py3-none-any.whl

I believe before this change, you'd have made this-is-not-cool-0.0.1.tar.gz (which should have been this_is_not_cool-0.0.1.tar.gz, normalized like wheels, IMO, and as far as I've been told, all other backends normalize that way). The file with spaces in it is invalid on PyPI. The upload failure even tells you Start filename for 'Zipped-Album-Player' with 'zipped_album_player'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants