PEP 639: Implement License-Expression and License-File #828

ewdurbin · 2024-09-03T16:10:11Z

~~Work in progress, current open questions:~~

~~Is the dependency on license-expression OK? Should vendoring or a new parser be considered?~~
- ~~went the vendoring route per feedback from @abravalheri~~
- migrated to a pure python parser based on pypa/hatchling's implementation
How much validation to apply to License-File, as-is as far as metadata goes the only clear spec is "Path delimiters MUST be the forward slash character (/), and parent directory indicators (..) MUST NOT be used.". It is unclear how to check for other delimiters, as paths may have no delimiters whatsoever
- validations with path lib seem to fit the bill!
~~How to handle deprecations/conflicts. License and License-Expression are mutually exclusive, but it doesn't appear that such a conflict has existed before.~~
- Based on the current behavior of the library, this kind of concern appears to be left to the caller. The metadata module appears to be focused on correctly parsing individual fields.

pyproject.toml

cdce8p · 2024-09-04T11:34:27Z

IIRC one of the concerns raised in the Discourse thread was regarding the total file size if license_expression would be added as a dependency or fully vendored. Just as example, the scancode-licensedb-index.json file is over 800kB.

I believe that's one of the reasons @ofek went a slightly different route with #799.

ewdurbin · 2024-09-04T13:19:08Z

See a828bc7, which is a variant of #799 to build/use a minimal compatible json blob from the upstream for use with the vendored license-expression. Also excluding the data files from the vendored package from the resulting sdist/wheel to save space.

Overall resulting size difference for sdist and wheel

file	24.1	before	after
sdist	148 kB	276 kB	212 kB
whl	54 kB	180 kB	112 kB

uncompressed:

file	24.1	before	after
sdist	2539 kB*	1690 kB	829 kB
whl	176 kB	1281 kB	426 kB

* this appears to be due to the inclusion of tests/.pytest_cache/v/cache/lastfailed and tests/.pytest_cache/v/cache/nodeids files in the sdist, without it it is 608 kB.

ofek

Can you please save the data as Python so deserialization is not required?
I think we shouldn't merge this until we satisfy the parsing needs of backends (i.e. user defined metadata) rather than just consumers of packages like PyPI.
Although I'm not a maintainer, I would be quite weary of packaging beginning to vendor dependencies.

ewdurbin · 2024-09-04T16:34:48Z

Can you please save the data as Python so deserialization is not required?

Looked closer and saw that hatchling's implementation is more complete than I initially understood, so I've migrated this PR to use an adapted version of that, and moved storage of the license data back to using the python based representation

I think we shouldn't merge this until we satisfy the parsing needs of backends (i.e. user defined metadata) rather than just consumers of packages like PyPI.

Which needs are those? Can you delineate them? I'm interested in seeing this merged so PyPI's implementation can progress.

Although I'm not a maintainer, I would be quite weary of packaging beginning to vendor dependencies.

See response to 1. :)

ofek · 2024-09-04T16:49:05Z

I think we shouldn't merge this until we satisfy the parsing needs of backends (i.e. user defined metadata) rather than just consumers of packages like PyPI.

Which needs are those? Can you delineate them? I'm interested in seeing this merged so PyPI's implementation can progress.

Now that you have normalize_license_expression which also raises an exception for invalid stuff, every backend would be satisfied. Thanks!

src/packaging/licenses/__init__.py

ofek

Looks fantastic to my non-maintainer eyes!

befeleme · 2024-09-05T09:55:06Z

Just a tiny nitpick: The commit message contains a typo: "PEP 693" instead of 639 - this will make digging in the git history harder in the future.

hroncok

How much validation to apply to License-File, as-is as far as metadata goes the only clear spec is "Path delimiters MUST be the forward slash character (/), and parent directory indicators (..) MUST NOT be used.". It is unclear how to check for other delimiters, as paths may have no delimiters whatsoever

I think the spirit of the rule is that there are no backlash-delimeters in the path. However, I am not sure if this is enforceable if a filename can contain a backlash.

Also, as the path MUST be relative, we could assert it does not start with /.

I also noticed a license file is never checked for existence. Is that intentional? The PEP states it MUST be present (and the file content MUST be UTF-8 encoded text).

tests/test_metadata.py

although inconsistent (see Requires-External, Provides-Extra, Provides-Dist, Obsoletes-Dist...) pluralize the result for multiple use of `License-File` https://packaging.pypa.io/en/stable/metadata.html#packaging.metadata.RawMetadata > Any core metadata field that can be specified multiple times or can hold multiple values in a single field have a key with a plural name.

Per feedback, integrate a variant of pypa#799 that builds a minimal JSON dataset to feed vendored license-expression 32K src/packaging/_spdx.json vs 848K src/packaging/_vendor/license_expression/data/scancode-licensedb-index.json

- back away from vendoring - adapt hatchling's license expression parser to accept well-formed LicenseRef- - back to python storage of license data

ewdurbin · 2024-09-05T10:30:04Z

Just a tiny nitpick: The commit message contains a typo: "PEP 693" instead of 639 - this will make digging in the git history harder in the future.

🤦🏼 resolved with a rebase/force-push

since the value is typed as a TypedDict, mypy will detect if new keys appear or others disappear

brettcannon · 2024-09-13T17:57:22Z

@ewdurbin I don't know why GitHub is showing my review as pending. Just let me know when you're ready for me to look at this PR again.

pradyunsg

I'll follow up separately with a review of the code but this LGTM based on a cursory look of that.

This also needs documentation, which I think we should add in this PR itself, along with using autodoc here.

src/packaging/licenses/spdx.py

src/packaging/licenses/__init__.py

tasks/licenses.py

ewdurbin · 2024-09-15T12:37:07Z

Docs added in 701217b, depends on pypa/packaging.python.org#1595

ewdurbin · 2024-09-15T12:37:55Z

re-requesting review from @pradyunsg and @brettcannon.

src/packaging/licenses/__init__.py

henryiii · 2024-09-19T05:24:45Z

src/packaging/licenses/__init__.py

+
+    python_expression = " ".join(python_tokens)
+    try:
+        result = eval(python_expression)


Could globals and locals be passed in here? That feels safer, regardless if it is or not.

…ady padded by spaces

ewdurbin · 2024-09-19T13:02:35Z

Outstanding un-resolved comments I am not opinionated on:

If some consensus around these could be found, are we good to proceed with a merge?

ewdurbin mentioned this pull request Sep 3, 2024

Implement PEP 639 pypi/warehouse#16620

Open

abravalheri reviewed Sep 3, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

ewdurbin force-pushed the pep_639 branch 3 times, most recently from 4bbae23 to e3dec5c Compare September 3, 2024 23:27

This was referenced Sep 4, 2024

[FR] Implement PEP 639 pdm-project/pdm-backend#256

Closed

Implement PEP 639 python-poetry/poetry#9670

Open

ewdurbin force-pushed the pep_639 branch from 7eae541 to fd1d83b Compare September 4, 2024 13:15

ofek reviewed Sep 4, 2024

View reviewed changes

ewdurbin force-pushed the pep_639 branch from d72a43f to b8e03f2 Compare September 4, 2024 16:36

ewdurbin force-pushed the pep_639 branch from b8e03f2 to c1530bd Compare September 4, 2024 17:01

ofek reviewed Sep 4, 2024

View reviewed changes

src/packaging/licenses/__init__.py Outdated Show resolved Hide resolved

ewdurbin mentioned this pull request Sep 4, 2024

Implement PEP 639 #829

Open

ofek approved these changes Sep 4, 2024

View reviewed changes

hroncok reviewed Sep 5, 2024

View reviewed changes

tests/test_metadata.py Outdated Show resolved Hide resolved

ewdurbin added 7 commits September 5, 2024 06:29

PEP 639: Implement License-Expression and License-File

de8b239

wire in vendoring library to manage vendored dependencies

44beda7

vendor license_expression and boolean.py

2b1a9f7

migrate to parser based on hatchling's

cec33f1

- back away from vendoring - adapt hatchling's license expression parser to accept well-formed LicenseRef- - back to python storage of license data

string -> re

cfb3af1

ewdurbin force-pushed the pep_639 branch from 2e7cfb8 to cfb3af1 Compare September 5, 2024 10:29

ewdurbin changed the title ~~PEP 693: Implement License-Expression and License-File~~ PEP 639: Implement License-Expression and License-File Sep 5, 2024

update typing for licenses.spdx

396e4ef

ewdurbin force-pushed the pep_639 branch from 190c23f to 396e4ef Compare September 13, 2024 14:38

ewdurbin added 9 commits September 13, 2024 10:48

Extend typing improvements in licenses.spdx to include Exception

21a2821

fixup names, Exception is not a good one.

4ac18f0

better enforcement of license-file paths

46a7491

subclass ValueError for invalid license expressions

cd7105f

and empty license expression is invalid

e469b7e

create a "NormalizedLicenseExpression" type

e699391

rename normalize -> canonicalize

22fa9cd

add tests to ensure license and exception ids conform

f952ab9

since the value is typed as a TypedDict, mypy will detect if new keys appear or others disappear

update name of var

30e34f1

ewdurbin force-pushed the pep_639 branch from 2214401 to 30e34f1 Compare September 13, 2024 16:25

match formatting standards

8906b16

brettcannon removed their request for review September 13, 2024 17:57

pradyunsg reviewed Sep 15, 2024

View reviewed changes

src/packaging/licenses/spdx.py Outdated Show resolved Hide resolved

src/packaging/licenses/__init__.py Show resolved Hide resolved

tasks/licenses.py Outdated Show resolved Hide resolved

tasks/licenses.py Outdated Show resolved Hide resolved

tasks/licenses.py Outdated Show resolved Hide resolved

ewdurbin added 4 commits September 15, 2024 08:05

reorganize the licenses module a bit

a361294

apply code-review suggestions for update_licenses task

9cee38e

fix tests after spdx module was made private

81efbda

add docs

701217b

ewdurbin force-pushed the pep_639 branch from 57b4318 to 701217b Compare September 15, 2024 12:33

ewdurbin requested review from brettcannon and pradyunsg September 15, 2024 12:37

Merge branch 'main' into pep_639

4539543

henryiii reviewed Sep 19, 2024

View reviewed changes

src/packaging/licenses/__init__.py Outdated Show resolved Hide resolved

henryiii reviewed Sep 19, 2024

View reviewed changes

add additional test cases and handle LicenseRef- identifiers not alre…

64d3647

…ady padded by spaces

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEP 639: Implement License-Expression and License-File #828

PEP 639: Implement License-Expression and License-File #828

ewdurbin commented Sep 3, 2024 •

edited

Loading

cdce8p commented Sep 4, 2024

ewdurbin commented Sep 4, 2024 •

edited

Loading

ofek left a comment

ewdurbin commented Sep 4, 2024

ofek commented Sep 4, 2024 •

edited

Loading

ofek left a comment

befeleme commented Sep 5, 2024 •

edited

Loading

hroncok left a comment •

edited

Loading

ewdurbin commented Sep 5, 2024

brettcannon commented Sep 13, 2024

pradyunsg left a comment

ewdurbin commented Sep 15, 2024

ewdurbin commented Sep 15, 2024

henryiii Sep 19, 2024

ewdurbin commented Sep 19, 2024

PEP 639: Implement License-Expression and License-File #828

Are you sure you want to change the base?

PEP 639: Implement License-Expression and License-File #828

Conversation

ewdurbin commented Sep 3, 2024 • edited Loading

cdce8p commented Sep 4, 2024

ewdurbin commented Sep 4, 2024 • edited Loading

ofek left a comment

Choose a reason for hiding this comment

ewdurbin commented Sep 4, 2024

ofek commented Sep 4, 2024 • edited Loading

ofek left a comment

Choose a reason for hiding this comment

befeleme commented Sep 5, 2024 • edited Loading

hroncok left a comment • edited Loading

Choose a reason for hiding this comment

ewdurbin commented Sep 5, 2024

brettcannon commented Sep 13, 2024

pradyunsg left a comment

Choose a reason for hiding this comment

ewdurbin commented Sep 15, 2024

ewdurbin commented Sep 15, 2024

henryiii Sep 19, 2024

Choose a reason for hiding this comment

ewdurbin commented Sep 19, 2024

ewdurbin commented Sep 3, 2024 •

edited

Loading

ewdurbin commented Sep 4, 2024 •

edited

Loading

ofek commented Sep 4, 2024 •

edited

Loading

befeleme commented Sep 5, 2024 •

edited

Loading

hroncok left a comment •

edited

Loading