feat(python): add support for uv dev and optional dependencies #8134

nikpivkin · 2024-12-19T07:44:36Z

Description

This PR adds support for optional and dev dependencies. Optional dependencies are handled the same way for poetry. The optional dependencies of the root package are added as direct and the rest as indirect.

Without `--include-dev-deps`

❯ ./trivy fs /Users/nikita/projects/trivy-test/uv-test -q --list-all-pkgs -f json | jq '.Results[] | .Packages[] | select(.Name == "pytest")'

With`--include-dev-deps`

❯ ./trivy fs /Users/nikita/projects/trivy-test/uv-test -q --list-all-pkgs -f json --include-dev-deps | jq '.Results[] | .Packages[] | select(.Name == "pytest")'
{
  "ID": "pytest@8.3.4",
  "Name": "pytest",
  "Identifier": {
    "PURL": "pkg:pypi/pytest@8.3.4",
    "UID": "5ccb63bab9c8e426"
  },
  "Version": "8.3.4",
  "Dev": true,
  "Relationship": "direct",
  "DependsOn": [
    "colorama@0.4.6",
    "iniconfig@2.0.0",
    "packaging@24.2",
    "pluggy@1.5.0"
  ],
  "Layer": {}
}

pyptoject.toml:

[project]
name = "uv-test"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"

[dependency-groups]
test = [
    "pytest>=8.3.4",
]

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

knqyf263 · 2024-12-19T10:39:06Z

docs/docs/coverage/language/python.md

@@ -27,7 +27,7 @@ The following table provides an outline of the features Trivy offers.
 | pip             | requirements.txt |            -            |     Include      |                  -                   |    ✓     |                    ✓                     |
 | Pipenv          | Pipfile.lock     |            ✓            |     Include      |                  -                   |    ✓     |                Not needed                |
 | Poetry          | poetry.lock      |            ✓            |     Exclude      |                  ✓                   |    -     |                Not needed                |
-| uv              | uv.lock          |            ✓            |     Exclude      |                  ✓                   |    -     |                Not needed                |
+| uv              | uv.lock          |            ✓            |     Include      |                  ✓                   |    -     |                Not needed                |


I don't remember why this page doesn't mention --include-dev-deps like Node.js, but we should.

Can use link for Include as in nodejs page?

knqyf263 · 2024-12-23T06:16:45Z

pkg/dependency/parser/python/uv/parse_testcase.go

 	// apk add jq
 	// uv pip list --format json |jq -c 'sort_by(.name) | .[] | {"ID": (.name + "@" + .version), "Name": .name, "Version": .version}' | sed 's/$/,/' | sed 's/\"\([^"]*\)\":/\1:/g'

 	// add a root project
 	// fill in the relationships between the packages
 	uvNormal = []ftypes.Package{
 		{ID: "normal@0.1.0", Name: "normal", Version: "0.1.0", Relationship: ftypes.RelationshipRoot},
+		{ID: "httpx@0.28.1", Name: "httpx", Version: "0.28.1", Relationship: ftypes.RelationshipDirect},


This package is not marked as a development dependency. Is it correct? I'm concerned transitive dependencies introduced by direct development dependencies are not marked correctly.

httpx is not a development dependency: uv add httpx==0.28.1 --extra socks

Why did we newly introduce this dependency? I thought the test case was updated for optional or development dependencies.

Did we need it to test extra packages?

Ah, I should have added about test cases in the description. Yes, I added some more test cases:

An optional dependency in the root package

Direct dependency with an extra dependency that is optional

OK, thanks.

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

knqyf263 · 2024-12-24T05:47:57Z

@nikpivkin Can you rebase the main branch? I think the linter will fail on map[string]struct{}.

nikpivkin · 2024-12-24T05:51:55Z

I'm working on it right now

DmitriyLewen · 2024-12-24T05:51:27Z

docs/docs/coverage/language/python.md

@@ -27,7 +27,7 @@ The following table provides an outline of the features Trivy offers.
 | pip             | requirements.txt |            -            |     Include      |                  -                   |    ✓     |                    ✓                     |
 | Pipenv          | Pipfile.lock     |            ✓            |     Include      |                  -                   |    ✓     |                Not needed                |
 | Poetry          | poetry.lock      |            ✓            |     Exclude      |                  ✓                   |    -     |                Not needed                |
-| uv              | uv.lock          |            ✓            |     Exclude      |                  ✓                   |    -     |                Not needed                |
+| uv              | uv.lock          |            ✓            |     Include      |                  ✓                   |    -     |                Not needed                |


Can use link for Include as in nodejs page?

DmitriyLewen · 2024-12-24T05:53:36Z

docs/docs/coverage/language/python.md

@@ -104,6 +104,8 @@ urllib3==1.26.15
 `requirements.txt` files don't contain information about dependencies used for development.
 Trivy could detect vulnerabilities on the development packages, which not affect your production environment.

+By default, Trivy doesn't report development dependencies. Use the `--include-dev-deps` flag to include them.


Trivy doesn't support --include-dev-deps flag for pip

Fixed 55fb02f

Can use link for Include as in nodejs page?

By the way, why are include/exclude used to link the table and sections and not the package manager names from the first column?

By default, we don't show Dev dependencies.
This link is intended to draw users attention to this and direct them to the section about the file, where they can see that --include-dev-deps flag needs to be used.

DmitriyLewen · 2024-12-24T05:55:27Z

docs/docs/coverage/language/python.md

@@ -122,6 +124,8 @@ Trivy could detect vulnerabilities on the development packages, which not affect

 License detection is not supported for `Pipenv`.

+By default, Trivy doesn't report development dependencies. Use the `--include-dev-deps` flag to include them.


Fixed 55fb02f

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

DmitriyLewen · 2024-12-24T08:18:15Z

docs/docs/coverage/language/python.md

+| pip             | requirements.txt |            -            |     [Include](#pip)      |                  -                   |    ✓     |                    ✓                     |
+| Pipenv          | Pipfile.lock     |            ✓            |     [Include](#pipenv)      |                  -                   |    ✓     |                Not needed                |
+| Poetry          | poetry.lock      |            ✓            |     [Exclude](#poetry)      |                  ✓                   |    -     |                Not needed                |
+| uv              | uv.lock          |            ✓            |     [Include](#uv)      |                  ✓                   |    -     |                Not needed                |


you only need to add links if the --include-dev-deps flag is used for the file

In nodejs all package managers have links. I think the documentation should be consistent.

Then we should update the nodejs documentation

In nodejs all package managers have links. I think the documentation should be consistent.

I missed that a link was added for Bun. We can remove the link for that.
But for other files - yarn, npm and pnpm support --include-dev-deps. That's why they have links.

OK, I'll open a PR for that.

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

DmitriyLewen

LGTM

knqyf263 · 2024-12-24T11:52:05Z

docs/docs/coverage/language/python.md

@@ -27,7 +27,7 @@ The following table provides an outline of the features Trivy offers.
 | pip             | requirements.txt |            -            |     Include      |                  -                   |    ✓     |                    ✓                     |
 | Pipenv          | Pipfile.lock     |            ✓            |     Include      |                  -                   |    ✓     |                Not needed                |
 | Poetry          | poetry.lock      |            ✓            |     Exclude      |                  ✓                   |    -     |                Not needed                |
-| uv              | uv.lock          |            ✓            |     Exclude      |                  ✓                   |    -     |                Not needed                |
+| uv              | uv.lock          |            ✓            |     Include      |                  ✓                   |    -     |                Not needed                |


I think we should write the default behavior, like Node.js. Or am I missing something?

Suggested change

| uv | uv.lock | ✓ | Include | ✓ | - | Not needed |

| uv | uv.lock | ✓ | Exclude | ✓ | - | Not needed |

Fixed 5d65e2f

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

nikpivkin added 2 commits December 19, 2024 12:34

feat(python): add support for uv dev dependencies

6800ba5

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

feat(python): add support for uv optional dependencies

3ce2c3c

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

nikpivkin marked this pull request as ready for review December 19, 2024 08:16

nikpivkin requested review from knqyf263 and DmitriyLewen as code owners December 19, 2024 08:16

knqyf263 reviewed Dec 23, 2024

View reviewed changes

docs: add note about --include-dev-deps

03647db

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

DmitriyLewen reviewed Dec 24, 2024

View reviewed changes

nikpivkin added 2 commits December 24, 2024 12:08

Merge remote-tracking branch 'origin' into uv-dev

7384d21

fix python docs

55fb02f

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

DmitriyLewen mentioned this pull request Dec 24, 2024

feat(python): add support for poetry dev dependencies #8152

Merged

7 tasks

DmitriyLewen reviewed Dec 24, 2024

View reviewed changes

docs: link table and sections

22c4d9a

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

nikpivkin force-pushed the uv-dev branch from e296afa to 22c4d9a Compare December 24, 2024 08:34

DmitriyLewen approved these changes Dec 24, 2024

View reviewed changes

knqyf263 reviewed Dec 24, 2024

View reviewed changes

fix python docs

5d65e2f

Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io>

knqyf263 approved these changes Dec 24, 2024

View reviewed changes

knqyf263 enabled auto-merge December 24, 2024 12:08

knqyf263 added this pull request to the merge queue Dec 24, 2024

github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Dec 24, 2024

Merge remote-tracking branch 'origin' into uv-dev

cd4470d

nikpivkin added this pull request to the merge queue Dec 24, 2024

Merged via the queue into aquasecurity:main with commit 49c54b4 Dec 24, 2024
17 checks passed

nikpivkin deleted the uv-dev branch December 24, 2024 14:04

aqua-bot mentioned this pull request Dec 24, 2024

release: v0.59.0 [main] #8041

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python): add support for uv dev and optional dependencies #8134

feat(python): add support for uv dev and optional dependencies #8134

nikpivkin commented Dec 19, 2024 •

edited

Loading

knqyf263 Dec 19, 2024

DmitriyLewen Dec 24, 2024

knqyf263 Dec 23, 2024 •

edited

Loading

nikpivkin Dec 23, 2024

knqyf263 Dec 24, 2024

knqyf263 Dec 24, 2024

nikpivkin Dec 24, 2024

knqyf263 Dec 24, 2024

knqyf263 commented Dec 24, 2024

nikpivkin commented Dec 24, 2024

DmitriyLewen Dec 24, 2024

DmitriyLewen Dec 24, 2024

nikpivkin Dec 24, 2024

nikpivkin Dec 24, 2024 •

edited

Loading

DmitriyLewen Dec 24, 2024

DmitriyLewen Dec 24, 2024

nikpivkin Dec 24, 2024

DmitriyLewen Dec 24, 2024

nikpivkin Dec 24, 2024

nikpivkin Dec 24, 2024

nikpivkin Dec 24, 2024

DmitriyLewen Dec 24, 2024

nikpivkin Dec 24, 2024

DmitriyLewen left a comment

knqyf263 Dec 24, 2024

nikpivkin Dec 24, 2024

		@@ -122,6 +124,8 @@ Trivy could detect vulnerabilities on the development packages, which not affect

		License detection is not supported for `Pipenv`.

		By default, Trivy doesn't report development dependencies. Use the `--include-dev-deps` flag to include them.

	\| uv \| uv.lock \| ✓ \| Include \| ✓ \| - \| Not needed \|
	\| uv \| uv.lock \| ✓ \| Exclude \| ✓ \| - \| Not needed \|

feat(python): add support for uv dev and optional dependencies #8134

feat(python): add support for uv dev and optional dependencies #8134

Conversation

nikpivkin commented Dec 19, 2024 • edited Loading

Description

Without --include-dev-deps

With--include-dev-deps

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 commented Dec 24, 2024

nikpivkin commented Dec 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikpivkin Dec 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikpivkin commented Dec 19, 2024 •

edited

Loading

Without `--include-dev-deps`

With`--include-dev-deps`

knqyf263 Dec 23, 2024 •

edited

Loading

nikpivkin Dec 24, 2024 •

edited

Loading