Add a PyPa importer, organize the code using a shared osv.py #780
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ziad <ziadhany2016@gmail.com>
Signed-off-by: Ziad <ziadhany2016@gmail.com>
Signed-off-by: Ziad <ziadhany2016@gmail.com>
TG1999
reviewed
Jun 27, 2022
| summary = build_description(summary=summary, description=details) | ||
| aliases = raw_data.get("aliases") or [] | ||
| if raw_id: | ||
| aliases.append(raw_id) |
There was a problem hiding this comment.
dedupe the aliases, raw_id might be already in aliases
There was a problem hiding this comment.
No, raw_id is a unique identifier for the vulnerability entry and The aliases field gives a list of IDs of the same vulnerability in other databases. so I think this can't be duplicated. this code from pysec importer #632 I just move it to osv.py and I try to make it useable so I can use osv.py in oss-fuzz and pypa importer
https://ossf.github.io/osv-schema/#aliases-field
https://ossf.github.io/osv-schema/#id-modified-fields
TG1999
requested changes
Jun 28, 2022
vulnerabilities/importers/osv.py
Outdated
|
|
||
| affected_packages = [] | ||
| if "affected" not in raw_data: | ||
| logger.error(f"affected_packages not found - {raw_id !r}") |
There was a problem hiding this comment.
Remove this logger, if "affected" is not in data we can't do anything
Signed-off-by: Ziad <ziadhany2016@gmail.com>
Signed-off-by: Ziad <ziadhany2016@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
|
@ziadhany this looks good to me, have you verified that these importers work and have you ran the default improver over the imported data? |
Add URLs to CPEs
pysec and pypa importer run as expected but some package have a invalid version range . and some weird like this should I remove the quotas or make a issue in |
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
And address review comments Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Add fixed packages in packages endpoint
Bumps [lxml](https://github.com/lxml/lxml) from 4.8.0 to 4.9.1. - [Release notes](https://github.com/lxml/lxml/releases) - [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt) - [Commits](lxml/lxml@lxml-4.8.0...lxml-4.9.1) --- updated-dependencies: - dependency-name: lxml dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Bump lxml from 4.8.0 to 4.9.1
Bumps [django](https://github.com/django/django) from 4.0.4 to 4.0.6. - [Release notes](https://github.com/django/django/releases) - [Commits](django/django@4.0.4...4.0.6) --- updated-dependencies: - dependency-name: django dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
…4.0.6 Bump django from 4.0.4 to 4.0.6
Run import then improve at each cycle. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
ReST is picky on spacing and indentations. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@ziadhany try to strip the comas when univers fails to parse this. Just ignore this, we can't parse this. This is invalid other than this I was not able to reproduce this issue, can you add some logs and tell us how the data looks like there |
|
@ziadhany any updates on this? |
Pypa importer logs Pysec importer logs : and ( remove quotas not make any thing better ) for invalid Versions : and I think the DefaultImprover for osv work without any problem. |
…tion Prepare Release 30.0.0rc2
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Also handle renaming of vulnd_id to vulnerability_id Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Use uuid instead of base36 Reference: aboutcode-org#811 Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This helps keep templates simple. Also move position of models Meta and __str__ to a conventional location right after the field definitions. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* Use search as form field name consistently aboutcode-org#907 This replaces the vulnerability_id and package_name and esnure we have a better looking URL with a "search" query string. * Also make "search" a required field and use standard HTML for validation. Remove JS validations. * Use vulnerability_id rather than PK for URL in templates * Remove redundant, repeat display of search queries. * Rename "Total records" to "results" in search results pages Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* Use purl slug and URL route for packages * Use in forms, urls and templates, including a get_absolute_url() method. * Rename VulnerabiltyForm forms to VulnerabiltySearchForm * Rename PackageForm forms to PackageSearchForm * Use new pagination template includes in search results templates. The pagination is the same repeated at the top and botton of the search results * Display on 20 search results per page. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This limit the flickering to the vulnerability details. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This new improver adds missing CVE reference id that were not created by their importer. Note that the current approach for improvers seems to be restricted to advisories and hsould be relaxed to better handle the general cases. Here this improver does not deal with advisories at all. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Use the same layout and remove generated comments Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Also validate full_clean in the improve_runner to ensure we do not have empty, invalid or blank URLs. Refactor code to add new Manager to VulnerabilityReference and Package Add convenience method accordingly to create Pckage from purls Reference: aboutcode-org#818 Co-authored-by: Tushar Goel <tushar.goel.dav@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Prepare release 30
There was a mistake commit in the history where the unique_together of VulnerabilityReference was reordered without creating a new migration This is fixing this to ensure we can migrate correctly Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Prepare v30.rc6
* Use named arguments when calling functions. * Use variables for tests expected and results * Split tests functions in smaller functions Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
|
All merged... Feel free to check the few updates I made. Thanks you ++ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Ziad ziadhany2016@gmail.com