Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor severity score model and fix incorrect suse scores #1636

Merged
merged 14 commits into from
Nov 13, 2024
Merged

Refactor severity score model and fix incorrect suse scores #1636

merged 14 commits into from
Nov 13, 2024

Conversation

keshav-space
Copy link
Member

@keshav-space keshav-space commented Nov 6, 2024 •
edited
Loading

  • Add a severities field to both AffectedByPackageRelatedVulnerability and Vulnerability.
  • Add url field to VulnerabilitySeverity and remove reference relationship.
  • Create migrations to remove corrupted suse scores and enable reprocessing of old suse advisories on the next import cycle.
  • Populate the new url field in VulnerabilitySeverity using references, set the severity-vulnerablity relationship from the old model.
  • Update UI, API, APIv2, importer/pipeline, and export command to use the new model.
  • Use new severity model for risk calculation
  • Use optimized queryset in risk pipeline

Resolves #1597

@keshav-space keshav-space changed the title Refactor severity score model and fix incorrect suses scores Refactor severity score model and fix incorrect suse scores Nov 6, 2024
@keshav-space keshav-space self-assigned this Nov 7, 2024
@keshav-space keshav-space force-pushed the 1597-refactor-vulnerability-severity branch from 193628e to 84a3a93 Compare November 11, 2024 14:11
@keshav-space
Refactor vulnerability severity and data migrations
db82a62 
- Add 'severities' field to AffectedByPackageRelatedVulnerability.
- Add 'severities' field to Vulnerability.
- Add 'url' field to VulnerabilitySeverity.
- Data migration to remove corrupted SUSE scores.
- Data migration to populate new VulnerabilitySeverity url field using reference.
- Data migration to populate Vulnerability 'severities' M2M relationship.
- Delete VulnerabilitySeverity reference field.

Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use the new severities in vulnerability detail view
05d84c3 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use severity.url insted of reference.url in template
46f4b5a 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use the new severities in API
c4636f4 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Update the importer to populate severities field
86947a5 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Export new severities relations
0ad0a19 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Preserve old API response for reference and severity
00354da 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Reprocess old suse advisory in next import cycle
287c834 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use new severity model for risk calculation
16cb2f0 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use optimized queryset in risk pipeline
b2d1e20 
- Prefetch related vulnerability, severities, references, and exploits
  for better performance

Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Increase batch size in risk pipeline
b5e2883 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use new severity model in APIv2
4107451 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space keshav-space force-pushed the 1597-refactor-vulnerability-severity branch from 84a3a93 to 4107451 Compare November 11, 2024 16:02
@keshav-space
Avoid fresh database query on prefetched cache
9d0791f 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
TG1999
TG1999 reviewed Nov 12, 2024
View reviewed changes
vulnerabilities/api_v2.py
model = VulnerabilitySeverity
fields = ["url", "value", "scoring_system", "scoring_elements", "published_at"]

def to_representation(self, instance):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need to_representation method here? if published_at is None, let it be sent as None

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this just to preserve the response structure we're already using in APIv2 https://github.com/aboutcode-org/vulnerablecode/blob/8a68c97dfa369ad048de3ece14cc1b3cf40591cc/vulnerabilities/api_v2.py#L16C33-L16C64.

IMO yes we should simpy send None when we don't have published date.

TG1999
TG1999 approved these changes Nov 13, 2024
View reviewed changes
Copy link
Contributor

@TG1999 TG1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

TG1999
TG1999 reviewed Nov 13, 2024
View reviewed changes
vulnerabilities/api.py
@@ -214,10 +220,21 @@ def to_representation(self, instance):

return data

def get_references(self, vulnerability):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit: Why we are having a method here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In our old API, references used to contain the nested severity, as severity and reference were related through a foreignkey relationship. In our new model we have dissociated reference and severity. To ensure we maintain compatibility for existing users of old API we're manually crafting the references to include the relevant severity.

@TG1999 TG1999 requested a review from tdruez November 13, 2024 14:30
@TG1999 TG1999 mentioned this pull request Nov 13, 2024
Merged
@keshav-space
Avoid filtering on prefetched query
a008c37 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space keshav-space merged commit 8bca5cc into main Nov 13, 2024
9 checks passed
@keshav-space keshav-space deleted the 1597-refactor-vulnerability-severity branch November 13, 2024 15:33
@keshav-space keshav-space mentioned this pull request Nov 13, 2024
Closed
@pombredanne pombredanne added this to the v35.0.0 - 2-next milestone Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TG1999 TG1999 TG1999 approved these changes

@pombredanne pombredanne Awaiting requested review from pombredanne

@tdruez tdruez Awaiting requested review from tdruez

Assignees

@keshav-space keshav-space

Labels
2-next
Projects
04-VulnerableCode Data Quality and next
Status: Validated
Milestone
v35.0.0 - 2-next
Development

Successfully merging this pull request may close these issues.

Incorrect severity score due to identical Reference URLs
3 participants
@keshav-space @TG1999 @pombredanne