Fix ggshield secret scan trying to scan too large documents #631
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`Scannable.is_longer_than()` currently compares the size passed to it to the decoded size, but what we actually want to compare it to is the *UTF-8 encoded size*. To implement this without reading the content multiple times, `Scannable` implementations now store the UTF-8 encoded size, and `Scannable` helper methods (`_is_file_longer_than()` and `_decode_bytes()`) return the UTF-8 encoded size in addition to their current return value. When the content they work on is already UTF-8, the helper methods try to avoid reading it until necessary.
agateau-gg
force-pushed
the
agateau/fix-maximum-size-check
branch
from
d822636
to
db208b9
Compare
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## main #631 +/- ##
==========================================
+ Coverage 93.98% 94.03% +0.05%
==========================================
Files 103 103
Lines 4804 4812 +8
==========================================
+ Hits 4515 4525 +10
+ Misses 289 287 -2
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Scannable.is_longer_than()currently compares the size passed to it to the decoded size, but what we actually want to compare it to is the UTF-8 encoded size.To implement this without reading the content multiple times,
Scannableimplementations now store the UTF-8 encoded size, andScannablehelper methods (_is_file_longer_than()and_decode_bytes()) return the UTF-8 encoded size in addition to their current return value.When the content they work on is already UTF-8, the helpers try to avoid reading it until necessary.
This PR also integrates the latest py-gitguardian changes, which are also related to this issue.
Issue: #561