-
Notifications
You must be signed in to change notification settings - Fork 11
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Requirements * Docker image * Tests config * CI/CD * Requirements * Tests config * Static code analysis * Update README * Update release notes * Requirements * Requirements * Old semgrep compatibility * Drop old Python * Drop old Python * No AST on Windows * Update README * Update release notes
- Loading branch information
Showing
59 changed files
with
1,062 additions
and
549 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,11 @@ | ||
| FROM python:3.9-slim | ||
| FROM python:3.11-slim | ||
|
|
||
| RUN apt update \ | ||
| && apt install -y make python3-lxml python3-yaml \ | ||
| && apt clean | ||
| WORKDIR /opt/whispers | ||
|
|
||
| WORKDIR /src | ||
| ADD . . | ||
|
|
||
| COPY dist/*.tar.gz . | ||
| RUN pip3 install -e . | ||
|
|
||
| RUN pip3 install *.tar.gz \ | ||
| && rm -rf *.tar.gz | ||
| RUN whispers --help | ||
|
|
||
| ENTRYPOINT [ "whispers" ] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,2 @@ | ||
| recursive-include whispers *.yml | ||
| include requirements.txt requirements-dev.txt |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,55 +1,81 @@ | ||
| # Whispers 2.2.0 release notes | ||
| # Whispers 2.3.0 release notes | ||
|
|
||
| * License change | ||
| * Compatibility improvements | ||
| * Standardize severity levels | ||
| * Minor code refactor | ||
| * Detection improvements | ||
| * Add XML cases | ||
| * Add `apikey-maybe` rule | ||
| * **New Feature:** 💫 <u>Static Code Analysis</u> 💫 is now supported! | ||
| * The present release complements classic Whispers' structured text analysis with [Semgrep](https://semgrep.dev)'s AST generator for [common programming languages](https://semgrep.dev/docs/supported-languages) like Python, PHP, Java/Scala/Kotlin, JavaScript/TypeScript, Go, etc etc. | ||
| * New argument `--ast` for enabling this feature via the CLI (it is disabled by default) | ||
| * New setting `ast: true` for enabling this feature via a custom config file (set to `ast: false` by default) | ||
| * Replaced [`astroid`](https://github.com/adeptex/whispers/blob/8f17f77e2199c55458ff125e3fb477a2a9349593/whispers/plugins/python.py) Python AST generator with [`semgrep`](https://github.com/adeptex/whispers/blob/master/whispers/plugins/semgrep.py) | ||
|
|
||
| * [Detection rule](https://github.com/adeptex/whispers/blob/master/whispers/rules) improvements | ||
| * Known API keys | ||
| * AWS account ID | ||
| * Passwords | ||
| * Creditcards | ||
|
|
||
| * Drop end-of-life Python support | ||
| * Versions 3.6 and 3.7 are no longer supported. Oldest supported version is Python 3.8. | ||
| * Last release that supports Python 3.6 and 3.7 is [Whispers 2.2.1](https://github.com/adeptex/whispers/releases/tag/2.2.1) | ||
|
|
||
| ## 💫 Licensing changes (again) 💫 | ||
| * Dependency tracking improvements | ||
| * New [`requirements-dev.txt`](https://github.com/adeptex/whispers/blob/master/requirements-dev.txt) file allows Dependabot updates for dev dependencies | ||
| * Modified [`setup.py`](https://github.com/adeptex/whispers/blob/master/setup.py) to read from `requirements.txt` and `requirements-dev.txt` | ||
| * Updated build CI to use Python 3.12.3 | ||
|
|
||
| Version 2.1 was released under [GNU General Public License v3.0](https://github.com/adeptex/whispers/blob/3f5282ea3855d658ea37ec96dfc693598c16d7a7/LICENSE), which is `intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users.` | ||
| * Debugging and troubleshooting | ||
| * Modified [`config.yml`](https://github.com/adeptex/whispers/blob/master/whispers/config.yml) to exclude known false positives | ||
| * Fixed [`Dockerfile`](https://github.com/adeptex/whispers/blob/master/Dockerfile) to work with `docker build -t whispers .` or the same `make image` | ||
| * New arg `--dump` for generating an AST of a file: `whispers --dump src/example.ts` | ||
|
|
||
| Version 2.2 is released under [BSD 3-Clause License](https://github.com/adeptex/whispers/blob/master/LICENSE), which is a permissive license that `prohibits others from using the name of the copyright holder or its contributors to promote derived products without written consent.` | ||
|
|
||
| This change removes source code disclosure requirement 🕵️ | ||
| ## 💫 New Feature: Static Code Analysis 💫 | ||
|
|
||
| With the release of Whispers 2.3, it is now possible to accurately apply Whispers' secret detection techniques for structured text to static code. Before this release, Whispers only supported structured text formats, such as JSON or XML. [Semgrep](https://semgrep.dev) is an open source SAST tool, which has a built-in feature for generating Abstract Structure Trees (ASTs) for [many common programming languages](https://semgrep.dev/docs/supported-languages). Generating an AST for static code yields an accurate structured text representation, which can be checked for secrets with Whispers' rules and plugins. As such, generating ASTs requires an additional "format conversion" step, which naturally affects runtime speed. When AST is enabled it will take longer to scan the same scope if any source code files are present. The increased amount of runtime time would be however long it takes to run the following command on all static code files in scope: | ||
|
|
||
| ## ❌ Breaking changes ❌ | ||
| ```sh | ||
| semgrep scan --metrics=off --quiet --dump-ast --json --lang $LANG $SRCFILE | ||
| ``` | ||
|
|
||
| Consider the following benchmarks: | ||
|
|
||
| ```sh | ||
| time whispers -F " " tests/fixtures | ||
| # 313 detected secrets | ||
| # 0,51s user 0,03s system 99% cpu 0,540 total | ||
| # 0,60s user 0,04s system 99% cpu 0,642 total | ||
|
|
||
| ### ❌ Severity levels ❌ | ||
| time whispers -a -F " " tests/fixtures | ||
| # 421 detected secrets | ||
| # 2,20s user 0,40s system 100% cpu 2,589 total | ||
| # 2,32s user 0,46s system 100% cpu 2,772 total | ||
| ``` | ||
|
|
||
| Severity level names have been adapted to a more common format. For example, `BLOCKER` is replaced by `Critical` and so on. The full list is as follows: | ||
| AST conversion is **disabled by default** - `semgrep` will **not** execute at all unless explicitly enabled. Custom config files that are missing `ast: false` or `ast: true` will default to `false`. | ||
|
|
||
| | Version 2.1 (before) | Version 2.2 (now) | | ||
| |---|---| | ||
| | `BLOCKER` | `Critical` | | ||
| | `CRITICAL` | `High` | | ||
| | `MAJOR` | `Medium` | | ||
| | `MINOR` | `Low` | | ||
| | `INFO` | `Info` | | ||
| ```yaml | ||
| ast: true # enable AST in config.yml | ||
| ``` | ||
| ⚠️ **Please update your custom rules and CLI args to reflect these changes** ⚠️ | ||
| ```sh | ||
| whispers --ast target/dir/or/file # enable AST in CLI | ||
| ``` | ||
|
|
||
| Instead of | ||
|
|
||
| > `whispers -s BLOCKER dir/or/file` | ||
| ## ❌ Breaking changes ❌ | ||
|
|
||
| ### ❌ Replaced `astroid` with `semgrep` ❌ | ||
|
|
||
| use | ||
| Before Whispers 2.3, only Python AST scanning was natively supported by `astroid`, and integrated via [`plugins/python.py`](https://github.com/adeptex/whispers/blob/8f17f77e2199c55458ff125e3fb477a2a9349593/whispers/plugins/python.py). With the release of Whispers 2.3, this functionality is superseded by `semgrep`, and integrated via [`plugins/semgrep.py`](https://github.com/adeptex/whispers/blob/master/whispers/plugins/semgrep.py). As a base line, the new `semgrep` plugin detects the same findings as the `astroid` plugin, but supports more programming languages. | ||
|
|
||
| > `whispers -s Critical dir/or/file` | ||
| Unfortunately `semgrep` has telemetry enabled by default, but can be turned off via [`--metrics=off`](https://github.com/adeptex/whispers/blob/master/whispers/plugins/semgrep.py#L57). In any case, `semgrep` will not execute unless explicitly enabled via args or config. | ||
|
|
||
| See [README](https://github.com/adeptex/whispers#readme) for details and examples. | ||
| ⚠️ **NOTE:** At the time of writing, `semgrep` [does not support Windows OS natively](https://github.com/semgrep/semgrep/issues/1330), and can only be installed through WSL. As such, compiled Whispers PE32+ executable comes without Static Code Analysis support. Installing Whispers on Windows via WSL with `pip3 install whispers` *does* have Static Code Analysis support. | ||
|
|
||
|
|
||
| # Changelog | ||
|
|
||
| |Version|Release notes| | ||
| |---|---| | ||
| |2.0.0|https://github.com/adeptex/whispers/releases/tag/2.0.0| | ||
| |2.1.0|https://github.com/adeptex/whispers/releases/tag/2.1.0| | ||
| |2.2.0|https://github.com/adeptex/whispers/releases/tag/2.2.0| | ||
| |Date|Version|Release notes| | ||
| |---|---|---| | ||
| |2021-12-07|2.0.0|https://github.com/adeptex/whispers/releases/tag/2.0.0| | ||
| |2022-07-29|2.1.0|https://github.com/adeptex/whispers/releases/tag/2.1.0| | ||
| |2023-10-23|2.2.0|https://github.com/adeptex/whispers/releases/tag/2.2.0| | ||
| |2024-06-15|2.3.0|https://github.com/adeptex/whispers/releases/tag/2.3.0| |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| autoflake~=1.4 | ||
| autopep8~=1.7 | ||
| black==22.8; python_version <= '3.6' | ||
| black~=24.3; python_version > '3.6' | ||
| build~=0.8 | ||
| coverage~=4.5 | ||
| coverage-badge~=1.0 | ||
| flake8~=5.0 | ||
| isort~=5.9 | ||
| pytest~=7.0 | ||
| pytest-mock~=3.6 | ||
| pip-tools~=6.2 | ||
| wheel~=0.37 | ||
| twine~=3.4 |
Oops, something went wrong.