Scan only PR commits for Gitleaks instead of whole codebase

[DariuszPorowski]

Fixes #2487

Proposed Changes

1. --redact flag for gitleaks should be default to prevent exposing detected secrets to logs
2. common functions consolidated into utils, based on review comment: Scan only PR commits for Gitleaks instead of whole codebase #2504 (comment)
3. CI env use case, based on discussion: AzureCommentReporter vars reflect official Azure DevOps naming #2510 (comment)
4. Missed doc fixes in [documentation] runner - doc fix for env list of values #2449 (fixes Not possible to pass array of values for envs with local runner (cli) #2448)
5. updated url to new gitleaks home: https://github.com/zricethezav/gitleaks -> https://github.com/gitleaks/gitleaks

Readiness Checklist

Author/Contributor

• Add entry to the CHANGELOG listing the change and linking to the corresponding issue (if appropriate)
• If documentation is needed for this change, has that been included in this pull request

Reviewing Maintainer

• Label as breaking if this is a large fundamental change
• Label as either automation, bug, documentation, enhancement, infrastructure, or performance

[echoix]

For the pyproject.toml, you can keep it for another PR, it's already in the plans to switch to it, but neither of us had time to learn about it yet! If you are used to it, we'll gladly accept a first implementation 😄
When building, there is a warning saying pip version 23 will enforce it, ie something like

DEPRECATION: xxxxxxx is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at pypa/pip#8559

So we'll need to go to it a way or another.

[DariuszPorowski]

@echoix that's the reason I put the note in the 1st line NOTE: pyproject.toml and poetry is just for my dev env, will remove on ready PR - so do not worry about these files for now :)
I can prapare PR with pyproject.toml and poetry in the future, poetry with venv changed my life ;)

[Kurt-von-Laven]

Yeah, Poetry is great!

[DariuszPorowski]

@nvuillam +/- this PR is ready, minor cleanup, and changes left, but have Q around REPOSITORY_GITLEAKS_PR_COMMITS_SCAN. Currently, it is false but happy to put true as default because then validate all code base = false will does the job without any extra envs. But this is a kind of braking change and may impact current users' setup.
So, I'm happy to hear feedback and design decisions.

[nvuillam]

@DariuszPorowski it looks great, i'll check that tomorrow, thanks for the PR:)

[nvuillam]

Cc @Kurt-von-Laven @bdovaz @echoix

[DariuszPorowski]

@nvuillam no rush, take your time. Tomorrow I am off so will continue on Thursday.

[DariuszPorowski]

@nvuillam just following up :)

[nvuillam]

@DariuszPorowski you have many failing test cases ^^

[DariuszPorowski]

@nvuillam +/- this PR is ready, minor cleanup, and changes left, but have Q around REPOSITORY_GITLEAKS_PR_COMMITS_SCAN. Currently, it is false but happy to put true as default because then validate all code base = false will does the job without any extra envs. But this is a kind of braking change and may impact current users' setup.
So, I'm happy to hear feedback and design decisions.

@nvuillam actually follow up was on this question not on the code hehe ;)

[nvuillam]

@DariuszPorowski if it breaks existing config of ML using gitleaks and VALIDATE_ALL_CODEBASE=false , i'd prefer to stay deactivated by default, to avoid a breaking change like u said ^^

[DariuszPorowski]

@nvuillam I'm sorry it took so long but the last month has been hectic at work. PR ready for review :)

[nvuillam]

@DariuszPorowski same here, I understand 🤣

Your PR looks great but I have some doubts about the documentation... it seems you directly updated generated markdown file https://github.com/oxsecurity/megalinter/blob/cff15a01ef2d29d904b97b268df8b54225429b4a/docs/descriptors/repository_gitleaks.md , so the next time the doc will be rebuilt, it will be overwritten

If you want "free text" doc to be added, you need to update linter_description property in the YML descriptor file :)

You can verify that by yourself using bash build.sh --doc locally

[nvuillam]

@DariuszPorowski Same remark about custom variables: if you add them in the descriptor, they will be automatically added to the doc & json schema ^^

[DariuszPorowski]

@nvuillam Oh gosh, the last changes to te main were painful to integrate with this PR :D hope now is good, docs generated automatically

[nvuillam]

@DariuszPorowski i can assure you they were even harder to implement 😅

But it's for the better good: with next version you'll just have to trust MegaLinter core code with your env variables (including secrets defined in CI/CD), no need anymore to trust the dozens of embedded linters 😎

[nvuillam]

@DariuszPorowski plz can u merge main in your branch ? :)

[DariuszPorowski]

Sure, will do over the weekend

[nvuillam]

I did it :)

[nvuillam]

@DariuszPorowski thanks for this great PR :)