chore(iast): redaction algorithms refactor II

[avara1986]

Summarize

Refactor of the IAST redaction system. The old algorithms had several problems:

Description

This PR continues this #9126

• Migrate SQL Injection to this new algorithm
• Remove deprecated code

Checklist

• Change(s) are motivated and described in the PR description
• Testing strategy is described if automated tests are not included in the PR
• Risks are described (performance impact, potential for breakage, maintainability)
• Change is maintainable (easy to change, telemetry, documentation)
• Library release note guidelines are followed or label changelog/no-changelog is set
• Documentation is included (in-code, generated user docs, public corp docs)
• Backport labels are set (if applicable)
• If this PR changes the public interface, I've notified @DataDog/apm-tees.
• If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.

Reviewer Checklist

• Title is accurate
• All changes are related to the pull request's stated goal
• Description motivates each change
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Change is maintainable (easy to change, telemetry, documentation)
• Release note makes sense to a user of the library
• Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: avara1986/APPSEC-52733-iast_redaction_refactor
Commit report: 2ccb3de
Test service: dd-trace-py

✅ 0 Failed, 110819 Passed, 5340 Skipped, 36m 18.96s Total duration (1h 8m 42.24s time saved)

[brettlangdon]

I reviewed everything but the tests so far.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0.67340% with 295 lines in your changes are missing coverage. Please review.

Project coverage is 6.78%. Comparing base (2230214) to head (02b9f23).

Files	Patch %	Lines
...ec/iast/taint_sinks/test_sql_injection_redacted.py	0.00%	95 Missing ⚠️
...iast/_evidence_redaction/sql_sensitive_analyzer.py	0.00%	51 Missing ⚠️
tests/contrib/django/test_django_appsec_iast.py	0.00%	47 Missing ⚠️
...c/iast/taint_sinks/test_path_traversal_redacted.py	0.00%	21 Missing ⚠️
ddtrace/appsec/_iast/taint_sinks/_base.py	0.00%	14 Missing ⚠️
...iast/taint_sinks/test_header_injection_redacted.py	0.00%	14 Missing ⚠️
...ests/appsec/iast/taint_sinks/test_sql_injection.py	0.00%	14 Missing ⚠️
tests/appsec/iast/test_taint_utils.py	0.00%	8 Missing ⚠️
ddtrace/appsec/_iast/_utils.py	25.00%	6 Missing ⚠️
ddtrace/appsec/_iast/constants.py	0.00%	5 Missing ⚠️
... and 7 more

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #9163       +/-   ##
===========================================
- Coverage   78.53%    6.78%   -71.75%     
===========================================
  Files        1277     1247       -30     
  Lines      120386   118456     -1930     
===========================================
- Hits        94547     8040    -86507     
- Misses      25839   110416    +84577     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-05-08 07:28:19

Comparing candidate commit 02b9f23 in PR branch avara1986/APPSEC-52733-iast_redaction_refactor with baseline commit 2230214 in branch main.

Found 14 performance improvements and 4 performance regressions! Performance is the same for 149 metrics, 9 unstable metrics.

scenario:coreapiscenario-context_with_data_listeners_and_all_listeners

• 🟩 max_rss_usage [-711.399KB; -653.798KB] or [-3.309%; -3.041%]

scenario:coreapiscenario-context_with_data_no_listeners

• 🟩 max_rss_usage [-573.363KB; -522.726KB] or [-2.672%; -2.436%]

scenario:coreapiscenario-core_dispatch_listeners

• 🟩 max_rss_usage [-582.131KB; -532.801KB] or [-2.712%; -2.482%]

scenario:coreapiscenario-core_dispatch_no_listeners

• 🟩 max_rss_usage [-1.090MB; -1.036MB] or [-5.073%; -4.821%]

scenario:coreapiscenario-core_dispatch_only_all_listeners

• 🟩 max_rss_usage [-569.479KB; -515.142KB] or [-2.655%; -2.402%]

scenario:coreapiscenario-core_dispatch_with_results_listeners_and_all_listeners

• 🟥 max_rss_usage [+700.530KB; +750.683KB] or [+3.369%; +3.610%]

scenario:coreapiscenario-core_dispatch_with_results_only_all_listeners

• 🟩 max_rss_usage [-714.920KB; -510.603KB] or [-3.343%; -2.388%]

scenario:coreapiscenario-get_item_exists

• 🟩 max_rss_usage [-570.895KB; -516.184KB] or [-2.662%; -2.407%]

scenario:coreapiscenario-get_item_missing

• 🟩 max_rss_usage [-561.146KB; -512.825KB] or [-2.618%; -2.392%]

scenario:coreapiscenario-set_item

• 🟩 max_rss_usage [-570.474KB; -436.732KB] or [-2.665%; -2.040%]

scenario:httppropagationextract-empty_headers

• 🟥 max_rss_usage [+670.463KB; +763.546KB] or [+3.235%; +3.684%]

scenario:httppropagationinject-with_tags_invalid

• 🟩 max_rss_usage [-706.929KB; -670.966KB] or [-3.304%; -3.136%]

scenario:span-start-finish

• 🟩 max_rss_usage [-711.343KB; -612.894KB] or [-3.310%; -2.852%]

scenario:span-start-finish-telemetry

• 🟩 max_rss_usage [-878.278KB; -586.861KB] or [-4.064%; -2.716%]

scenario:span-start-finish-traceid128

• 🟩 max_rss_usage [-983.389KB; -719.727KB] or [-4.576%; -3.349%]

scenario:tracer-large

• 🟥 max_rss_usage [+642.426KB; +724.000KB] or [+2.923%; +3.294%]

scenario:tracer-medium

• 🟩 max_rss_usage [-618.485KB; -536.997KB] or [-2.869%; -2.491%]

scenario:tracer-small

• 🟥 max_rss_usage [+614.497KB; +689.669KB] or [+2.932%; +3.291%]