feat: utf8 sanitizer

[fracasula]

Description

This is needed by the IngestionSvc so that we don't propagate messages with invalid UTF-8 sequences to Pulsar.

If we do propagate them, then the SrcRouter won't be able to push those messages to RudderServer, because then the RudderServer gateway would fail with an error similar to the following:

pq: invalid byte sequence for encoding "UTF8": 0xd4 0x6d

Linear Ticket

< Linear_Link >

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 30.60%. Comparing base (bc95a3f) to head (f91980c).
Report is 10 commits behind head on main.

❗ Current head f91980c differs from pull request most recent head d0689e5

Please upload reports for the commit d0689e5 to get more accurate results.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #485       +/-   ##
===========================================
- Coverage   78.98%   30.60%   -48.38%     
===========================================
  Files          90       44       -46     
  Lines        6747     3486     -3261     
===========================================
- Hits         5329     1067     -4262     
- Misses       1153     2303     +1150     
+ Partials      265      116      -149     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lvrach]

I believe that both functions can be replaced with standard library ones:

https://pkg.go.dev/bytes#ToValidUTF8

and

utf8.Valid

[fracasula]

I believe that both functions can be replaced with standard library ones:

https://pkg.go.dev/bytes#ToValidUTF8

and

utf8.Valid

@lvrach I'm happy to replace Invalid with utf8.Valid but I think we should consider keeping our own sanitizer because it does zero allocations while bytes.ToValidUTF8 uses append.

Given the effort spent on the IngestionSvc with all the memory optimizations, I think it makes sense to consider it. Wdyt?

[fracasula]

@lvrach I copied the exact same test cases that they have for bytes.ToValidUTF8 and I've added them to our test scenarios.