Set a configurable maximum payload size for the message decoder and add an invalid message handler to catch invalid messages

[vietj]

Motivation:

The gRPC message decoder uses the default limit allowed by the gRPC HTTP/2 transport (2^32 bytes). The default maximum size should be smaller and configurable. This can be used to consume memory of a server or a client and is referenced as CVE-2024-8391

Changes:

Add options for configuring the maximum message size and use a lower default value (256KB) for both client and server. In addition an invalid message handler can be set on the GrpcReadStream to catch invalid message reports and let the application recover invalid messages. The invalid message handler can be triggered by a capacity overflow or a decoder exception.

Results:

gRPC server and client now uses a smaller default maximum message size which can be configured according to the application needs. Invalid message handler can also be set to catch invalid messages.

[sarah-jay-peper]

Hi!

Thank you for creating this fix.

We have updated our services to 4.5.10 quite quickly and got a tiny bit burned by the surprisingly low default limit, tho. Could the default limit be raised to the 4MB limit of gRPC itself?