[Event Hubs] Review the use of default DataTransformer

[ramya-rao-a]

From the notes in #1481

We need to review how the default DataTransformer behaves and whether received message body is in expected format, when for the various cases where the message body is

• a primitive type: string, number, boolean, null, undefined (this works)
• a Buffer or AMQP typed objects (sent from other language libraries)
  • we receive the buffer, toString() it and then JSON.parse() it, so what was sent is definitely not what is received

For more notes, see #1481

[mikeharder]

An interesting case I found is Buffer vs Uint8Array. Buffer is sent as a byte array, while Uint8Array is sent as a (much larger) JSON string. I think it would be best if the API handled these the same way.

// Sent as byte[]
await sender.send({ body: Buffer.alloc(1024) });

// Sent as JSON string
await sender.send({ body: new Uint8Array(1024) });

[ramya-rao-a]

Including this work to be done as part of the Event Hubs work we are planning for this month.
The changes would be done in amqp-common which is shared between Service Bus and Event Hubs

[ramya-rao-a]

Review conclusions:

• All primitive types (string, number, boolean, null, undefined), json objects, arrays and buffers get encoded and decoded properly. This is thanks to JSON.stringify() and JSON.parse()
• Any AMQP typed objects sent by sdks in other languages are decoded properly only if the stringified buffer can be JSON parsed. TODO: Test all AMQP types
• For message bodies that fail the JSON.stringify() function during encoding, the recommendation to users is to use Buffers. TODO: Update the error message with this recommendation
• For message bodies that fail the JSON.parse() function during decoding, the body is returned as is. This is our catch all.

Regarding @mikeharder's point on Uint8Array:
@bterlson Are there any other buffer representations we should cater to?

[ramya-rao-a]

From #7253, we have the problem of message body getting wrapped in quotes when the body is of type string which needs to be fixed as well

[richardpark-msft]

We also need to consider for Service Bus:

• We're doing track 2 at the moment, so we have the option of picking a different default behavior (looks like we'd have to change the default behavior since we decided to do string -> JSON here)
  • Another option is to create a new encoder/decoder. For EH it won't be the default but we could consider doing that for SB.
• @ramya-rao-a - do you know what the end result was for @mikeharder 's question above? Can users bypass the encoder by just passing some form of array? There is always that workaround but from customer issues it appears that's not intuitive (at least for strings).
• Need to make sure our cross-language story makes sense. If we go with "strings -> bytes" it's probably fine. If we go with anything more complicated interop is going to be annoying.

[chradek]

We received a request to support disabling the automatic JSON parsing on event bodies we do when receiving events. While we won't attempt to JSON parse an event that was sent as a byte array, it's not always possible for the same team to control how events are sent to Event Hubs. In high throughput cases, this can cause an unnecessary increase in CPU usage.

At the very least, we should allow disabling the JSON parsing - potentially with a flag on EventHubConsumerClientOptions or SubscriptionOptions. Would love to consider allowing users to specify their serialization/deserialization strategy as well.

[github-actions]

Hi @ramya-rao-a, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.