Normalise API int64 type across Node SDK

[jwulf]

The JSON number value passed by the REST API for processInstanceKey (for example) is of type $int64.

This can't be reliably represented by the JavaScript number type, which is a 2^53 range floating point value.

This means that systems that are running for a long time can generate keys that cannot be represented by the SDK.

The gRPC library deals with this by parsing int64 to a string representation. Since we don't do arithmetic on keys, it might as well be a string.

The REST library that I am currently using, however, (got) represents these as number, leading to two issues:

1. The imprecision issue that will become an L1 in production when a system hits key values that cannot be represented as the JS number type.

2. The impedance mismatch that ZeebeGrpcClient returns long keys as type string, and the other APIs expect a number type as input; and vice versa.

• https://stackoverflow.com/questions/64402932/why-java-long-max-is-not-same-with-vue-axios-data-how-to-fix-that
• https://stackoverflow.com/questions/13502398/json-integers-limit-on-size
• https://jsoneditoronline.org/indepth/parse/why-does-json-parse-corrupt-large-numbers/

I'm looking for alternative REST clients that can serialise long integer types as JavaScript string.

[jwulf]

OK, got supports custom JSON parse/stringify functions:

https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#parsejson

This library could be used:

https://github.com/josdejong/lossless-json

[jwulf]

Hmmm, OK. I got it working. I can use a custom JSON parser with got to turn all numbers into LosslessNumbers.

ZeebeGrpc returns int64 types as type string, and leaves int32 numbers as number. This is consistent, and it is based on introspection of the protocol.

We can't detect the type of the field over REST. We know it is a number, but we don't know if it is an int32 or int64 field from the data itself.

I could encode this in the typing of the Dtos and use intellisense to determine whether the LosslessNumber should be marshalled to a string or to a number.

Going back the other way does not seem to be a problem.

Let me check if an API that takes an int64 number can accept a string. That could be a problem for the intellisense approach, because then we have two contracts to enforce - one with the API and the other with the application code.

[jwulf]

REST fields that take an int64 can take a string.

So, how can we emulate the behaviour of the ZeebeGrpcClient, and marshal int64 values to string and int32 values to number?

There must be an I/O boundary library that allows you to define the schema and that generates both the Dto typings and the runtime serialisers. I'll look at that tomorrow.

Maybe this? https://github.com/JohnWeisz/TypedJSON

Or this? https://github.com/colinhacks/zod

[jwulf]

OK, I have an approach for this.

A custom JSON parser that uses lossless-json to parse all JSON number types to a LosslessNumber.

The custom parser takes as an argument a Dto class. The class has decorated properties. The custom parser introspects the class and reads the type:Int32 and type:Int64 annotations, then uses those to convert the LosslessNumber to either a JS number or a JS string.

The class thus acts as both the metadata for the parser, via decorators on a Dto class, and also the type information for the consumer via the class interface.

[jwulf]

I've extended this to deal with nested Dtos.

It's in lib/LosslessJsonParser.ts

[jwulf]

This feature is now complete.

All known int64 type values are parsed to a JavaScript string type.

With unknown value types that come in via JSON number values, they are parsed to number unless they contain an unsafe int64 value, in which case the parser throws an exception.

A mapping Dto class can be provided that specifies mapping a JSON number field to either a string or bigint type. The parser will map the number value to this type, and the stringify function will do the reverse mapping.