Add ZSTD to codecs and types

[paambaati]

Support for the ZStandard compression codec is landing in Kafka 2.1.0, due for release 29th October, 2018.

Following kafkajs-lz4, I also plan to write a similar kafkajs-zstd codec soon.

References -

1. https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression
2. https://issues.apache.org/jira/browse/KAFKA-4514
3. KAFKA-4514: Add Codec for ZStandard Compression apache/kafka#2267
4. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=91554044

[tulios]

Nice, but looking at your references this will require some more work. I will create a github project with the necessary issues, but just to highlight some things:

New error code

It also introduces a new error code, UNSUPPORTED_COMPRESSION_TYPE (74)

Bump the produce API up to the required version

Zstd will only be allowed for the bumped produce API. That is, for older version clients(=below KAFKA_2_1_IV0), we return UNSUPPORTED_COMPRESSION_TYPE regardless of the message format

Bump the fetch API up to the required version

Zstd will only be allowed for the bumped fetch API. That is, if the requested partition uses zstd and the client version is below KAFKA_2_1_IV0, we return UNSUPPORTED_COMPRESSION_TYPE regardless of the message format. Down-conversion of zstd-compressed records will not be supported.

New experimental flag so we can test until the APIs are stable

[tulios]

Here is the project:
https://github.com/tulios/kafkajs/projects/2

[tulios]

@paambaati PR #873 adds the last bit for ZSTD to work, it has been a while, but it's finally here 😄