How to support snappy compression after aiokafka 0.9.0?

[3figs]

Hi,
Our server app is using snappy compression to send message over kafka. This has no problem with aiokafka 0.8.1. Recently we upgraded aiokafka version to 0.10.0, it suddenly gave us error "UnsupportedCodecError('Libraries for snappy compression codec not found')".

We checked the release notes and found that snappy was removed since 0.9.0. But we have to support snappy compression because that's the default compression type used by the server and the server is not controlled by our group.

What should I do in order to support snappy compression now?

Thanks
Steven

[ods]

We checked the release notes and found that snappy was removed since 0.9.0.

No, snappy compression is still supported. But now it uses other dependency library for its implementation. Nothing is needed to change if you install aiokafka with proper extras (pip install 'aiokafka[snappy]' or pip install 'aiokafka[all]').

[3figs]

Thanks for the reply, but I hit different issues when trying to do "pip install 'aiokafka[snappy]'":

First hit cramjam issue, asking for Cargo and Rust. After adding Cargo and Rust, it hit pysam isseu.
Did you also hit such issue when doing "pip install 'aiokafka[snappy]'"?

#15 [11/16] RUN pip3 install --no-cache-dir 'aiokafka[snappy]'
#15 sha256:547b55ce5a15d4b43971df5e7851968ea726cec4ec1b93eacc812d97c47bfb63
#15 0.733 Requirement already satisfied: aiokafka[snappy] in /usr/lib/python3.11/site-packages (0.10.0)
#15 0.764 Requirement already satisfied: async-timeout in /usr/lib/python3.11/site-packages (from aiokafka[snappy]) (4.0.3)
#15 0.774 Requirement already satisfied: packaging in /usr/lib/python3.11/site-packages (from aiokafka[snappy]) (23.1)
#15 1.014 Collecting cramjam; extra == "snappy" (from aiokafka[snappy])
#15 1.298 Downloading cramjam-2.8.1.tar.gz (1.1 MB)
#15 1.551 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 4.3 MB/s eta 0:00:00
#15 1.617 Installing build dependencies: started
#15 5.381 Installing build dependencies: finished with status 'done'
#15 5.382 Getting requirements to build wheel: started
#15 5.452 Getting requirements to build wheel: finished with status 'done'
#15 5.453 Preparing metadata (pyproject.toml): started
#15 5.514 Preparing metadata (pyproject.toml): finished with status 'error'
#15 5.519 error: subprocess-exited-with-error
#15 5.519
#15 5.519 × Preparing metadata (pyproject.toml) did not run successfully.
#15 5.519 │ exit code: 1
#15 5.519 ╰─> [6 lines of output]
#15 5.519
#15 5.519 Cargo, the Rust package manager, is not installed or is not on PATH.
#15 5.519 This package requires Rust and Cargo to compile extensions. Install it through
#15 5.519 the system's package manager or via https://rustup.rs/
#15 5.519
#15 5.519 Checking for Rust toolchain....
#15 5.519 [end of output]
#15 5.519
#15 5.519 note: This error originates from a subprocess, and is likely not a problem with pip.
#15 5.521 error: metadata-generation-failed

Collecting pysam (from Rust->-r requirements.txt (line 7))
.....
checking for BZ2_bzBuffToBuffCompress in -lbz2... no
#14 89.66 configure: error: libbzip2 development files not found
#14 89.66
#14 89.66 The CRAM format may use bzip2 compression, which is implemented in HTSlib
#14 89.66 by using compression routines from libbzip2 http://www.bzip.org/.

[ods]

What platform do you use? cramjam has prebuilt binary wheels for all major platforms, so building from source is usually not needed.
Have you tried to install libbzip2-dev (bzip2-dev, libbz2-dev, or how is it called in your distribution) as error message suggest? If this doesn't help, try asking cramjam's devs for help.

[3figs]

We are using alpine 3.18.4. Our jenkins server has been having issue recently, hence I failed to do the final verification for update.
pip install 'aiokafka[snappy]' is working. I'm able to use aiokafka 0.10.0 with snappy now (Need to install cargo and rust). Thanks a lot.