librdkafka missing from fpco/stack-build:lts-18.28

[ivanbrennan]

The librdkafka system library is not installed in the fpco/stack-build:lts-18.28 docker image, causing failures if you attempt to use hw-kafka-client:

hw-kafka-client             > Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.7: Missing dependency on a foreign
hw-kafka-client             > library:
hw-kafka-client             > * Missing (or bad) C library: rdkafka
hw-kafka-client             > This problem can usually be solved by installing the system package that
hw-kafka-client             > provides this library (you may need the "-dev" version).

I'm not sure when librdkafka went missing from the stack-build images, but it was previously added specifically to support hw-kafka-client, and we can verify it's presence in some images, e.g.

    $ docker run --rm --interactive --tty fpco/stack-build:lts-16.31 apt list --installed | grep kafka
    librdkafka++1/now 1.0.0~1confluent5.2.4-1 amd64 [installed,local]
    librdkafka-dev/now 1.0.0~1confluent5.2.4-1 amd64 [installed,local]
    librdkafka1/now 1.0.0~1confluent5.2.4-1 amd64 [installed,local]

    $ docker run --rm --interactive --tty fpco/stack-build:lts-16.31 dpkg -L librdkafka-dev
    /.
    /usr
    /usr/lib
    /usr/lib/x86_64-linux-gnu
    ...
    /usr/lib/x86_64-linux-gnu/librdkafka.so
    /usr/lib/x86_64-linux-gnu/librdkafka++.so

Likewise, we can verify its absence in lts-18.28:

    $ docker run --rm --interactive --tty fpco/stack-build:lts-18.28 apt list --installed | grep kafka

    $ docker run --rm --interactive --tty fpco/stack-build:lts-18.28 dpkg -L librdkafka-dev
    dpkg-query: package 'librdkafka-dev' is not installed

I don't know if 18.28 is the only image that librdkafka went unexpectedly missing from, as I've only checked 16.31 and 18.28.

[juhp]

I wonder what is going wrong, since we were/are still installing librdkafka-dev apparently, as can be seen in debian-bootstrap.sh.

Also lts-18 is no longer maintained: dunno if it is possible for you to test lts-20.

[juhp]

Also you can try docker pull ghcr.io/commercialhaskell/stackage/build:lts18, etc

[alaendle]

@juhp Do you know what these fpco images are all about? The regular build uses the ghcr images you've pointed to and they are defined by https://github.com/commercialhaskell/stackage/blob/master/Dockerfile and https://github.com/commercialhaskell/stackage/tree/master/docker - on the other hand the fpco images are defined here https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles. Both images are build/published on our server - but I really wonder why we have these variants? And if we need to support them? 😄

[juhp]

Ah I think I see the problem: the Dockerfile for stack-build still assumes the installation setup is a single script debian-bootstrap.sh, whereas it needs to read the newer docker/*.sh scripts as well.

But I don't really see any particular reason not to use the github registry stackage image instead. Well I am not clear why/if we still need both types of image: if the docker hub stack-build image is redundant perhaps we could even stop pushing it.

[alaendle]

Also #6785 indicates that there are still other users of the fpco/stack-build images.

@juhp - 👍 and

stackage/automated/dockerfiles/lts-20.0/Dockerfile

Line 47 in 8cf2cc2

wget -qO- https://raw.githubusercontent.com/fpco/stackage/$BOOTSTRAP_COMMIT/debian-bootstrap.sh | sed "s/^GHCVER=9.0.1$/GHCVER=$GHC_VERSION/" | GHCVER=$GHC_VERSION bash; \

fails silently due to the fact that https://raw.githubusercontent.com/fpco/stackage/80f68bd733963b71c09b1e8ef6c19bf5b3347245/debian-bootstrap.sh no longer exists 😄

@ivanbrennan Does ghcr.io/commercialhaskell/stackage/build satisfy your needs?

[juhp]

One other difference though is that the github image doesn't include ghc anymore (since we use stack and its installed compiler(s) anyway): I also see that the current docker hub lts image still has ghc-9.2.5...

[alaendle]

Dockerfile for fpco points now to the new scripts (docker/*.sh) - so my expectation would be that the next lts (21.6 or 21.7) images include librdkafka again. Nonetheless I advice to use the ghcr images - they are used internally during our builds and are therefore better maintained.

[alaendle]

@ivanbrennan Thanks for pointing this out - if there are still issues just let us know.

[alaendle]

I've fixed and rebuild images for lts-21.6.

But I really wonder if we should publish/maintain these images. I tend to discard them, because we now have the images on GHCR and the images published to fpco on docker hub where pretty useless since lts-18 - and there were almost no complaints. Also building these images takes lot of time for LTS builds (which is already painful).

However this is another part of the story. To summarize things here - from now one the fpco images should be functional again.