GLIBC version mismatch in Promtail binary

[iljalukin]

Describe the bug
When attempting to run Promtail, the following error occurs:
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail)
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)

To Reproduce
Install and run promtail 3.1.0 from apt.grafana.com using instructions from https://apt.grafana.com

Expected behavior
Promtail should run without GLIBC version errors.

Environment:

• local install on Debian 11 (bullseye)

Screenshots, Promtail config, or terminal output
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail)
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)

[mrlhansen]

I can confirm that I am seeing the same issue with the RPM repository when installing on AlmaLinux 8.10.

[jostrasser]

Can confirm the same on Ubuntu 20.04.6 LTS (Focal Fossa):

Jul 3 10:52:36 loki promtail[377]: /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/promtail)

A rollback/downgrade to promtail 3.0 is fixing it temporarily.

[Smith4545]

Please roll back any rpm+apt repositories distributing this binary. How can something like this just be distributed to a wide audience? Is there no testing for this stuff? This just breaks log aggregation for whole companies.

[jytx]

I have the same problem

[anhdowastaken]

Same with Rocky Linux 8.8

[Vitaliy155]

the same issue on AlmaLinux release 8.9

[jacobamey]

Same issue on Amazon Linux 2

[Aethylred]

same issue on Rocky8

# promtail --version
promtail: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by promtail)
promtail: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by promtail)

[adamw-linadm]

almalinux 8.10 also :(

[jostrasser]

@JStickler any updates when this will be fixed?

[mmagdy88]

Same issue here on Ubuntu 20

Edit: Working on Ubuntu 22 because GLIBC is version 2.35, but on Ubuntu 20, GLIBC is version 2.31

[bursan85]

Good evening,

Issue is present with promtail-3.1.1-1.x86_64, so i went for an older version this package for the moment.
In my case, for AlmaLinux release 8.10 (Cerulean Leopard) i have installed an older version:
dnf install promtail-2.9.9-1.x86_64

[Alex4Codes]

CentOS 8 stream problem version 3.1.1 - /lib64/libc.so.6: version `GLIBC_2.32' not found

[bpfoster]

3.0.1 is also affected

[shantanugadgil]

This affects Amazon Linux 2 as well (CentOS 7 derivative + updated kernel etc.)

Amazon Linux 2023 works ok.

EDIT: Downgrading to 2.9.9 works on older OSes.

[boite]

Tried downgrading to 3.0.1 and 2.9.10 and finally got it working with 2.9.9 on Ubuntu 20

$ grep -i pretty /etc/os-release
PRETTY_NAME="Ubuntu 20.04.6 LTS"
$ dpkg -s promtail | grep Version
Version: 2.9.9

[shantanugadgil]

I assume the fix would be to use a build container/os of a lower glibc version!

[adamw-linadm]

Or build trully static go binary.

[fauust]

Same here, install from binaries, downgrade from 2.9.10 to 2.9.9 as a workaround (Debian 11, x86_64).

[bkkoli]

Same Issue occurs to me.

My OS is Debian 11.7
My Kernel version is 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

[bkkoli]

I check my GLIBC Version as belows.

ldd --version

My GLIBC Version is ldd (Debian GLIBC 2.31-13+deb11u6) 2.31

If you use a version of Promtail that is compatible with the GLIBC version of your OS, this may fix the problem.

[bkkoli]

@iljalukin
In my case, Promtail v2.9.9, It works well.

[iljalukin]

@iljalukin In my case, Promtail v2.9.9, It works well.

This does not solve my problem, as I want to install the latest available version via the APT repository as described above.
My workaround is to install version 3.0.0 with apt install promtail=3.0.0 and pin it with apt-mark hold promtail.
Debian 11 is the current LTS version, which will be maintained until 2026 and it would be nice if promtail could be installed there in the current version.

I hope that the developers will pay attention to this issue and fix it.

[mrlhansen]

It's honestly a little concerning that this has not been fixed after two months. This breaks log aggregation on most enterprise Linux distributions used in production. I have decided to disabled the Grafana RPM repository completely and moved working versions of the RPMs that I need into a private repository to avoid these things affecting production. But if this is not fixed in a reasonable time, then Grafana Loki is not the right solution.

[BlackMoon2020]

Indeed concerning, hopefully due to holiday, meanwhile running old version. If not fix in long term, will need to look for alternative

[bkkoli]

I think there is no different way except updating OS which support target GLIBC library.

In production, Because It is hard to update OS version, It is better to make binary file less dependent on libraries like GLIBC closely related to the operating system.

[shantanugadgil]

#13393 (comment)

#13393 (comment)

I am not a GitHub workflow expert, so I am not able to submit the relevant PR to downgrade the "build environment", but I believe that the workflow is building on the latest Ubuntu (based on a rudimentary search inside the .github/ directory) which would have a glibc >= 2.32

My use case is to support Amazon Linux 2, which uses glibc 2.26

$ cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)

$ ldd --version
ldd (GNU libc) 2.26
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

$ rpm -qa | grep glibc-2
glibc-2.26-64.amzn2.0.2.x86_64

[shantanugadgil]

@cyriltovena could you help with this?

[cervenka-riege]

This looks to me like a pretty significant issue, with a relatively easy fix. This issue is preventing us from updating above 2.9.9.

Same for us.

[shantanugadgil]

same issue persists with the latest 3.2.0 release as well.

[BlackMoon2020]

Very concerning, 3.2.0 rollout and still not working on linux.

So its now a fact loki/promtail will not support linux then it should be mention in main page.
Will start looking for alternative, cannot stay in prod with unsupported application

[danpoltawski]

So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application

I think the alternative that Grafanalabs want us to take is to move to Alloy - but it would be helpful if this was communicated explicitly rather than just leaving promtail to rot. 😢 it’s quite concerning lack of engagement given the impact, like you say.

[BlackMoon2020]

So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application

I think the alternative that Grafanalabs want us to take is to move to Alloy - but it would be helpful if this was communicated explicitly rather than just leaving promtail to rot. 😢 it’s quite concerning lack of engagement given the impact, like you say.

Tested this am, alloy. Compatible with existing promtail conf.

Alloy run --storage.path=/....(data folder) --config.format=/alloypath/conf/promtail_config.yaml

Feeding nicely loki, clearly the way to go, running in uat in one server, need to check further.
Thx for your input.

You can also convert promtail conf but for now easier to use the option to run promtail format easier for migration.

[IgorOhrimenko]

Debian 11 end of life as August 31st, 2026, but it has glibc 2.31. Why I have to use only 2.9.9 and cannot latest version promtail on the Debian 11?

[J0WI]

I could be completely wrong, but could changing the "build OS" here, help?

loki/.github/workflows/release.yml

Line 21 in 4e17ad8
runs-on: "ubuntu-latest"

The runner image doesn't matter here, they build the binaries in the loki-build-image. I think this was introduced with 71a8f2c. Here, they're pulling in libsystemd from bookworm which is build against the newer version of glibc in Debian:

loki/loki-build-image/Dockerfile

Line 85 in ae955ed

libsystemd-dev jq && \

This image is then used to build the binaries in .github/workflows/minor-release-pr.yml respectively patch-release-pr.yml. They both upload their artifacts to a BUILD_ARTIFACTS_BUCKET. release.yml is just downloading it from there to publish on GitHub.

I'd suggest to revert the build-image to FROM golang:${GO_VERSION}-bullseye for now. The bullseye tag is supported until the GA of trixie. See also docker-library/golang#466.

[nevesm]

Same here on CentOS Linux 7 (Core) using promtail-3.2.0.x86_64.rpm

[shantanugadgil]

Another workaround is to build the binary on the OS you want to run it on (lowest GLIBC versioned OS).

Needs an installed version of golang and gcc

git clone from main or the tag v3.2.0

# make promtail
# ./clients/cmd/promtail/promtail --version
# promtail, version 3.2.0 (branch: HEAD, revision: 659f5421d)
  build user:       user@machine
  build date:       2024-09-27T11:31:41Z
  go version:       go1.23.1
  platform:         linux/arm64
  tags:             unknown

[SckyzO]

Hello, same bug here, So i just build promtail from source.

In WSL, just install goland

sudo apt install golang gcc

Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0

cd /tmp
git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki
cd loki
CGO_ENABLED=0 go build ./clients/cmd/promtail
ls -al promtail

[TomiCheng]

redhat 8.10 also -_-

[x4e-jonas]

git clone from main or the tag v3.2.0

# make promtail
# ./clients/cmd/promtail/promtail --version

Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0

cd /tmp
git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki
cd loki
CGO_ENABLED=0 go build ./clients/cmd/promtail
ls -al promtail

You need to set PROMTAIL_JOURNAL_ENABLED="true" or build with -tags promtail_journal_enabled otherwise promtail is not linked to libsystemd/ glibc at all.

[Jayclifford345]

Hi all, we are extremely sorry this issue has gone under the radar for so long. We should have identified this issue faster. The Loki team is now aware of the issue and is working on a fix within the build process.

[digdilem]

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

[cstyan]

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

Promtail is in essence deprecated in the sense that all of our long term efforts are behind Grafana Alloy. For Promtail we're continuing to accept bug fixes, security patches, and minor updates/fixes such as this one. However, Promtail will not be getting new features.

[slim-bean]

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

As @cstyan mentioned, Alloy has superseded Promtail in every way at this point and we would consider it the superior product.

We forked Promtail code into Alloy some time ago making maintenance a challenge for us.

We've not been in any particular hurry to announce long term plans around Promtail because the overall maintenance burden is not that high. That being said, we've decided not to add new features to it and rather only focus new features on Alloy and announcing deprecation of Promtail is a logical next step.

All that being said, I would love to understand your sentiment better though @digdilem, I think our opinion is that moving to Alloy is an improvement but if that's not everyone's perception we would love to learn more to hopefully make sure our decisions are not alienating users.

[digdilem]

All that being said, I would love to understand your sentiment better though @digdilem, I think our opinion is that moving to Alloy is an improvement but if that's not everyone's perception we would love to learn more to hopefully make sure our decisions are not alienating users.

@slim-bean - thanks for the opportunity. My sentiment is fairly simple and it's not that strong. I'm a sysadmin who helps maintain a couple of hundred linux vms running literally hundreds of different pieces of software, of which promtail is one. My devs asked for it to be installed on around 40 machines, I did so, and I maintain it. I'm not particularly angry or emotional, but I can identify my concerns and, I hope, express them clearly.

Grafana made it easy in providing repos for our target distro (EL - specifically Rocky 8 and 9), and quite well behaved software. It didn't take much effort to please the devs, and happy devs is one of our KPIs. All good.

Then we noticed this error - no problem, software is complicated and breaks. A dependency issue is relatively common and understandable. We found this thread and our generally positive view of Grafana was slightly dented. Why was it being ignored? Why was Grafana not fixing a pretty major issue reported in the official way, that looks easy to do? It's clear from comments that I'm not alone in being puzzled and when there's nobody listening, your mind starts to wander.

Ignored and uninformed, I (speaking personally) started wondering if there was an agenda to let promtail die an ignoble death to force people to adopt alloy. We've all seen this type of cynical behaviour before and it's damaging to the users, to the software company's reputation, and to foss generally. I wasn't convinced Grafana would do this - I like the company and it does good stuff - and I'm glad that niggling voice in my head wasn't correct. This was an oversight, nothing more.

In my experience, people - either as users of software or in life generally - want tomorrow to be much like today. Waking up one morning and finding out you've now got to spend some unplanned hours to migrate from one piece of software to another upsets the balance of the day and makes sysadmins grumpy. We like warning, we like clear communications, we like to plan ahead. The IT world is more full of dodgy practices than ever before (see recent price hikes and model changing (eg vmware/prtg - and they won't be the last)) and we're probably more quick to see shapes in the shadows where they don't actually exist. I'm guilty of that here, I think.

Sorry, written too much already - but the basis of my sentiment is that I have no issues with alloy replacing promtail. Provided it's as well behaved from a system's point of view, great. Change is the one constant in IT. We don't need alloy - promtail gets logs from our java services and sends them to loki. That's what we need it for and it does that well. Whatever new features alloy has, we almost certainly don't need or want them - but I'm not emotionally attached to promtail so if we have to change because you can't support both forever, so be it.

My sentiment was stirred because this issue wasn't being responded to - nobody likes to be ignored. I'm glad it was just an accidental oversight and not deliberate. No big drama here.

[Jayclifford345]

Hey all, thank you for your patience on this issue. We have a PR ready to change the build image causing the problem. We wanted to do some testing to make sure doing this wouldn't invalidate for new OS's. Keep us in the loop with your own testing once this release it cut so we can keep an eye on things.

#14368

Once again thank you for bringing this to out attention.

[chaudum]

Re-opening, because the issue was automatically closed by #14368 but new versions haven't been released yet.

[skazazes-work]

Is there a timeline for a working release? I am currently in the process of setting up promtail and do not want to manually install 2.9.9 if a new release will be out in the coming days.

Thanks!

[chaudum]

Is there a timeline for a working release? I am currently in the process of setting up promtail and do not want to manually install 2.9.9 if a new release will be out in the coming days.

Thanks!

There is no timeline for 3.2.x and 3.1.x patch release yet.

I understand that you would wanna wait for them, but tbh, there hasn't been many changes in Promtail between 2.9.x and 3.2.x

[smaddock]

It would be helpful if this documentation was updated to refer to Alloy; the Loki Getting Started docs are updated but not the Loki Install docs.

[Jayclifford345]

Great catch @smaddock! Will get that updated now

<-- Update -->
Docs Update:
#14404