Setup Linux CI for Qt5.15.8+KDAB using minimum glibc version compatibility

[alexjba]

Description

Prebuilt binaries of Qt5.15.8 are compiled using glibc 2.28 and migrating to this version will breaking compatibility with older linux versions.

The purpose of this task is to setup the Linux CI to use latest Qt5.15.8 + KDAB fixes and keep the glibc dependency to a lower version (Ideally 2.23 or lower). Also, we will have this issue when moving to Qt6 and solving this will also help long term with newer qt versions.

There are multiple options that could be considered:

1. Finding an out of the box setup - precompiled Qt binaries compatible with older linux versions
2. Compiling Qt5.15.8 with KDAB patches + Status and link with glibc 2.23 (probably the easiest way to do it is on a linux distribution that uses this version)
3. Try this hack https://github.com/wheybags/glibc_version_header and enable Nix environment on CI
4. https://github.com/phusion/holy-build-box

On the CI side there are two approaches that were considered when migrating to Qt5.15: Docker and Nix. Nix is preferred, but it has the same limitation - will break compatibility with older linux dostros. We're using Docker for now.

See: #9222, #9232

[anastasiyaig]

@alexjba @jakubgs we need that for all platforms (windows and mac inclusive). windows now is 5.15.2 and arm mac is 5.15.8 when intel is 5.15.5

[jakubgs]

Actually, we could combine using Nix together with use of https://github.com/wheybags/glibc_version_header.
And that might be the best option for us, since it would provide locked Qt for both Linux and MacOS, and possibly even Windows if we managed to use WSL for CI. Though that might require extra work.

[yakimant]

Hi guys, I started to research on this issue.

Here is a first finding - there is no binary for Qt 5.15.3+ available.

❯ aqt list-qt linux desktop
5.9.0 5.9.1 5.9.2 5.9.3 5.9.4 5.9.5 5.9.6 5.9.7 5.9.8 5.9.9
5.10.0 5.10.1
5.11.0 5.11.1 5.11.2 5.11.3
5.12.0 5.12.1 5.12.2 5.12.3 5.12.4 5.12.5 5.12.6 5.12.7 5.12.8 5.12.9 5.12.10 5.12.11 5.12.12
5.13.0 5.13.1 5.13.2
5.14.0 5.14.1 5.14.2
5.15.0 5.15.1 5.15.2
6.0.0 6.0.1 6.0.2 6.0.3 6.0.4
6.1.0 6.1.1 6.1.2 6.1.3
6.2.0 6.2.1 6.2.2 6.2.3 6.2.4
6.3.0 6.3.1 6.3.2
6.4.0 6.4.1 6.4.2 6.4.3
6.5.0 6.5.1 6.5.2 6.5.3
6.6.0

which taked binaries from https://download.qt.io/online/qtsdkrepository/linux_x64/desktop/

See miurahr/aqtinstall#572

Do I miss smth here and there are other "Prebuilt binaries of Qt5.15.8" for Linux?

[yakimant]

Homebrew maintained binaries:
https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged

For example 5.15.8_3-1:
https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1
(includes Linux one)

[yakimant]

Ubuntu packages:

• 20.04: 5.12.8
• 22.04: 5.15.3

https://packages.ubuntu.com/search?keywords=qtbase5-dev&searchon=names&suite=all&section=all

[yakimant]

Compilation doc:
https://doc.qt.io/qt-5/linux.html#building-qt-5-from-source

[caybro]

23.04 apparently has 5.15.8

[alexjba]

Homebrew maintained binaries: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged

For example 5.15.8_3-1: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1 (includes Linux one)

Yes, at that time we've found the binaries in Homebrew, nix packages and the Ubuntu distro @caybro is using.

@caybro Can you please share the url for the Ubuntu distro you're using? IDK if it helps, but for the record 😄

[caybro]

Homebrew maintained binaries: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged
For example 5.15.8_3-1: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1 (includes Linux one)

Yes, at that time we've found the binaries in Homebrew, nix packages and the Ubuntu distro @caybro is using.

@caybro Can you please share the url for the Ubuntu distro you're using? IDK if it helps, but for the record 😄

Not using Ubuntu but Fedora but here you go 😆
https://packages.fedoraproject.org/search?query=qt5-*

[alexjba]

@yakimant The downside of using Homebrew packages is that the Docker image will increase in size. Around 8gb if I remember correctly. Probably it doesn't really matter if we're downloading the image only once per machine and this doesn't affect the CI jobs duration.

[yakimant]

That's interesting, nix uses KDE patched sources for Qt5:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-5/5.15/srcs-generated.json

But original for Qt6:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-6/srcs.nix

[caybro]

That's interesting, nix uses KDE patched sources for Qt5: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-5/5.15/srcs-generated.json

But original for Qt6: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-6/srcs.nix

Because for Qt 6, there's no difference between publicly available OSS and commercial versions

[yakimant]

Thanks for you comments!

How can I reproduce the issue?

No issues for Windows and macOS, right?
(Only finding/building the binaries)

[alexjba]

To reproduce this you'll need to download Qt5.15.8+ binaries and use them to compile status desktop on the Docker image we're currently using.

[yakimant]

Cool! Is it a build time or runtime issue?

[yakimant]

Also, can you please share, which 5.15.8+ binaries did you use, that are compiled with glibc 2.28?
And why it is a problem, if Ubuntu 20.04 has glibc 2.31, which is newer:
https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libc-dev-bin&searchon=names

[alexjba]

Also, can you please share, which 5.15.8+ binaries did you use, that are compiled with glibc 2.28?
And why it is a problem, if Ubuntu 20.04 has glibc 2.31, which is newer:
https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libc-dev-bin&searchon=names

I've been using the homebew binaries. Most probably these ones: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/62123271?tag=5.15.8. It was most probably a link error. Sorry for not documenting it better and I'm writing now from memory 👎 But it's easy to get to the error. All it's needed is to configure the docker image used on CI to use a newer QT and compile status-desktop using this Qt version.

It's a problem because we need to support latest non-LTS Ubuntu version and compiling the app with a more recent glibc will break the compatibility. But now the latest non-LTS Ubuntu version is questionable to me.

@John-44 I see here the latest non-LTS Ubuntu version is now 18.04 https://www.geekersdigest.com/updated-list-of-linux-distributions-with-long-term-support-lts/. Does this mean we must now support 18.04 and the 16.04 support is a best effort?

[yakimant]

@alexjba, I'm quite stuck in crush debugging.
Let's have a look together when you have time.

Or we could have a small pairing session before.
Maybe you could give me some hints, so I could move forward.

[yakimant]

@micieslak helped to disable Qt WebEngine, which we expected to cause a crash: #13540

The app is running and login is succesfull, no crash anymore.

@stefandunca, can you please help to understand, how much do we need / use it?
@anastasiyaig, can you please continue smoke testing?

Thank you guys, great help with the project!

[stefandunca]

...
@stefandunca, can you please help to understand, how much do we need / use it? @anastasiyaig, can you please continue smoke testing?
...

As discussed privately, we need this for Wallet Connect. There is a workaround using a WebView, but it would require more work to re-implement.
Also, the outdated web browser feature for dapp integration, which is now disabled, requires it.

[alexjba]

...
@stefandunca, can you please help to understand, how much do we need / use it? @anastasiyaig, can you please continue smoke testing?
...

As discussed privately, we need this for Wallet Connect. There is a workaround using a WebView, but it would require more work to re-implement. Also, the outdated web browser feature for dapp integration, which is now disabled, requires it.

Not sure it would help. It probably depends on what is the root cause of this crash.
WebView still depends on WebEngine on linux.
https://doc.qt.io/qt-5/qtwebview-index.html

I guess the first step would be to pinpoint the root cause.

[anastasiyaig]

@yakimant i tested this build with my autotests and all looks good. The only thing i noticed, is when i run application in test mode (adding env var STATUS_RUNTIME_TEST_MODE = True) - then my tests cant attach to the nix build. All good in this mode for old build, no errors

Nix build with test mode enabled:
https://ci.status.im/job/status-desktop/job/e2e/job/manual/1466/allure/#categories/c4e6cb3e88347080bb258e8e1856aa48/74c37dd22099f53/
imeoutError: Unable to establish TCP connection with localhost:62966.

Normal nightly with test mode enabled:
https://ci.status.im/job/status-desktop/job/e2e/job/manual/1467/allure/

Wonder what that could be , because my big hope was to upgrade to 5.15.8 and run my tests with this env var

[anastasiyaig]

I also found this today (just normal run of my tests, no test mode enabled):

aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.35' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.35' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.35' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)

as result, squish cant connect and all tests failed

06:57:16:772 Debug: Using server configuration file '/home/jenkins/workspace/status-desktop/e2e/manual@3/squish.ini'
06:57:16:772 Info: Starting server of Squish version 7.2.1.
06:57:16:773 Debug: Read configuration file '/opt/squish-runner-7.2.1/etc/squishserverrc'
06:57:16:773 Info: Listening on 127.0.0.1:5886
06:57:16:773 Info: Listening on ws://127.0.0.1:45517?tGIe2BW4n3oLux1OFV0W
06:57:16:773 Info: Announcing server presence via SSDP 
06:57:17:776 Warning: Channel Connection closed during handshake:43026 rejected: 127.0.0.1
06:57:17:776 Warning: Channel Connection closed during handshake:43040 rejected: 127.0.0.1
07:21:17:280 Debug: Shutting down server

Logs are available here https://ci.status.im/job/status-desktop/job/e2e/job/manual/1564/

@yakimant do you have any isight what that could be? The host selected was Linux 03, maybe it is not yet properly set up?

[yakimant]

• The second issue is about system libstdc++ used, which needs newer version of glibc.

Possible solutions:

• bundle libstdc++

What's interesting libc from Nix Store /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6 is searched, not the /lib/... one. Maybe we need to cleanup the /nix/store/... paths from libs.

[yakimant]

We have a progress on qtwebengine/wallet issue after Athens offsite.

The build loads post-login screen:
https://ci.status.im/job/status-desktop/job/systems/job/linux/job/x86_64/job/package_nix/38/

@anastasiyaig, can you please have a look?
We also nee to sync on the issues you mentioned above.

Meanwhile I will rebase PR from master.

[yakimant]

Doesn't run on 22.04 in my case, just stucks on loading after entering password.

Logs arent different from 20.04 run, but just for the reference, these are the last lines:

Critical: AppMain: Unknown section type (qrc:/app/mainui/AppMain.qml:1169, expression for currentIndex)
Warning: qrc:/app/mainui/AppMain.qml:1139:21: Unable to assign [undefined] to int (qrc:/app/mainui/AppMain.qml:1139, )
Warning: qrc:/app/AppLayouts/Onboarding/OnboardingLayout.qml:151:9: Unable to assign [undefined] to QObject* (qrc:/app/AppLayouts/Onboarding/OnboardingLayout.qml:151, )
Warning: qrc:/main.qml:173:9: Unable to assign [undefined] to QObject* (qrc:/main.qml:173, )
WRN 2024-05-10 12:22:43.493+02:00 Error decoding signal                      topics="signals-manager" tid=12035 file=signals_manager.nim:49 err="Unknown signal received: mediaserver.started"
ERR 2024-05-10 12:22:43.996+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.061+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.075+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.333+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.336+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.879+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule

[yakimant]

Update on current issues:

• Ubuntu 22 - most probably VM or local setup issue. Test ran on the app on CI agen with Ubuntu 22. We need someone to check this on local machine.
• Test mode - I see it's commented out in Jeninsfile, so low priority
• Issue /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)` - solved. I striped RPATH from the AppImage.

So I see no blockers. @anastasiyaig, can you confirm, when you are back.
Especially the Ubuntu 22.

[anastasiyaig]

sure @yakimant , thanks for your time and effort, i will follow up on that when I am back

[yakimant]

@alexjba, @caybro, if you are interested, here is the build:
https://ci.status.im/job/status-desktop/job/systems/job/linux/job/x86_64/job/package_nix/41/

If we don't find blockers - I will polish the code and merge.

The next steps (if everything goes well) would be:

• Build will run in parallel and I can switch later
• Update Qt version to 5.15.12
• Try nix-appimage as a better approach.

[yakimant]

According to @anastasiyaig, smoke tests are passing.
So I will do some cleanup in PR and merge, build will run in parallel.

[anastasiyaig]

yup right we can do more tests afterwards

[yakimant]

Leftovers from #9232:

• jenkins-lib: refactor linux.bundle to have named arguments
• qt: workarounds to be able to build on nixpkgs 20.04
• go: switch to build from binary
• upgrade appimagetool & linuxdeployqt (add runtime jdependency to 1st)
• check shell inputs are not redundant, we might added more than needed
• remote nix store cleanup
• qa: test go upgrade didn't break anything, test translations, run full e2e suit

Next:

• Try nix-appimage
• status-desktop build derivation
• macOS and Windows support
• Qt latest updgrade

This list will be updated

[yakimant]

The latest opensourse version is 5.15.14:

• https://github.com/qt/qt5/tags
• https://download.qt.io/official_releases/qt/5.15/

Comercial is 5.15.17:

• https://www.qt.io/blog/tag/qt-5-15

They delay source code by 1 year, so it will appear like that:

• 5.15.15 - 2024 Aug 31
• 5.15.16 - 2024 Nov 17
• 5.15.17 - 2025 May 22