port appimage to linuxdeployqt type2 #98

LyzardKing · 2018-06-20T11:57:03Z

As discussed on IRC
This PR uses the new linuxdeployqt for the appimage build.

Note that I disabled the irc part temporarily

LyzardKing · 2018-06-20T12:24:47Z

There is an automated "continuous build" upload tool we can use easily:
https://github.com/probonopd/uploadtool#usage

Specifically we need it in the travis integration, as described here:
https://github.com/probonopd/linuxdeployqt#using-linuxdeployqt-with-travis-ci

Normally that builds also appimages for pull requests, uploads to transfer.sh and links in the PR.
Could that be a useful feature?

LyzardKing · 2018-06-20T16:23:10Z

@probonopd
How can I copy files before building the appimage?
I try with the cp python and cp tcltk,
but I get these errors:

cp: cannot stat â€˜(0x00007ffc95693000)â€™: No such file or directory
â€˜/usr/lib/x86_64-linux-gnu/libtk8.6.soâ€™ -> â€˜appdir/usr/lib/libtk8.6.soâ€™

as can be seen in the travis log: https://api.travis-ci.org/v3/job/394609547/log.txt

EDIT: the second line is just the quotes not showing in the txt file...the error is the first line

LyzardKing · 2018-06-20T16:31:26Z

Although extracting the appimage, the file is in the right place...so that's not a problem?
There is an issue though...
If I run it (and it is built on travis) I get:


./Scribus-c9b5ebe-x86_64.AppImage: /tmp/.mount_ScribuNxavbh/usr/bin/../lib/libc.so.6: version `GLIBC_2.25' not found (required by /lib/x86_64-linux-gnu/libexpat.so.1)
./Scribus-c9b5ebe-x86_64.AppImage: /tmp/.mount_ScribuNxavbh/usr/bin/../lib/libc.so.6: version `GLIBC_2.26' not found (required by /usr/lib/x86_64-linux-gnu/libp11-kit.so.0)
./Scribus-c9b5ebe-x86_64.AppImage: /tmp/.mount_ScribuNxavbh/usr/bin/../lib/libc.so.6: version `GLIBC_2.25' not found (required by /lib/x86_64-linux-gnu/libbsd.so.0)

probonopd · 2018-06-20T17:02:13Z

Thanks for you great work @LyzardKing. Two things should be changed:

First, libc.so.6 should not end up in the bundle since it is on the excludelist. I wonder why it does.libc.so.6 is a basic system library that we should assume to be present in every target system (distribution) in a recent enough version.

Possibly this is a linuxdeployqt bug (to be confirmed). Can you run linuxdeployqt -show-exclude-libs and post the output?

Second, something pulls in libstdc++-4.8-dev amd64 4.8.4-2ubuntu1~14.04.4 and libstdc++6 amd64 4.8.4-2ubuntu1~14.04.4 which are newer than what comes with stock Ubuntu 14.04, and will cause issues if not handled properly.

You may need to use something like this.

LyzardKing · 2018-06-20T17:04:05Z

I fixed a bug in my build (I didn't change some hardcoded paths, so I believe it was looking at some libraries outside the appimage).
But the libc.so being in the appimage is still an issue, yes...

LyzardKing · 2018-06-20T17:07:53Z

Do I have to run linuxdeployqt -show-exclude-libs in travis?

probonopd · 2018-06-20T17:09:35Z

Yes, so that we can learn what might be going wrong in the linuxdeployqt that is being used.

Looks like a bug to me given this...

https://github.com/probonopd/linuxdeployqt/blob/5542062897811e9ff061e8debd929467e921be4e/tools/linuxdeployqt/excludelist.h#L26

LyzardKing · 2018-06-20T17:13:23Z

The ldd I'm using to pull extra libraries does have libc.so.6..so that might be the issue:

ldd appdir/usr/_tkinter.so | grep "=>" | awk '{print $3}'
/usr/lib/libBLT.2.5.so.8.6
/usr/lib/x86_64-linux-gnu/libtk8.6.so
/usr/lib/x86_64-linux-gnu/libtcl8.6.so
/lib/x86_64-linux-gnu/libc.so.6
/usr/lib/x86_64-linux-gnu/libX11.so.6
/lib/x86_64-linux-gnu/libm.so.6
/usr/lib/x86_64-linux-gnu/libXft.so.2
/usr/lib/x86_64-linux-gnu/libfontconfig.so.1
/usr/lib/x86_64-linux-gnu/libXss.so.1
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libz.so.1
/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libxcb.so.1
/usr/lib/x86_64-linux-gnu/libfreetype.so.6
/usr/lib/x86_64-linux-gnu/libXrender.so.1
/lib/x86_64-linux-gnu/libexpat.so.1
/usr/lib/x86_64-linux-gnu/libXext.so.6
/usr/lib/x86_64-linux-gnu/libXau.so.6
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6
/usr/lib/x86_64-linux-gnu/libpng16.so.16
/lib/x86_64-linux-gnu/libbsd.so.0
/lib/x86_64-linux-gnu/librt.so.1

Is there a way to compare this with the list generated by linuxdeployqt?
I thought this was done automatically...

LyzardKing · 2018-06-20T17:16:48Z

what I have is this:
ldd appdir/usr/_tkinter.so | grep "=>" | awk '{print $3}' | xargs -I '{}' cp -v '{}' appdir/usr/lib || true;
copied from the old appimage format.
Is there a way to set in the xargs to verify if it's in the excluded libs?

probonopd · 2018-06-20T17:23:24Z

You are right,

https://github.com/scribusproject/scribus/pull/98/files#diff-354f30a63fb0907d4ad57269548329e3R69

is likely causing this. Note that linuxeployqt does more to libraries than just copying them in, it also changes their rpath and makes sure that it does not copy the blacklisted ones.

What happens if you remove that line and do

# Call linuxdeployqt for the first time as usual, then run:
./linuxdeployqt-continuous-x86_64.AppImage --appimage-extract
mv appdir/usr/_tkinter.so appdir/usr/lib/
( cd appdir/usr/ ; ln -s .(lib/_tkinter.so . )
./squashfs-root/usr/bin/linuxdeployqt -bundle-non-qt-libs appdir/usr/lib/_tkinter.so
# Then call linuxdeployqt for the second time as usual

instead? Will it deploy the additional dependencies of appdir/usr/lib/_tkinter.so?

LyzardKing · 2018-06-20T17:34:52Z

So I don't have to manually copy with the ldd?
I'm just asking because it takes a long time to compile..

LyzardKing · 2018-06-20T17:35:49Z

Also, the first linuxdeployqt is without -appimage, right? that can be done only at the end..
what about the -extra-plugins, do those go both times?

probonopd · 2018-06-20T17:36:45Z

So I don't have to manually copy with the ldd?

Yes. That's the plan. I wonder what will happen :-)

Also, the first linuxdeployqt is without -appimage, right? that can be done only at the end..

Yes.

Note that I updated the above script, there was a mistake.

LyzardKing · 2018-06-20T17:40:50Z

I pushed the changes...did I do it right?

LyzardKing · 2018-06-20T20:05:30Z

@probonopd It ignores the -extra-plugins=iconengines/libqsvgicon.so
without the extra linuxdeployqt pass it added that properly..but it seems that running it more than once
doesn't really work
How can I add it manually?

probonopd · 2018-06-21T05:33:01Z

.travis.yml

+      mv appdir/usr/_tkinter.so appdir/usr/lib/;
+      ( cd appdir/usr/ ; ln -s ./lib/_tkinter.so . );
+      ./squashfs-root/usr/bin/linuxdeployqt -bundle-non-qt-libs appdir/usr/lib/_tkinter.so;
+      cp /opt/qt58/plugins/iconengines/libqsvgicon.so appdir/usr/plugins/;


Never manually copy stuff into the AppImage. This does not properly deploy its dependencies, and does not set the rpath. Instead, pass in -extra-qt-plugins as described here: https://github.com/probonopd/linuxdeployqt/#adding-extra-qt-plugins

LyzardKing · 2018-06-21T08:12:43Z

Ok, so I guess it's not a missing plugin, and the sed replacement used in the previous appimage script is no good since this is not a script...
It might still have harcoded paths for the icon folder

LyzardKing · 2019-01-19T13:16:46Z

@probonopd
my last comment to this PR is what I meant in the other issue you opened..

LyzardKing · 2019-01-19T13:24:51Z

.travis.yml

-    make -j"$NPROC";
-    make install;
-    bash -ex ./AppImage-package/bundle.sh;
+      cmake . -DCMAKE_INSTALL_PREFIX=/usr -DWANT_HUNSPELL=1 -DWITH_PODOFO=1 -DWANT_GRAPHICSMAGICK=1 -DWANT_DEBUG=0 -DWANT_SVNVERSION=0 -DWANT_GUI_LANG=en_US  -DBUILD_APPIMAGE_BUNDLE=1;


@probonopd
Here I added a BUILD_APPIMAGE setting..but I have no idea how it's supposed to work...
I should use this to change the build process, change the paths, and for example the updates integration...
in the paths file I used Q_OS_LINUX...should I use BUILD_APPIMAGE? and how

Where is -DBUILD_APPIMAGE_BUNDLE=1 defined? I've never heard of it

No, I added it in the travis buid command, to then use it as a condition in the build process...
I need a way to select some options only if I'm building an appimage...

I have no idea how it's supposed to work

Well if you code something that even you have no idea how it's supposed to work, how should I? No offense, but maybe if you could explain what you want to do then I might be able to help or at least suggest alternative solutions.

Also I think an AppImage should be built for each run of the CI.

lyzardking's proposal was a not 100% complete patch that needed further work.

it was a best effort to help in a/ fixing the appimage to build again b/ get it to work correctly with python (iirc)

iirc, the -DWANT_APPIMAGE_BUNDLE variable is probably needed to switch from relative to absolute paths for SHARE.

LyzardKing · 2019-01-20T18:16:32Z

@probonopd If a docker image is the new solution to dependency issues,
then, whithout moving to gitlab the script can be used in github as well, with travis, if the current devs are familiar with that...
https://docs.travis-ci.com/user/docker/
If we can avoid another layer I think it would be easier to manage..
also, isn't the point of appimages that they are built on older systems? why is the docker image based on 18.04?

probonopd · 2019-01-20T18:25:37Z

@probonopd If a docker image is the new solution to dependency issues,
then, whithout moving to gitlab the script can be used in github as well, with travis, if the current devs are familiar with that...

Yes, but how do you create and host a custom Docker container on Travis CI? I am doing it in https://gitlab.com/probono/scribus-ci-docker/.

also, isn't the point of appimages that they are built on older systems?

Well, the point of AppImages is that they run on older systems too, not just the latest bleeding-edge ones. To achieve this, we generally recommend to build what goes into an AppImage on a build system no newer than the oldest still-supported distribution, which for Ubuntu currently is 14.04.

A fallback option (which I am currently testing) if it is not possible to build on the oldest still-supported distribution is to bundle everything inside the AppImage, down to glibc and ld-linux. Then the resulting AppImage should run also on systems older than the build system. But again, it is a fallback.

why is the docker image based on 18.04?

Because I wanted to have a newer Qt and other dependencies than what is available for Ubuntu 14.04, and I didn't want to build everything from source. And because I wanted to experiment with an AppImage that bundles everything.

Still, if we can make it work on 14.04 it's much better...!

LyzardKing · 2019-01-20T19:31:29Z

@probonopd
the build now works again, but I need to enable the path changes (https://github.com/scribusproject/scribus/pull/98/files#diff-14e7e8d3c493d0f0857d6d05320a8c2dR73)
if I only want to change them when building an appimage, how would I use cmake for that?
I tried https://github.com/scribusproject/scribus/pull/98/files#diff-af3b638bc2a3e6c650974192a53c7291R1115
(as with older versions, the travis error is on macos)

probonopd · 2019-01-20T19:46:36Z

That's awesome news 👍

Why would you want to do that only for AppImage builds? If Scribus loads its resources from a path relative to itself, it will work both for AppImage and for "normal" installs...

LyzardKing · 2019-01-20T19:49:49Z

I guess that would be a question for @aoloe
If the paths are correct also for a normal release..
then we could remove the need for an extra option and just say that works on any linux distro

aoloe · 2019-01-21T14:02:31Z

what do the other projects do?
i can ask the debian maintainer, what he thinks about this...

probonopd · 2019-01-21T18:31:45Z

Inkscape has a configure option to switch on binreloc, which means relative paths will be used for everything.

mapreri · 2019-01-22T12:47:06Z

scribus/scpaths.cpp

+// ones inside the bundle, relative to the executable.
+#ifdef Q_OS_LINUX
+	// Set the application name expliticly.
+	QCoreApplication::setApplicationName("scribus");


What's the goal of this? Are you imagining occasions where the default of "executable name" is wrong?

I added that because the file configs were set in a different location every time I installed a new version.
I can't remember now (I told @aoloe in irc a while back) but I believe the folders were named scribus-<version> or similar

And this fixed it? It feels weird, I would love to have more details about this… @aoloe ?

the details should be somewhere in irc...
The issue is that there was no application name set properly...
we could test this again if you want..

mapreri · 2019-01-22T12:51:02Z

scribus/scpaths.cpp

+	m_templateDir = QString("%1/share/scribus/templates/").arg(pathPtr);
+	m_libDir = QString("%1/lib/scribus/").arg(pathPtr);
+	m_pluginDir = QString("%1/lib/scribus/plugins/").arg(pathPtr);
+	m_qmlDir = QString("%1/share/scribus/qml/").arg(pathPtr);


All the scribus in the hardcoded paths there should use whatever C++ const is defined by the cmake variable TAG_VERSION.

why should we add a version number in the folders?
would that not just add a new folder on every update?

What is TAG_VERSION defined as. By default it is an empty string, but whoever builds it can use it to be able to build different co-installable builds of scribus.

For example, for the Ubuntu PPA I'm providing 3 builds, 'scribus', 'scribus-ng' and 'scribus-trunk', all of them coinstallable. For example, in the latest TAG_VERSION is -trunk.

I don't know why you would be interested in adding a "number", but I am. :)

Ok, I thought it was a value that was not empty by default...
how would I add the cmake value in the cpp code?

mapreri · 2019-01-22T12:56:49Z

The paths are correct for a regular release, and except from the point I made as an inline comment, it wouldn't affect the Debian and Ubuntu packages.

However, I notice that now that all of linux, macos and windows (these are the only platform scribus supports, right?) would be overriding compile-time consts with the value obtained at runtime. Which is probably fine, but at this point you should probably looking also at cleaning up cmake and stop bothering defining such paths there (and while at it, check that all the thoughts cmake is having are reflected in the runtime detection logic).

LyzardKing · 2019-01-22T13:00:05Z

Thanks for looking at it @mapreri
Just a couple of questions, to better understand the build process and folder structure.

aoloe · 2019-01-22T14:45:22Z

However, I notice that now that all of linux, macos and windows (these are the only platform scribus supports, right?)

if we are realistic, yes.
if you ask some members of the team, scribus will run on your razor, too.
ok, just joking. as far as i know, beos, freebsd and os/2 are also supported. the list might not be complete.

LyzardKing · 2019-02-24T15:56:15Z

So I guess this PR is useless now, if the builds are now happening on gitlab...
It can be closed then..

aoloe · 2019-05-03T08:01:08Z

can anybody have a look at https://gitlab.com/impagina/scribus-patched/compare/master...appimage ?

it should implement:

add an optional "RELATIVE_PATH" to the CMakeLists.txt file
compiling in a build directory
only use the gtk2 theme in gnome (other DE can be added)
a bit of clean up in the bundle.sh file

probonopd · 2019-05-03T17:31:02Z

Looks good to me 👍

aoloe · 2019-05-04T08:46:53Z

submitted as https://bugs.scribus.net/view.php?id=15671

once this is sorted out, do you think i can close this request?

probonopd · 2019-05-04T09:02:39Z

Yes, I think so. Thanks for your work on this @aoloe

LyzardKing force-pushed the master branch 4 times, most recently from 46a0030 to 1247184 Compare June 20, 2018 15:13

LyzardKing force-pushed the master branch 3 times, most recently from 33e59a8 to fdd7e51 Compare June 20, 2018 18:57

LyzardKing force-pushed the master branch 2 times, most recently from d6a1560 to de5bebd Compare June 20, 2018 21:54

probonopd reviewed Jun 21, 2018

View reviewed changes

LyzardKing force-pushed the master branch from de5bebd to 9aba129 Compare June 21, 2018 07:16

LyzardKing force-pushed the master branch 3 times, most recently from bd6d861 to 36146a6 Compare June 21, 2018 13:22

LyzardKing commented Jan 19, 2019

View reviewed changes

Add appimage cmake option

9a50fdd

LyzardKing force-pushed the master branch from 9877ec3 to 9a50fdd Compare January 19, 2019 14:45

Add recent poppler build

68b5442

Add recent poppler build

2398c83

LyzardKing force-pushed the master branch from 77436ca to 2398c83 Compare January 20, 2019 18:25

LyzardKing added 2 commits January 20, 2019 19:27

Merge branch 'master' into master

3d54814

Move comment

b715142

remove older ppa

e12e51d

LyzardKing force-pushed the master branch from bacf726 to e12e51d Compare January 20, 2019 19:32

Revert to Q_OS_LINUX

e51649a

mapreri reviewed Jan 22, 2019

View reviewed changes

LyzardKing closed this Mar 28, 2020

port appimage to linuxdeployqt type2 #98

port appimage to linuxdeployqt type2 #98

Conversation

LyzardKing commented Jun 20, 2018 • edited Loading

LyzardKing commented Jun 20, 2018 • edited Loading

LyzardKing commented Jun 20, 2018 • edited Loading

LyzardKing commented Jun 20, 2018

probonopd commented Jun 20, 2018 • edited Loading

LyzardKing commented Jun 20, 2018

LyzardKing commented Jun 20, 2018

probonopd commented Jun 20, 2018

LyzardKing commented Jun 20, 2018 • edited Loading

LyzardKing commented Jun 20, 2018

probonopd commented Jun 20, 2018 • edited Loading

LyzardKing commented Jun 20, 2018

LyzardKing commented Jun 20, 2018 • edited Loading

probonopd commented Jun 20, 2018 • edited Loading

LyzardKing commented Jun 20, 2018

LyzardKing commented Jun 20, 2018 • edited Loading

Choose a reason for hiding this comment

LyzardKing commented Jun 21, 2018

LyzardKing commented Jan 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LyzardKing commented Jan 20, 2019

probonopd commented Jan 20, 2019 • edited Loading

LyzardKing commented Jan 20, 2019

probonopd commented Jan 20, 2019 • edited Loading

That's awesome news 👍

LyzardKing commented Jan 20, 2019

aoloe commented Jan 21, 2019

probonopd commented Jan 21, 2019

Choose a reason for hiding this comment

LyzardKing Jan 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mapreri commented Jan 22, 2019

LyzardKing commented Jan 22, 2019

aoloe commented Jan 22, 2019

LyzardKing commented Feb 24, 2019

aoloe commented May 3, 2019

probonopd commented May 3, 2019 • edited Loading

aoloe commented May 4, 2019

probonopd commented May 4, 2019

LyzardKing commented Jun 20, 2018 •

edited

Loading

LyzardKing commented Jun 20, 2018 •

edited

Loading

LyzardKing commented Jun 20, 2018 •

edited

Loading

probonopd commented Jun 20, 2018 •

edited

Loading

LyzardKing commented Jun 20, 2018 •

edited

Loading

probonopd commented Jun 20, 2018 •

edited

Loading

LyzardKing commented Jun 20, 2018 •

edited

Loading

probonopd commented Jun 20, 2018 •

edited

Loading

LyzardKing commented Jun 20, 2018 •

edited

Loading

probonopd commented Jan 20, 2019 •

edited

Loading

probonopd commented Jan 20, 2019 •

edited

Loading

LyzardKing Jan 22, 2019 •

edited

Loading

probonopd commented May 3, 2019 •

edited

Loading