Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

help building for alpine #334

Open
toxic0berliner opened this issue Oct 28, 2024 · 15 comments
Open

help building for alpine #334

toxic0berliner opened this issue Oct 28, 2024 · 15 comments

Comments

@toxic0berliner
Copy link

Hello,

I've been trying to build duc for alpine to run it as a minimal container.
I do have it running on debian:12 quite smoothely, but I'm stuck with alpine, compile looks good, but I'm gettin gthe following error when trying to index anything :

Error opening: /duc/.duc.db - Unknown error, contact the author
Unknown error, contact the author

I am using my own dockerfile available here, as you can see I have tried to get all dependencies and had to build tkrzw from source. Maybe my using the master branch of tkrzw is the issue and you have a dependency to a specific version ?

Thanks in advance for any kind of help sorting this out.
@l8gravely
Copy link
Collaborator

l8gravely commented Oct 29, 2024 via email

@toxic0berliner
Copy link
Author

toxic0berliner commented Oct 30, 2024 •
edited
Loading

Thanks a lot for your quick feedback !

Can you tell me which version of duc you're trying to use? [...] you're going to
get a bunch of garbage when you use --version, which I need to fix.

The version is indeed 1.5.0-rc1 but there is not much garbage in the output 😄 :

f5b996fee160:~# duc --version
Unknown option 'database'
duc version: 1.5.0-rc1
options: cairo x11 ui tkrzw

And since you're using tkrzw, could you use the utilities to check the
file?

Sure, I tried this to recreate the issue and get a db file that it claims it can't open. (click to unfold and see my exact commands to build this db & the error again) 
f5b996fee160:~# pwd
/duc
f5b996fee160:~# ls -alh ./
total 16K
drwxrwxrwx    1 root     root        4.0K Oct 28 21:35 .
drwxr-xr-x    1 root     root        4.0K Oct 30 10:11 ..
drwxr-xr-x    2 root     root        4.0K Oct 28 21:35 db
-rw-r--r--    1 root     root        1.8K Oct 28 21:25 duc_startup.sh
f5b996fee160:~# ls -alh /tmp
total 8K
drwxrwxrwt    1 root     root        4.0K Oct 28 21:35 .
drwxr-xr-x    1 root     root        4.0K Oct 30 10:11 ..
f5b996fee160:~# duc index /tmp -d ./tmp.db
Error opening: ./tmp.db - Unknown error, contact the author
Unknown error, contact the author
f5b996fee160:~# ls -alh
total 5M
drwxrwxrwx    1 root     root        4.0K Oct 30 10:17 .
drwxr-xr-x    1 root     root        4.0K Oct 30 10:11 ..
drwxr-xr-x    2 root     root        4.0K Oct 28 21:35 db
-rw-r--r--    1 root     root        1.8K Oct 28 21:25 duc_startup.sh
-rw-r--r--    1 root     root        5.0M Oct 30 10:17 tmp.db

and this is the tkrzw output :

f5b996fee160:~# tkrzw_dbm_util inspect --dbm hash /duc/tmp.db
OpenAdvanced failed: NOT_IMPLEMENTED_ERROR: unsupported compression
f5b996fee160:~# tkrzw_dbm_util --version
Tkrzw utilities 1.0.32

To be honest it's a first for me compiling duc and especially it's tkrz dependency. I'vre read your doc and I would actually fit your description of "build the db somewhere and read it somewhere else" which would qualify me more for the sqlite db type, but the benefits of tkrzw seem great to me with faster run time given the many files I'm indexing... So getting tkrz & duc to work on alpine would be great as I'll be using the same alpine image everywhere to both build and display the indexes anyway.

Now to help you with docker to allow you helping me more, I have published the docker image to save you the trouble of actually building it and.
Duc is actually being build inside the container by the instructions in the dockerfile, I could have the dockerfile build the duc binary using ubuntu then run this binary inside an alpine container, I will try to see how I can do that and if it works better, I fear though it has to do with the tkrz lib and if there is a mismatch between what I build with and what I run with this also might be an issue...
To run it yourself and investigate you can :

  1. install docker if you don't have it already : https://docs.docker.com/engine/install/debian/ (they have repos for other distros too)
  2. pull the latest image I have build&published: docker pull toxic0berliner/duc-docker:alpine
  3. instantiate the container with the above image and get a shell inside it: docker run --rm -it --hostname duc-docker toxic0berliner/duc-docker:alpine bash

in there you can do pretty much anything you like and you'll be in the exact same environment as my own development but also production containers 😄

Or you could build it yourself : (click to unfold more details)
  1. clone the repo git clone https://github.com/toxic0berliner/duc-docker.git
  2. install docker if you don't have it already : https://docs.docker.com/engine/install/debian/ (they have repos for other distros too)
  3. in the duc-docker folder, simply run docker build . which will run the Dockerfile and build an image layer by layer. I have updated the Dockerfile so it actually suceeds in building the image, so your build will end like this:
=> exporting to image                                                                                                                                                                 0.0s
=> => exporting layers                                                                                                                                                                0.0s
=> => writing image sha256:dfd62bf88a9d6b9fe7a978328daae0c99cb899fbe4bf8fc5f7d30a99e201650e                                                                                           0.0s

3 warnings found (use docker --debug to expand):
- LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 5)
- LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 64)
- JSONArgsRecommended: JSON arguments recommended for CMD to prevent unintended behavior related to OS signals (line 68

the sha256:df...50e part is what you are looking for: it's the id of the image

  1. instantiate the container with the above sha256:df...50e image and get a shell inside it: docker run --rm -it --hostname duc-docker sha256:dfd62bf88a9d6b9fe7a978328daae0c99cb899fbe4bf8fc5f7d30a99e201650 bash

The Dockerfile in my repo is the build instructions executed by docker when building the image, it is stll a bit of a mess but still quite understandable so if you want to try other ways of doing things feel free to modify it or tell me anything you'd like to try I can attempt it 😄

which will allow you to update the `Dockerfile` allowing you to re-build duc or tkrz in any way you prefer. But of course I'll be happy with any help here and will try to aply modifications to my build script myself too !

@toxic0berliner
Copy link
Author

I have tested building duc binary with debian:12 which works and copy the resulting binary into an alpine image, but all I get is this:

/usr/local/bin # ./duc
sh: ./duc: not found

The binary is there, it's executable, so it's most probably inside the binary that it's looking for something it doesn't find, so I think my best chance is most probably to build duc within alpine itself for it to find whatever it needs.

I did just realize that for debian I use the tkrzw from debian repo which seems to package something from https://dbmx.net/tkrzw/ which does seem the same source as I used but not the same version as it uses 1.0.25-1

Attempting another build with 1.0.25-1 instead... but sadly I still get the same error... build suceeds but indexing says unknown error.
If this can help here is the content of the db for an empty directory:

duc-builder:~# duc index /tmp/empty -d /duc/tmp.db
Error opening: /duc/tmp.db - Unknown error, contact the author
Unknown error, contact the author
duc-builder:~# cat /duc/tmp.db
TkrzwHDB
1P%TkrzwREC
1duc-builder:~

I'm open to any new idea, I'll be looking more closely at the duc build process now trying to find if there are any kind of errors I might have missed...

@l8gravely
Copy link
Collaborator

l8gravely commented Oct 30, 2024 via email

@l8gravely
Copy link
Collaborator

l8gravely commented Oct 30, 2024 via email

@toxic0berliner
Copy link
Author

Wow, that's a lot of answers and a lot of effort you put in trying to help me, so first, thanks so much !

Making one thing crystal clear first : building duc inside docker based on a debian image or an ubuntu image runs flawlessly and I will publish a version of that dockerfile publicly too for others to use freely.
But as you can see the errors you're facing with the duc_startup and apache thing are some wrappers around your cgi output that I forked from another docker port that seems unmaintained, running on an old ubuntu image, and that I wanted to tweak anyways as I'll be building my custom version of duc where I added "dark mode", well, "low-effort" dark mode at least and rsized the graphs to 400px to fit on my phone screen 🤣 and that too is building and running fine inside a debian-based container today already, will publish that too shortly.
Anyway, those errors have nothing to do with duc and you could have picked an earlier layer to run, but hey, no matter, I'm gonna make it easier for you before the end of this message ;)

So don't sweat too much as what I'm now trying to do is make things "cleaner", using alpine is nothing I'm very familiar with but alpine is a very popular distro in docker world mainly because it's very small, and probably cause of the musl thing I'm unfamiliar with but which if I understand makes it lets say more "self contained". Not sure about that last part, but since it's so widely used, on a docker or kubernetes cluster there are some advantages to running several different containers based on the same base images, mostly space-wise, so this is the whole reason why I'm willing to have the alpine version as I never use alpine myself on my systems, I only consume many docker images that others build and are based on alpine mot often. So trying to follow the crowd mainly ;)

Let's try to address all your points now:
The version of alpine I used is in the Dockerfile : FROM: alpine:latest (the latest being implied), which as of now shows to be 3.20.3
ldd might be informative but is a foreign language to me: (not a C developer sorry and only sysadmin by night 😄 )

duc-builder:~# ldd duc
/lib/ld-musl-x86_64.so.1: duc: Not a valid dynamic program

Now, after applying your advice of apk add gcompat and using your ./configure --disable-cairo --disable-ui --disable-x11 --with-db-backend=tkrzw the build suceeds and it looks different at least for ldd :

duc-builder:~# ldd /usr/local/bin/duc
        /lib/ld-musl-x86_64.so.1 (0x7fec84b17000)
        libtkrzw.so.1 => /usr/local/lib/libtkrzw.so.1 (0x7fec84920000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fec84b17000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7fec84687000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fec84663000)

But, the duc index command still fails and tkrz_dbm_util inspect --dbm hash index.db also fails with the same unsupported compression message.

So I see you already have a quite close env to debug this yourself with an alpine VM, still I have tried to make docker easier for you as it seems you have tried to put yourself in my very shoes, so I did update my repo, you can git pull or take a new one, the README now show how to build :
Run docker build -t duc-docker . (the -t flag gives the image a user-defined name aka tag 😄) then docker run --rm -it --hostname duc-docker duc-docker bash to get a shell in the container .
This last command will create a fresh container from the image every time and remove the container and everything you did inside it each time the bash command terminates.

But since I feel you come from the non-docker world, I believe you might prefer to do this instead:
docker run -it --hostname duc-builder --name duc-docker-debug duc-builder bash then exit out of the container.
Now you have an exited(stopped) container named duc-docker-debug that will remain there in the last state you left it until you explicitly delete it with docker rm duc-docker-debug.
An you can enter it at your leisure with docker start -ai duc-docker-debug, once you exit out of it it goes back to being stopped but keeping safe any changes or builds you might have done inside it, they will be right there were you left them when you next run docker start -ai duc-docker-debug

And lastly, to save you all the build time, you can instead use this
docker run -it --hostname duc-builder --name duc-docker-debug toxic0berliner/duc-docker:alpine bash, this will download from the image I built from hub.docker.com and let you run from the very same point as me.
Then exit out of this rist run command and use docker start -ai duc-docker-debug to start&enter the container again as you wish.
Whenever in the future I publish newer images with some progress, you will have to docker pull toxic0berliner/duc-docker:alpine to download the new one, docker rm duc-docker-debug to remove your stopped container and then reuse the docker run then docker start commands.

I have done some cleanup in the Dockerfile, you can have a look I have added the packages you advised (packages were available for gcompat, zstd, zstd-libs) they will be included in your build, even tried zstd-dev and zstd-static on my side with no more luck (but they are not included in the build yet you need to apk add them if you wish them)

Maybe I'm missing something else still..

Looking at the build process (with colors) I've spotted only a few warnings:

src/duc/cmd-histogram.c: In function 'histogram_db':
src/duc/cmd-histogram.c:47:38: warning: format '%d' expects argument of type 'int', but argument 4 has type 'size_t' {aka 'long unsigned int'} [-Wformat=]
   47 |                 printf("%3d %10s %'10d\n",i, pretty, count);
      |                                  ~~~~^               ~~~~~
      |                                      |               |
      |                                      int             size_t {aka long unsigned int}
      |                                  %'10ld
depbase=`echo src/duc/cmd-index.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I.    -Isrc/libduc -Isrc/libduc-graph -Isrc/glad -g -O2 -MT src/duc/cmd-index.o -MD -MP -MF $depbase.Tpo -c -o src/duc/cmd-index.o src/duc/cmd-index.c &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/duc/cmd-info.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I.    -Isrc/libduc -Isrc/libduc-graph -Isrc/glad -g -O2 -MT src/duc/cmd-info.o -MD -MP -MF $depbase.Tpo -c -o src/duc/cmd-info.o src/duc/cmd-info.c &&\
mv -f $depbase.Tpo $depbase.Po
src/duc/cmd-info.c: In function 'info_db':
src/duc/cmd-info.c:56:36: warning: '0' flag ignored with '-' flag in gnu_printf format [-Wformat=]
   56 |                             printf("2^%-02d  %'d\n",i, count);
      |                                    ^~~~~~~~~~~~~~~~
src/duc/cmd-info.c:56:48: warning: format '%d' expects argument of type 'int', but argument 3 has type 'size_t' {aka 'long unsigned int'} [-Wformat=]
   56 |                             printf("2^%-02d  %'d\n",i, count);
      |                                              ~~^       ~~~~~
      |                                                |       |
      |                                                int     size_t {aka long unsigned int}
      |                                              %'ld
depbase=`echo src/duc/cmd-ls.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\

which don't help me further sadly.

It feels like I'm simply missing a dependency again, especially given the message from tkrz_dbm_util, but can't figure out what... Since I feel the only compression is zstd I will attempt a manual build of zstd instead of using the packaged one, just to see if for some reason it's not finding it...

But for tonight I'm done, thanks again for all your help !

@l8gravely
Copy link
Collaborator

l8gravely commented Oct 31, 2024 via email

@l8gravely
Copy link
Collaborator

l8gravely commented Oct 31, 2024 via email

@toxic0berliner
Copy link
Author

Alredy had zstd, added zstd-dev and it turns out for tkrzw 1.0.25 the flag is --enable-zstd but still no luck.
Which version of tkrzw would you advise? 1.0.25 complained about the flag you mentioned... In the dockerfile I'm preparing I am downloading and building tkrzw anyway, I can either pin it to a version of our choosing or pull from their master branch on GitHub.

@l8gravely
Copy link
Collaborator

l8gravely commented Nov 1, 2024 via email

@toxic0berliner
Copy link
Author

toxic0berliner commented Nov 1, 2024 •
edited
Loading

Thanks in advance! And again, don't apologize or rush it there is no emergency and I myself have family over for the weekend and limited availability. I probably won't make it this weekend but my current plan was to find out which flags they use to build tkrzw for the Debian repos and try to use the same since it worked fine building on Debian using their packaged libs

@l8gravely
Copy link
Collaborator

l8gravely commented Nov 1, 2024 via email

@toxic0berliner
Copy link
Author

thanks for the feedback and sorry for the delay.
Seems There is no "liblzma-dev" package for alpine, so I removed it as well from the tkrzw configure options.
I switched to v1.5.0-rc2 but it fails to build claiming libarchive is not found. It was not previously showing as missing, I added it with apk nonetheless but still no luck.
I've become a father very recently so time is becoming limited on my side sadly, I will try to dig into it a bit more when I can.
Thanks again for all the help already !

@l8gravely
Copy link
Collaborator

l8gravely commented Dec 7, 2024 via email

@l8gravely
Copy link
Collaborator

l8gravely commented Dec 7, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@l8gravely @toxic0berliner