zlib v1.2.12 fails to build shared library

[mtl1979]

When using configure, compiler is not correctly detected as gcc and -fPIC is not added to SFLAGS. This causes relocation issues when linking under Linux.

[thesamesam]

This is #608 which has a patch in develop (05796d3).

[mtl1979]

I tried tip of develop, but it still fails.

[thesamesam]

Please share the full log of the failure you're seeing.

[mtl1979]

Using ar
Checking for x86_64-pc-linux-gnu-gcc...
Checking for shared library support...
Building shared library libz.so.1.2.12.1-motley with cc.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
+ make -j2
cc -ggdb -D_LARGEFILE64_SOURCE=1 -I. -Isrc.d/ -c -o example.o src.d/test/example.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o adler32.o src.d/adler32.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o crc32.o src.d/crc32.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o deflate.o src.d/deflate.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o infback.o src.d/infback.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o inffast.o src.d/inffast.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o inflate.o src.d/inflate.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o inftrees.o src.d/inftrees.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o trees.o src.d/trees.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o zutil.o src.d/zutil.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o compress.o src.d/compress.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o uncompr.o src.d/uncompr.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o gzclose.o src.d/gzclose.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o gzlib.o src.d/gzlib.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o gzread.o src.d/gzread.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -c -o gzwrite.o src.d/gzwrite.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -I. -Isrc.d/ -c -o minigzip.o src.d/test/minigzip.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/adler32.o src.d/adler32.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/crc32.o src.d/crc32.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/deflate.o src.d/deflate.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/infback.o src.d/infback.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/inffast.o src.d/inffast.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/inflate.o src.d/inflate.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/inftrees.o src.d/inftrees.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/trees.o src.d/trees.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/zutil.o src.d/zutil.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/compress.o src.d/compress.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/uncompr.o src.d/uncompr.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/gzclose.o src.d/gzclose.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/gzlib.o src.d/gzlib.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/gzread.o src.d/gzread.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -include zconf.h -DPIC -c -o objs/gzwrite.o src.d/gzwrite.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -I. -Isrc.d/ -D_FILE_OFFSET_BITS=64 -c -o example64.o src.d/test/example.c
cc -ggdb -D_LARGEFILE64_SOURCE=1 -I. -Isrc.d/ -D_FILE_OFFSET_BITS=64 -c -o minigzip64.o src.d/test/minigzip.c
ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o
cc -ggdb -D_LARGEFILE64_SOURCE=1 -o example example.o -L. libz.a
cc -ggdb -D_LARGEFILE64_SOURCE=1 -o minigzip minigzip.o -L. libz.a
cc -shared -ggdb -D_LARGEFILE64_SOURCE=1 -o libz.so.1.2.12.1-motley adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo  -lc
cc -ggdb -D_LARGEFILE64_SOURCE=1 -o example64 example64.o -L. libz.a
/usr/bin/ld: trees.lo: warning: relocation against `_length_code' in read-only section `.text'
/usr/bin/ld: deflate.lo: relocation R_X86_64_PC32 against symbol `z_errmsg' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make: *** [Makefile:282: libz.so.1.2.12.1-motley] Error 1
make: *** Waiting for unfinished jobs....

[mtl1979]

I checked all the variables inside configure, and uname is not set correctly (it's set to gnu, not Linux) when CHOST is set to what dpkg-architecture -qDEB_HOST_MULTIARCH returns... This breaks detecting full name of gcc binary and wrong parameters are passed to linker.

For some weird reason, zlib 1.2.11 could manage to find gcc even though full name of the binary wasn't detected.

[rubyFeedback]

I am running into similar issues and I had to downgrade back to 1.2.11; many other programs failed to compile
when I moved to 1.2.12.

For instance harfbuzz just now:

c++  -o util/hb-view util/hb-view.p/hb-view.cc.o -Wl,--as-needed -Wl,--no-undefined -fPIC '-Wl,-rpath,$ORIGIN/../src' -Wl,-rpath-link,/home/Temp/rbt/harfbuzz-4.2.0/BUILD_DIRECTORY/src -Wl,--start-group src/libharfbuzz.so.0.40200.0 /usr/lib64/libfreetype.so /usr/lib/libcairo.so /usr/lib/libglib-2.0.so -Wl,--end-group
/usr/bin/ld: /usr/lib/libpng16.so.16: undefined reference to `inflateReset2@ZLIB_1.2.3.4'
/usr/bin/ld: /usr/lib/libpng16.so.16: undefined reference to `inflateValidate@ZLIB_1.2.9'
/usr/bin/ld: /usr/lib/libxml2.so.2: undefined reference to `gzopen64@ZLIB_1.2.3.3'
/usr/bin/ld: /usr/lib/libxml2.so.2: undefined reference to `gzdirect@ZLIB_1.2.2.3'

[mtl1979]

When it doesn't detect the system is Linux, it doesn't set soname or use version map, so the exported symbols don't include symbol versions.

[henkela]

Switching to CMake instead of configure / make at least solves the issue with the relocation issue on rh8.5.

[mtl1979]

Switching to CMake instead of configure / make at least solves the issue with the relocation issue on rh8.5.

When cross-compiling, using configure is a lot easier as no toolchain file is needed; just set CHOST before running configure. Paths and filenames can differ depending on the operating system version. We use a build matrix with over 10 different compilers and 3 operating systems.

[tbeu]

I am running into similar issues and I had to downgrade back to 1.2.11

Same here.

[vtorri]

have you tried to use git bisect to find the commit which introduced thebug ? (https://www.git-tower.com/learn/git/faq/git-bisect/)

[mtl1979]

have you tried to use git bisect to find the commit which introduced thebug ? (https://www.git-tower.com/learn/git/faq/git-bisect/)

Git blame would be a lot faster as I know what line the bug is...

Commit hash is 7301420 ;)

[tbeu]

7301420 got introduced for v1.2.4.5 more than 10 years ago. I thought the reported issue is a more recent regression of v1.2.12.

[mtl1979]

7301420 got introduced for v1.2.4.5 more than 10 years ago. I thought the reported issue is a more recent regression of v1.2.12.

That's the first commit that fails, hence introducing the bug... It had workaround that worked up to v1.2.11, but not in v1.2.12 anymore... I can't track workarounds when going commits backwards as that would require modifying commits after fetching the commit in script and that is not possible in non-interactive shell.

[ljavorsk]

This issue has also affected the Fedora distribution and thus we cannot rebase zlib to its newest 1.2.12 version.

Did you find any workarounds for this yet? @mtl1979

[mtl1979]

This issue has also affected the Fedora distribution and thus we cannot rebase zlib to its newest 1.2.12 version.

Did you find any workarounds for this yet? @mtl1979

Only workaround I know is to use incorrect CHOST (already canonicalized) and then manually specify CC using full path to compiler... This should work with the tip of develop branch...

[ljavorsk]

@mtl1979 This PR solves it for me: #607

[mtl1979]

@mtl1979 This PR solves it for me: #607

That PR fixes the issue when not cross-compiling... But when cross-compiling, that PR is not enough like I mentioned above.

[madler]

Git blame would be a lot faster as I know what line the bug is...

What line is the bug and what is the bug? (That commit has all the changes for that version, so it doesn't help find your line.)

[mtl1979]

@madler https://github.com/madler/zlib/blob/develop/configure#L35 ... The regex is too simplified... Canonicalizing is the cleanest solution, but it requires copying over the canonicalizing script.

See https://github.com/zlib-ng/zlib-ng/blob/develop/configure#L25

[madler]

What was your workaround that 1.2.12 broke?

[mtl1979]

@madler We did normalize the triplet before calling zlib's configure, but that didn't work anymore as the prefix of the compiler binary isn't same as the normalized CHOST. That is why zlib-ng only use the normalized CHOST to deduce the operating system. Not all Ubuntu versions have symbolic link for all relevant binaries from the normalized triplet to original triplet.

[madler]

So what changed in 1.2.12 that made it not work anymore?

[mtl1979]

@madler Obviously handling of empty CC variable... Like I said earlier, part of that was fixed in 1.2.12.1, but the fix was incomplete and only worked when not cross-compiling. To detect operating system as Linux and finding correct gcc, it needs to use two different triplets as I already mentioned... I tried to just fix the sed regular expression, but that would only fix it for recent Ubuntu versions, not all other operating systems it doesn't detect correctly.

[madler]

I assume by fixed, you are referring to commit 05796d3 . Please explain, with an example, what about the fix is incomplete, and how the result differs from 1.2.11. What is a "triplet"? How were there two different "triplets" in 1.2.11?

[mtl1979]

@madler Triplet is target for a compiler, for example aarch64-linux-gnu, canonicalized version of that same triplet is aarch64-unknown-linux-gnu. You can test the sed regular expression with both of them to see if it parses them correctly.

[madler]

As you pointed out, that regular expression hasn't changed since 2011. What did 12.2.11 do that it no longer does, even with commit 05796d3 ?

[mtl1979]

@madler I've already explained the core issue... There is two alternatives, either it uses wrong binaries, targeting host operating system instead of cross-compile target, or it fails during linking shared library due to incorrect linker flags. Specifying CC manually doesn't change the filename for other binaries like it does when using CHOST. It might be combination of a long-standing bug/regression and compiler packagers dropping symbolic links from canonicalized triplet to non-canonicalized triplet.

Obviously using non-canonicalized triplet is the right/preferred choice as then the triplet matches prefix used by the toolchain.

[madler]

Yes, I know the end result. I'm trying to understand how. Let me ask more explicit questions. In your example, configure is trying to run the executable x86_64-pc-linux-gnu-gcc. Does that executable exist? If so, what does it output with the -v option? If it doesn't exist, what are you expecting to happen?

[mtl1979]

Yes, I know the end result. I'm trying to understand how. Let me ask more explicit questions. In your example, configure is trying to run the executable x86_64-pc-linux-gnu-gcc. Does that executable exist? If so, what does it output with the -v option? If it doesn't exist, what are you expecting to happen?

It doesn't exist. Closest existing binary is /usr/bin/x86_64-linux-gnu-gcc.

To answer what should happen, is that it should ignore -unknown-, -pc- etc.

[mtl1979]

This is all the targets for gcc I use:

mika@DESKTOP-AUKPB81:/mnt/c/build/git/zlib-ng$ ls /usr/bin/*-gcc
/usr/bin/aarch64-linux-gnu-gcc  /usr/bin/arm-linux-gnueabihf-gcc  /usr/bin/c99-gcc               /usr/bin/powerpc-linux-gnu-gcc    /usr/bin/powerpc64le-linux-gnu-gcc  /usr/bin/x86_64-w64-mingw32-gcc
/usr/bin/arm-linux-gnueabi-gcc  /usr/bin/c89-gcc                  /usr/bin/i686-w64-mingw32-gcc  /usr/bin/powerpc64-linux-gnu-gcc  /usr/bin/x86_64-linux-gnu-gcc

[madler]

Ok. By "ignore", do you mean "remove"? Are you expecting configure to convert x86_64-pc-linux-gnu to x86_64-linux-gnu?

You must be giving CHOST=x86_64-pc-linux-gnu to configure. Where is that coming from? Shouldn't it be CHOST=x86_64-linux-gnu?

[mtl1979]

Ok. By "ignore", do you mean "remove"? Are you expecting configure to convert x86_64-pc-linux-gnu to x86_64-linux-gnu?

You must be giving CHOST=x86_64-pc-linux-gnu to configure. Where is that coming from? Shouldn't it be CHOST=x86_64-linux-gnu?

No... I mean ignore that it doesn't exist in CHOST... Pretend that it is there when it isn't passed in CHOST to the sed regular expression... Like I said earlier, CHOST should/will/does match the prefix of gcc binary shown in my previous post. Debian/Ubuntu doesn't report it as it's redundant and serve no purpose.

[madler]

I'm going to try something different. I am going to remove the sed command completely, set uname to $CHOST in its full glory, and search for -linux somewhere in uname. Then the use of something like CHOST=x86_64-linux-gnu should work, since it will see that it's Linux, and the command x86_64-linux-gnu-gcc will exist and run. Would that work for you?

Please try the current develop branch, with 013c0ca .

[mtl1979]

I'm going to try something different. I am going to remove the sed command completely, set uname to $CHOST in its full glory, and search for -linux somewhere in uname. Then the use of something like CHOST=x86_64-linux-gnu should work, since it will see that it's Linux, and the command x86_64-linux-gnu-gcc will exist and run. Would that work for you?

Please try the current develop branch, with 013c0ca .

Linker command-line looks correct and it gives no error when I try abicheck on zlib-ng repository without altering CHOST before running zlib's configure.

[madler]

So is this issue resolved?

[mtl1979]

So is this issue resolved?

Works for me ;)

[madler]

Thank you.