native-engine linux binary produced by CI is not centos5 compatible

[kwlzn]

Exception caught: (<type 'exceptions.OSError'>)
  File "/home/kw/pants/src/python/pants/bin/pants_exe.py", line 50, in <module>
    main()
  File "/home/kw/pants/src/python/pants/bin/pants_exe.py", line 44, in main
    PantsRunner(exiter).run()
  File "/home/kw/pants/src/python/pants/bin/pants_runner.py", line 57, in run
    options_bootstrapper=options_bootstrapper)
  File "/home/kw/pants/src/python/pants/bin/pants_runner.py", line 46, in _run
    return LocalPantsRunner(exiter, args, env, options_bootstrapper=options_bootstrapper).run()
  File "/home/kw/pants/src/python/pants/bin/local_pants_runner.py", line 37, in run
    self._run()
  File "/home/kw/pants/src/python/pants/bin/local_pants_runner.py", line 77, in _run
    self._exiter).setup()
  File "/home/kw/pants/src/python/pants/bin/goal_runner.py", line 184, in setup
    goals, context = self._setup_context()
  File "/home/kw/pants/src/python/pants/bin/goal_runner.py", line 157, in _setup_context
    self._daemon_graph_helper
  File "/home/kw/pants/src/python/pants/bin/goal_runner.py", line 97, in _init_graph
    exclude_target_regexps=exclude_target_regexps)
  File "/home/kw/pants/src/python/pants/bin/engine_initializer.py", line 146, in setup_legacy_graph
    scheduler = LocalScheduler(dict(), tasks, project_tree, native)
  File "/home/kw/pants/src/python/pants/engine/scheduler.py", line 81, in __init__
    scheduler = native.lib.scheduler_create(self._context_handle,
  File "/home/kw/pants/src/python/pants/engine/subsystem/native.py", line 411, in lib
    self._lib_field = _FFI.dlopen(binary)
  File "/home/kw/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/cffi/api.py", line 139, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/home/kw/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/cffi/api.py", line 769, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/home/kw/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/cffi/api.py", line 758, in _load_backend_lib
    return backend.load_library(name, flags)

Exception message: cannot load library /home/kw/.cache/pants/bin/native-engine/linux/x86_64/667356a98d647a8ab41ab22bd265f6bd7ce0457e/native-engine: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/kw/.cache/pants/bin/native-engine/linux/x86_64/667356a98d647a8ab41ab22bd265f6bd7ce0457e/native-engine)

$ ldd native-engine
ldd: warning: you do not have execution permission for `./native-engine'
./native-engine: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./native-engine)
./native-engine: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by ./native-engine)
./native-engine: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./native-engine)
  linux-vdso.so.1 =>  (0x00007fff1cb49000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00007f0a43206000)
  libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0a42fea000)
  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0a42ddc000)
  libc.so.6 => /lib64/libc.so.6 (0x00007f0a42a83000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f0a43712000)
$ ls -al /lib64/libc.so.6
lrwxrwxrwx 1 root root 11 Aug 21  2015 /lib64/libc.so.6 -> libc-2.5.so
$ rpm -q glibc
glibc-2.5-123.el5_11.3
glibc-2.5-123.el5_11.3

[stuhood]

Hm. This is potentially because we have a literal libc dependency:

pants/src/rust/engine/Cargo.toml

Lines 9 to 11 in 40c214a

	[dependencies]
	libc = "0.2.0"
	fnv = "1.0.3"

[kwlzn]

seems like dropping the direct libc dep didn't help. according to the docs rust dynamically links against libc always.

the way I see it, our options are:

1. conduct engine builds on a centos5 travis shard (if even possible) to target an older libc, for which newer platforms should have compatibility libs in place.

2. look at statically linking musl (an alternative libc).

3. require that shops supporting older centos versions build their own binaries.

[stuhood]

It looks like it should be possible to install a musl enabled rustup build, and then target x86_64-unknown-linux-musl instead... I'll look at this tonight.

[kwlzn]

seems like dlopen+musl+static binaries don't play nicely together.

[jsirois]

Noting that EOL for CentOS 5 is in March.

[kwlzn]

managed to get a native-engine build on a centos5 box that works correctly on centos5/6/7 - after rebuilding centos5's curl with a modern openssl to fix ssl issues with rustup.

this may be an issue for other shops using pants that also aren't current wrt linux versions, but I don't think there's any strong benefit to continuing to pursue the static linking route given the potential risks and added complexity. worst case, if building their own engine bins becomes a burden for users we could potentially add a centos5 travis shard or something.