Improve dynamic loading in a static executable

[iapaddler]

The goal for executables is to build once and run in all distros. There are a couple of ways to do this. The preferred approach is the use of a static executable. We build the CLI as a Go static if at all possible. This enables the use of the CLI executable in GNU and Musl based distros.

A static executable becomes problematic in two scenarios.

1. When we scope and run a corresponding Go static exec we load libscope.so, locate the libscope loader, load the corresponding Go static exec, hook functions as needed and start the corresponding Go static exec.
2. When attaching to a remote process, we locate the dlopen function in the remote process by calculating the address based on the libc of the current process.

The current approach for scenarios 1 and 2 make use of the dynamic loader in the form of dlopen, dlsym and dl_iterate_pheader. None of these are available within the context of a static executable. There are several options including not using any dynamic linker functionality. Without going into detail about the ramifications of alternatives, this issue is intended to define the behavior associated with attempting to enable dynamic linker support in a static exec.

A GNU libc utilizes linkage to an external library for implementation of dynamic loader functionality. In short, it requires that all of libc and dependencies must be included. Various mechanisms are theoretically possible. This has an interesing discussion of possibilities.

Due to the external dependencies and related complications involved in attempting this with a GNU libc we opted to look into how this could be enabled with Musl libc. The libc code is currently available in libscope so this represented a potential extension of current capabilities.

It turns out that there are two implementations of dynamic loader code in Musl libc. One for static linkage and one for dynamic. The dynamic linker code used for static linkages, and included in a libc.a, are stubs with no dynamic linker functionality. The first step involved including the dynamic ldso code in a static build.

Once a functional dynamic loader is included, we proceed to attempt to use dlopen. It becomes abundantly clear that the dynamic loader environment needs to be initialized before it can be utilized.

The dynamic loader functionality is not initialized in any libc init. It turns out that initialization is performed when ldso is executed. Stage two of ldso performs all initialization of the dynamic loader. This is the ld.so defined in the .interp section of an Elf executable. This is normally performed by ld.so as a result of an execve syscall of a dynamic exec. Given that we are attempting the use of a static exec, this is not performed from the exec syscall.

It is possible to perform the initialization independent of lds.so. The entry point, either the ldso start or the start of the stage 2 loader, can be accessed directly. The required parameters include a pointer to the Elf image of the current executable and a pointer to the stack as provided to the main function. The Elf image is readily accessible in /proc/self/exe. The stack provided to the original main function is available from the original argv parameter. When we open and read the first part of /proc/self/exe and reference argv -1 we provide the required params.

Dynamic loader initialization proceeds up to the point where vectors are created from entries in the .dynamic section of the current Elf image. Of course, there is no dynamic program header in a static exec. Therefore, the init fails at that point. It is theoretically possible to create a set of "null" dynamic header entries. It might be possible to null a set of manually created dynamic entries by ensuring that the DT_NEEDED tag is not set. That has not been validated. Just a thought.

Taking the next step would appear to require code changes in Musl libc. We are trying not to make changes like this for numerous reasons. A thought, also not validated, indicates that the ldaddr function used in dynlink.c could be modified to return manually created null dynamic headers.