POSIX arch: support for POSIX API, non-host libC and AMP

[aescolar]

This is a description of how to add support in the POSIX architecture for: cleaner support for the POSIX API/shim ; (optionally) using a different libC than the host libC; and AMP or multi IC/multiSOC in a single platform/target. Doing this requires some changes to the way Zephyr is built for the POSIX arch, and should just enable these features without drawbacks.
(With AMP I refer to having multiple CPUs in a POSIX arch "board", where each CPU runs its own OS. All CPUs may run Zephyr or they may run different OSes, think nRF53 like).

In very short the idea is that (dynamic library version):

• Each CPU's embedded software is compiled as a separate dynamic library.
• In each of these libraries all symbols are "hidden" from the executable that loads them and other libraries, except for a couple of them (boot_cpu() and awake_cpu() like). This avoids namespacing issues and decouples embedded CPUs' SW between them and from the executable which will run them (see https://gcc.gnu.org/wiki/Visibility ). (This seems also supported by clang)
• Each of these CPU's embedded SW may be compiled either targetting the host libC, or may include its own libC (for ex. one of the libC provided with Zephyr) (which won't be visible outside of that library)
• The native_posix like executable does not contain any more the embedded side of the SW in itself but loads at runtime these libraries.
• HW models for HW which is accesible from all/several CPUs would belong in the overall executable/runner.
• For targets with multiple CPUs, HW models which are meant to be only accessible from 1 CPU would naturally belong compiled in that CPU library. Though technically it could be in the common runner. If several CPUs would have their own separate and repeated HW peripherals, those would be better kept with each CPU library.
• Access from the embedded side to the host OS side (i.e. for the POSIX arch itself, or for drivers backends) could be neatly handled with trampolines provided by the runner (e.g. the embeded side code would call a host_pthread_create() which would be provided by the runner, which would simply call thru to the host OS pthread_create()). Note that the embedded library does technically retain access to all symbols in the runner and the host OS unless it provides its own. The trampoline functionality is only necessary for symbols which are also provided in the embedded side, and in any case clarifies what is what in the embedded side.

Just like before all compilation is done targetting the host architecture, debugging and instrumenting would also work as they do today (library unloading at exit may be disabled to facilitate valgrinds symbol name resolution). Coverage generation with gcov would also work as before.

I had discussed this very briefly with @pfalcon around 1 year ago, but never had time to implement it.
Overall this requires a bit of refactoring in the C side, and a bit more of hacking in the cmake side as effectively it means building the embedded part of each CPU targetted by Zephyr and the runner as separate units linked separetedly.

As a quick proof of this, here a minimal example of how it would be done. You need to use your imagination to map it to Zephyr (you can play with nm/objdump to look at the compile output, and run with gdb):

::::::::::::::
liby/stdio.c
::::::::::::::
/* This libC own putchar */
int putchar(int c){
    return c+10;
}
::::::::::::::
liby/stdio.h
::::::::::::::
int putchar(int c);
::::::::::::::
a.c
::::::::::::::
int a = 10;
static int b = 100;
int c;

#include <stdio.h>

__attribute__ ((visibility ("default"))) int f_a(void){
  c = putchar('!'); /*'!' == 33*/
  printf("hola\n"); /*being naughty calling a non defined C function, which will be resolved to the parent executable printf == the host libC*/
  /* Similarly if putchar was not defined it would have hit the main executable putchar, that is libc's putchar
   */ 
  
  return a + b + c;
}
::::::::::::::
a2.c
::::::::::::::
__attribute__ ((visibility ("default"))) int f_pre_a(void){
  extern int a;
  a = a + 20;
}
::::::::::::::
b.c
::::::::::::::
int a = 0;
static int b = 0;
int c = 13;

__attribute__ ((visibility ("default"))) int f_b(void){
  return a+b+c;
}
::::::::::::::
main.c
::::::::::::::
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

extern int f_pre_a(void);
extern int f_a(void);
extern int f_b(void);

int main(void) {
  //extern int a; a=100; //This line will fail as it a is hidden in both liba.o and libb.o
  f_pre_a();
  fprintf(stdout, "f_a = %i\n", f_a());
  fprintf(stdout, "f_b = %i\n", f_b());
  putchar('\\'); //calling the host libC putchar
  putchar('\n');
}
::::::::::::::
compile.sh
::::::::::::::
#! /bin/bash

set -e

COVERAGE_COMP=--coverage
COVERAGE_LINK="-shared-libgcc --coverage"

gcc -c liby/stdio.c -o liby/stdio.o -fPIC -fvisibility=hidden -g $COVERAGE_COMP
gcc -c a.c -o a.o -fPIC -fvisibility=hidden -g -nostdinc -Iliby/ -ffreestanding -fno-builtin $COVERAGE_COMP
#-fno-builtin to avoid getting confused with gcc replacing some of the C library functions
# with its own versions or replacing the calls with more optimal ones
gcc -c a2.c -o a2.o -fPIC -fvisibility=hidden -g  $COVERAGE_COMP
gcc -c b.c -o b.o -fPIC -fvisibility=hidden -g  $COVERAGE_COMP
gcc -fvisibility=hidden -Bsymbolic -shared a.o a2.o -fPIC -o liba.so -nostdlib liby/stdio.o $COVERAGE_COMP
gcc -shared -fvisibility=hidden b.o -fPIC -o libb.so $COVERAGE_COMP

gcc main.c liba.so libb.so -o hidden.exe -g -Wl,-rpath,./ $COVERAGE_LINK

You can imagine this example as "a*.c" (liba.so) being the 1st CPU embedded SW, compiled with its own libC (liby/), and "b.c" (libb.so) the 2nd CPU embedded SW, with main.c being the overall runner. You can see that even though liba and libb have a few symbols with the same names they are nicely kept separated. And that liba indeed calls into its own liby/ putchar() instead of the host libC:

$ ./hidden.exe 
hola
f_a = 173
f_b = 13
\

If you play a bit with the code or the build script, you can for example change it so liba builds with the host libC so the output would be instead:

$ ./hidden.exe 
!hola
f_a = 163
f_b = 13
\

---

Or alternatively we can do it with static libraries. Where, for each CPU, we pre-link (incremental relocatable link) all object files that would contain that CPU SW into one big object file. And then we "localize" all hidden symbols (all which were not explicitly set to "default" visibility in the C code) with objcopy (we make them "static") before linking.

::::::::::::::
compile_static.sh
::::::::::::::
#! /bin/bash

set -e

COVERAGE_COMP=--coverage
COVERAGE_LINK=--coverage

gcc -c liby/stdio.c -o liby/stdio.o -fPIC -fvisibility=hidden -g $COVERAGE_COMP
gcc -c a.c -o a.o -fPIC -fvisibility=hidden -g -nostdinc -Iliby/ -ffreestanding -fno-builtin $COVERAGE_COMP
#-fno-builtin to avoid getting confused with gcc replacing some of the C library functions
#with its own versions or replacing the calls with more optimal ones
gcc -c a2.c -o a2.o -fPIC -fvisibility=hidden -g  $COVERAGE_COMP
gcc -c b.c -o b.o -fPIC -fvisibility=hidden -g  $COVERAGE_COMP

ld -i a.o a2.o liby/stdio.o -o liba.pre.o
ld -i b.o -o libb.pre.o

objcopy --localize-hidden liba.pre.o liba.o
objcopy --localize-hidden libb.pre.o libb.o

gcc main.c liba.o libb.o -o hidden_static.exe -g $COVERAGE_LINK

---

All this is effectively solving namespace collisions in C, where we build one program executable which contains different components (libraries) which reuse the same names for different symbols.
This can be because we have an extra C library, or because we have several instances of the Zephyr OS each with their own app, or because we are exposing the POSIX API on top of Zephyr while at the same time we are using the host POSIX API.

[aescolar]

Related to #13054 & #6044

[pfalcon]

embedded libC

Can it please be elaborated what it's meant by this?

[aescolar]

embedded libC

Can it please be elaborated what it's meant by this?

(I changed a bit the text) I meant, in general, any other libC than the default host libC, which could be compiled for the host architecture.

[pfalcon]

(I changed a bit the text) I meant, in general, any other libC than the default host libC, which could be compiled for the host architecture.

Thanks for clarification.

And to clarify another point, by AMP you mean generic asymmetric multi-processing, not specifically support for integration with a library like OpenAMP (which is otherwise supported, or worked on, for integration with Zephyr)?

[aescolar]

And to clarify another point, by AMP you mean generic asymmetric multi-processing

Yes. So today native_posix (or the nrf52_bsim) emulates 1 processor, with 1 OS (running 1 thread at a time). So this would be support for more than 1 processor (each single threaded still though) running each their own OS (or baremetal).
Say, for example, like a nRF5340, where one processor could be running the BT host and application, and the other processor could be running the BT controller.

[aescolar]

Note that Zephyr's linker script symbol definition and symbol reordering is supported (both building as dynamic and as static libraries). Each resulting library containing an embedded CPU SW would have its own set of linker defined symbols hidden from the other CPU libraries.
SORTABLE_INT in the code below would be a demo of one of the many symbols we short around in Zephyr; Where both the CPU "a" and "b" have them with overlapping names.

::::::::::::::
a2.c
::::::::::::::
#include "common_header.h"

__attribute__ ((visibility ("default"))) int f_pre_a(void){
  extern int a;
  a=a+20;
}

SORTABLE_INT(c, 3, 3);
::::::::::::::
a.c
::::::::::::::
#include <stdio.h>
#include "common_header.h"

int a = 10;
static int b = 100;
int c;

SORTABLE_INT(a, 1, 1);
SORTABLE_INT(d, 4, 4);
SORTABLE_INT(b, 2, 2);

__attribute__ ((visibility ("default"))) int f_a(void){
  c = 0;
  
  print_linker_symbols();
  c = putchar('!'); /*'!' == 33*/
  printf("hola\n"); /*being naughty calling a non defined C function, which will be resolved to the parent executable printf == the host libC*/
  /* Similarly if putchar was not defined it would have hit the host libC one
   */ 
  
  return a + b + c;
}
::::::::::::::
b.c
::::::::::::::
#include "common_header.h"

int a = 0;
static int b = 0;
int c = 13;

SORTABLE_INT(d,  5, 2);
SORTABLE_INT(a,  9, 1);
SORTABLE_INT(b,  1, 4);
SORTABLE_INT(c,  2, 3);

__attribute__ ((visibility ("default"))) int f_b(void){
  print_linker_symbols();
  return a+b+c;
}
::::::::::::::
common_module.c
::::::::::::::
#include <stdio.h>

void print_linker_symbols(void){
	extern int __sortable_int_start[];
	extern int __sortable_int_end[];

	int *i_ptr;
	int i;

	printf("__sortable_int_start = %p\n", __sortable_int_start);
	printf("__sortable_int_end = %p\n"  , __sortable_int_end);
	for (i = 0, i_ptr = __sortable_int_start; i_ptr < __sortable_int_end; i++ , i_ptr++) {
		printf("%i: %i\n",i ,*i_ptr);
	}
}
::::::::::::::
main.c
::::::::::::::
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

extern int f_pre_a(void);
extern int f_a(void);
extern int f_b(void);

int main(void) {
  //extern int a; a=100; //This line will fail as it a is hidden in both liba.o and libb.o
  f_pre_a();
  fprintf(stdout, "f_a = %i\n", f_a());
  fprintf(stdout, "f_b = %i\n", f_b());
  putchar('\\'); //calling the host libC putchar
  putchar('\n');
}
::::::::::::::
common_header.h
::::::::::::::
#define _DO_CONCAT(x, y) x ## y
#define _CONCAT(x, y) _DO_CONCAT(x, y)

#define Z_STRINGIFY(x) #x
#define STRINGIFY(s) Z_STRINGIFY(s)

#define SORTABLE_INT(name, level, value)	\
	static int _CONCAT(__sortable_int_, name) __attribute__((__used__)) \
	__attribute__((__section__(".sortable_int_" STRINGIFY(level))))\
	= value

void print_linker_symbols(void);
::::::::::::::
linker.script
::::::::::::::
SECTIONS
 {
	sortable_ints :
	{
		__sortable_int_start = .;
		KEEP(*(SORT(.sortable_int_[0-9])));
		__sortable_int_end = .;
	}
 } INSERT AFTER .data;
::::::::::::::
linker_symbols_to_localize
::::::::::::::
__sortable_int_start
__sortable_int_end
::::::::::::::
linker.version.script
::::::::::::::
{ local: __sortable_int_*; };
::::::::::::::
compile_dynamic.sh
::::::::::::::
#! /bin/bash

COVERAGE_COMP=--coverage
COVERAGE_LINK="-shared-libgcc --coverage"
COMPILE_FLAGS="-fPIC -fvisibility=hidden -g $COVERAGE_COMP"

gcc -c common_module.c -o common.o $COMPILE_FLAGS

gcc -c liby/stdio.c -o liby/stdio.o $COMPILE_FLAGS
gcc -c a.c -o a.o $COMPILE_FLAGS -nostdinc -Iliby/ -ffreestanding -fno-builtin
gcc -c a2.c -o a2.o $COMPILE_FLAGS
gcc -c b.c -o b.o $COMPILE_FLAGS

gcc -shared -fvisibility=hidden -Bsymbolic a.o a2.o common.o -fPIC -o liba.so -nostdlib liby/stdio.o $COVERAGE_COMP  -T linker.script -Wl,--version-script=l
inker.version.script
gcc -shared -fvisibility=hidden b.o  common.o -fPIC -o libb.so $COVERAGE_COMP -T linker.script -Wl,--version-script=linker.version.script

gcc main.c liba.so libb.so -o hidden_dynamic.exe -g -Wl,-rpath,./ $COVERAGE_LINK
::::::::::::::
compile_static.sh
::::::::::::::
#! /bin/bash

COVERAGE_COMP=--coverage
COVERAGE_LINK=--coverage
C_FLAGS="-g $COVERAGE_COMP"

gcc -c common_module.c -o common.o -fvisibility=hidden $C_FLAGS
gcc -c liby/stdio.c -o liby/stdio.o -fvisibility=hidden $C_FLAGS
gcc -c a.c -o a.o -fvisibility=hidden -nostdinc -Iliby/ -ffreestanding -fno-builtin $C_FLAGS
gcc -c a2.c -o a2.o -fvisibility=hidden $C_FLAGS
gcc -c b.c -o b.o -fvisibility=hidden $C_FLAGS

ld -i a.o a2.o common.o liby/stdio.o -o liba.pre.o -T linker.script
ld -i b.o common.o -o libb.pre.o -T linker.script

objcopy --localize-hidden liba.pre.o liba.o --localize-symbols=linker_symbols_to_localize
objcopy --localize-hidden libb.pre.o libb.o --localize-symbols=linker_symbols_to_localize

gcc main.c liba.o libb.o -o hidden_static.exe $C_FLAGS

[cfriedt]

Visibility:

• it is non-portable (i.e. visibility works with elf-targets only)

AMP:

• are different OSes being run or just different applications?
• can't this be achieved just running separate processes?

Non-host libc:

• in general, there are no problems running a non-host libc
• 1 exception I have seen is calling fcntl() from within uart_native_posix.c
• I worked around this with dlopen() / dlsym() (both POSIX compliant and portable)

[aescolar]

@cfriedt
About visibility, even if we wanted to support other than elf (which we don't today), you can see there is two proposals. The one with static linking does not rely on hidden symbols at runtime, but local symbols to each library at link time.
About your AMP question (note that it is a separate topic. This proposed change enables the use case to some degree, but would require further changes), in general it refers to the use case where you have 2 separate microcontrollers in the same IC, where each microcontroller is running its own OS (and application) but still with some degree of a shared memory space and peripherals, as opposed to SMP where two quite coupled microcontrollers run the same kernel with a common kernel state.
"can't this be achieved just running separate processes?" yes, with some significant drawbacks when modeling some architectures. In any case, as mentioned this proposal just enables it as a welcome side-effect.
About the last part of your comment, I would prefer if we discussed your PR in your PR itself instead of here to avoid confusing future readers.