native: fix C++ for FreeBSD #14631

miri64 · 2020-07-28T09:37:16Z

Contribution description

Cross-compiling C++ for 32-bit on a 64-bit FreeBSD is not as easy as installing the respective lib via package management as it is in most Linux distributions. This configures the build system to honor the libstdcpp feature for 64-bit FreeBSD systems.

Testing procedure

Merge #14458, install vagrant on your system by following the README provided there and do

cd dist/tools/vagrant/freebsd
vagrant plugin install vagrant-disksize
vagrant up     # ignore potential error messages
vagrant ssh
cd RIOT
make -C tests/cpp_ctor

Without this PR, the test won't link, with it it will. You can also test other C++ tests. Most of them will skip compilation because the libstdcpp feature is required, but those who do not will successfully link.

Compiling with DISABLE_LIBSTDCPP=0 in the environment should still fail linking.

Issues/PRs references

Found in #14503, requires #14458 for testing but not to be merged.

cpu/native/Makefile.dep

maribu · 2020-07-28T11:03:18Z

Looks good to me. There is no vagrant package for Alpine (yet), so I cannot test this easily.

Maybe someone else can test this? On the other hand, vagrant should make sure that you tested in a very reproducible environment, so that the "works-on-my-machine-syndrome" shouldn't pop up here anyway. Thus, I wouldn't mind to ACK trusting your test result.

Feel free to squash.

miri64 · 2020-07-28T11:05:00Z

Squashed

miri64 · 2020-07-28T11:11:00Z

Maybe @fjmolinas can also have a look regarding the saneness of build system integration.

miri64 · 2020-07-28T13:35:28Z

Squashed

At least I thought I did Oo

fjmolinas

I don't like the use of DISABLE_LIBSTDCPP, it replicates the same as FEATURES_USED. If its broken then the feature should not be provided IMO, since the original PR introducing it was by @maribu, did you have a use case for this?

Anyway my comments are all about removing that variable.

fjmolinas · 2020-07-28T13:59:42Z

cpu/native/Makefile.features

-ifeq (FreeBSD,$(OS))
-  DISABLE_LIBSTDCPP ?= 1
-endif


# libstdc++ on FreeBSD is broken (does not work with -m32) ifneq (FreeBSD amd64, $(OS) $(OS_ARCH)) FEATURES_PROVIDED += libstdcpp endif

I can't make multiline comments but L8-12 would be removed.

fjmolinas · 2020-07-28T14:12:38Z

cpu/native/Makefile.dep

+ifeq (FreeBSD,$(OS))
+  ifeq ($(OS_ARCH),amd64)
+    DISABLE_LIBSTDCPP ?= 1
+  endif
+  ifeq ($(DISABLE_LIBSTDCPP),1)
+    # static C++ constructors need guards for thread safe initialization
+    ifneq (,$(filter cpp,$(FEATURES_USED)))
+      USEMODULE += cxx_ctor_guards
+    endif
+  endif
+endif
+


Suggested change

ifeq (FreeBSD,$(OS))

ifeq ($(OS_ARCH),amd64)

DISABLE_LIBSTDCPP ?= 1

endif

ifeq ($(DISABLE_LIBSTDCPP),1)

# static C++ constructors need guards for thread safe initialization

ifneq (,$(filter cpp,$(FEATURES_USED)))

USEMODULE += cxx_ctor_guards

endif

endif

endif

ifeq (,$(filter libstdcpp,$(FEATURES_USED)))

# static C++ constructors need guards for thread safe initialization

ifneq (,$(filter cpp,$(FEATURES_USED)))

USEMODULE += cxx_ctor_guards

endif

endif

Is this in all cases or only for FreeBSD? If its only for FreeBSD then the first ifdef should be kept as well.

It's only for FreeBSD 64-bit, yes. Otherwise this could be moved to a more global Makefile.dep, I believe.

And why so? Only arch using them so far is atmega? Why did you add it for FreeBSD?

It provides the guards needed to call constructors during the initialization of statically allocated class instances. The C++ compiler will generate call to those utility functions. Usually those are implemented by the toolchain (e.g. as part of libstdc++). On FreeBSD the default libstdc++ is sadly not compatible with -m32, so undefined references to those implementation will occur while linking. The module is a compatible drop-in replacement for platforms without (working) libstdc++, such as ATmega or FreeBSD.

But true... with that knowledge having the condition (cpp && !libstdcpp) in Makefile.dep to include cxx_ctor_guards makes little sense :-/.

Then I keep it here. If I remove the dependency on DISABLE_LIBSTDCPP I make dependent on @fjmolinas answer to #14631 (comment)

Btw: We might want to drop cpu/esp32/cxx in favor of the platform independent cxa_ctor_guards.

Indeed, it's just a leftover required by former ESP32 toolchain versions that used POSIX threads. Although cxa_ctor_guards were built-in, they had to be replaced since they used the pthread_once function for singleton objects initialization where the parameter once was of incompatible type with that provided by RIOT's pthread module.

We are now using a self-compiled ESP32 toolchain version that no longer uses POSIX threads for thread-safety. That is, the toolchain does no longer guarantee thread-safety, as is the case with other platforms.

I think the ESP32 implementation is not correct. According to the documentation it is not thread safe, but thread safety was the very point those guards are there.

Aborting if a thread tries to access the variable during its initialization by another thread was the only way to get it to work at that time 😟

The cxx_ctor_guards just use an rmutex for locking. That way the same thread is able to call the ctor of classes that call the ctor of other classes within their ctor and is thread safe. If those guards are indeed still needed, the module cxx_ctor_guards should be a drop in replacement for every platform. (It only uses a single byte of whatever is passed as __guard, so no matter how small this type is defined on a specific platform, it will still work.)

The cxx_ctor_guards just use an rmutex for locking.

Ah, there is a new module cxx_ctor_guards. Sorry, I didn't notice that because I've been pretty busy the last few weeks rebasing and providing the new GPIO API after a half year break.

I don't remember exactly, but using a mutex for the ESP32 cxx_ctor_guards led to deadlocks. Anyway, we can drop it now. I will provide a PR.

boards/native/Makefile.include

fjmolinas · 2020-07-28T14:18:22Z

BTW my change suggestions are untested :)

miri64 · 2020-07-28T15:54:46Z

I don't like the use of DISABLE_LIBSTDCPP, it replicates the same as FEATURES_USED. If its broken then the feature should not be provided IMO, since the original PR introducing it was by @maribu, did you have a use case for this?

Anyway my comments are all about removing that variable.

I am not sure what you mean by that. From what I can tell, your suggestions don't allow for FreeBSD users to use libstdcpp if they e.g. cross-compile the lib themselves for 32-bit.

miri64 · 2020-07-28T16:13:13Z

Mhhhh I just noticed, while this compiles, many tests (not just C++ tests) are segfaulting now with the current version of this PR. I can't debug this however due to

RIOT/Makefile.include

Lines 650 to 658 in 0e3aa2f

    
           # Check if setsid command is available on the system for debug target 
        
           # This is not the case on MacOSX, so it must be built on the fly 
        
           ifneq (,$(filter debug, $(MAKECMDGOALS))) 
        
             ifneq (0,$(shell which setsid 2>&1 > /dev/null ; echo $$?)) 
        
               SETSID = $(RIOTTOOLS)/setsid/setsid 
        
               $(call target-export-variables,debug,$(SETSID)) 
        
               DEBUGDEPS += $(SETSID) 
        
             endif 
        
           endif

and

make -C tests/cpp_ctors/ debug
make: Entering directory '/home/vagrant/RIOT/tests/cpp_ctors'
[INFO] setsid binary not found - building it from source now
[INFO] compiling setsid from source now
gmake BINDIR=/home/vagrant/RIOT/dist/tools/setsid/bin -C /home/vagrant/RIOT/dist/tools/setsid/bin
gcc -o setsid setsid.c
setsid.c: In function 'main':
setsid.c:104:21: warning: implicit declaration of function 'wait' [-Wimplicit-function-declaration]
  104 |                 if (wait(&status) != pid) {
      |                     ^~~~
setsid.c:107:21: warning: implicit declaration of function 'WIFEXITED' [-Wimplicit-function-declaration]
  107 |                 if (WIFEXITED(status)) {
      |                     ^~~~~~~~~
setsid.c:108:28: warning: implicit declaration of function 'WEXITSTATUS' [-Wimplicit-function-declaration]
  108 |                     return WEXITSTATUS(status);
      |                            ^~~~~~~~~~~
/usr/local/bin/ld: /tmp//ccXmDAVU.o: in function `main':
setsid.c:(.text+0x283): undefined reference to `WIFEXITED'
/usr/local/bin/ld: setsid.c:(.text+0x296): undefined reference to `WEXITSTATUS'
collect2: error: ld returned 1 exit status
gmake[2]: *** [Makefile:2: setsid] Error 1
gmake[1]: *** [Makefile:18: /home/vagrant/RIOT/dist/tools/setsid/setsid] Error 2
make: *** [/home/vagrant/RIOT/makefiles/tools/targets.inc.mk:40: /home/vagrant/RIOT/dist/tools/setsid/setsid] Error 2
make: Leaving directory '/home/vagrant/RIOT/tests/cpp_ctors'

miri64 · 2020-07-28T16:19:22Z

RIOT/Makefile.include

Lines 650 to 658 in 0e3aa2f

# Check if setsid command is available on the system for debug target

# This is not the case on MacOSX, so it must be built on the fly

ifneq (,$(filter debug, $(MAKECMDGOALS)))

ifneq (0,$(shell which setsid 2>&1 > /dev/null ; echo $$?))

SETSID = $(RIOTTOOLS)/setsid/setsid

$(call target-export-variables,debug,$(SETSID))

DEBUGDEPS += $(SETSID)

endif

endif

Just removing that makes make debug possible... So another regression for FreeBSD ^^"

miri64 · 2020-07-28T16:21:02Z

$ make -C tests/cpp_ctors/ debug   
make: Entering directory '/home/vagrant/RIOT/tests/cpp_ctors'
gdb -q --args /home/vagrant/RIOT/tests/cpp_ctors/bin/native/tests_cpp_ctors.elf 
Reading symbols from /home/vagrant/RIOT/tests/cpp_ctors/bin/native/tests_cpp_ctors.elf...
(gdb) run
Starting program: /home/vagrant/RIOT/tests/cpp_ctors/bin/native/tests_cpp_ctors.elf 
warning: `/libexec/ld-elf.so.1': Shared library architecture i386:x86-64 is not compatible with target architecture i386.
warning: `/libexec/ld-elf.so.1': Shared library architecture i386:x86-64 is not compatible with target architecture i386.

Program received signal SIGSEGV, Segmentation fault.
0x2827241d in ?? ()
(gdb) quit
A debugging session is active.

	Inferior 1 [process 47909] will be killed.

Quit anyway? (y or n) y
make: Leaving directory '/home/vagrant/RIOT/tests/cpp_ctors'

:-/

miri64 · 2020-07-28T16:27:00Z

With #14631 (comment) addressed at least the non-C++ tests work again :D

fjmolinas · 2020-07-29T08:07:56Z

I don't like the use of DISABLE_LIBSTDCPP, it replicates the same as FEATURES_USED. If its broken then the feature should not be provided IMO, since the original PR introducing it was by @maribu, did you have a use case for this?

Anyway my comments are all about removing that variable.

I am not sure what you mean by that. From what I can tell, your suggestions don't allow for FreeBSD users to use libstdcpp if they e.g. cross-compile the lib themselves for 32-bit.

As I said my changes suggestions where only about removing the DISABLE_LIBSTDCPP extra variable which I don't like. If there is a use case where someone would have compiled his own version of libstdc++ for FreeBSD, I would have preferred an approach where the person that compiled his own libstdc++ would set FEATURES_PROVIDED+=libstdcc (maybe in a RIOT_MAKEFILES_GLOBAL_PRE), instead of adding a new variable that is read across Makefiles. Its like if someone provided pwriph_hwrng for a CPU that has the capabilities but not the code in master, we don't have a variable that does ENABLE_PERIPH_HWRNG. And also how does this handling translate to Kconfig? It doesn't so far, there is simply a HAS_LIBSTDCPP, so now this does not work the same for kconfig and with features. So IMO we should remove DISABLE_LIBSTDPP and HAS_LIBSTDCPP should be selected based on CPU_MODEL in Kconfig.

miri64 · 2020-07-29T08:17:01Z

Thanks for the explanation @fjmolinas. I might have found another solution I stumbled upon when researching #14631 (comment). I will try if that maybe solves the linking problems altogether.

maribu · 2020-07-29T08:21:16Z

I would have preferred an approach where the person that compiled his own libstdc++ would set FEATURES_PROVIDED+=libstdcc (maybe in a RIOT_MAKEFILES_GLOBAL_PRE),

+1

miri64 · 2020-07-29T09:20:20Z

I might have found another solution I stumbled upon when researching #14631 (comment). I will try if that maybe solves the linking problems altogether.

Apparently this is how the vagrant box is already configured :-/. No idea how to select the right ld_elf.so.1 at runtime. There exists an ld_elf32.so.1 besides ld_elf.so.1 in /libexec

[vagrant@freebsd ~/RIOT]$ ls /libexec/
ld-elf.so.1	ld-elf32.so.1	resolvconf

but no idea how to convince the elf file to use that instead of ld-elf.so.1 with -nodefaultlibs :-/ Anyone knows a bit more about dynamic linking than I do?

maribu · 2020-07-29T09:29:02Z

I think you can invoke ld_elf32.so.1 as if it would be an interpreter for the dynamically linked binary.

E.g. on Alpine I can run

$ /lib/ld-musl-x86_64.so.1 /bin/echo "Hallo World!"
Hallo World!

maribu · 2020-07-29T09:30:35Z

Maybe FreeBSD has something link binfmt on Linux, so that you can convince it to automagically run 32 bit apps correctly?

miri64 · 2020-07-29T09:38:51Z

Maybe FreeBSD has something link binfmt on Linux, so that you can convince it to automagically run 32 bit apps correctly?

Please be more specific on that.

maribu · 2020-07-29T09:57:04Z

Sorry, it is binfmt_misc, not binfmt. You can provide the magic numbers of binary formats and give an interpreter that should be used to call an application. E.g. :DOSWin:M::MZ::/usr/bin/wine: will result in Windoofs .EXE applications being interpreted with wine, so that ./some_windoofs_prog.exe would work, as if it would be native.

I successfully used this concept to run ELF files compiler for other targets on my notebook using qemu. E.g. I can just run chroot /mnt to fix a bricked SD Card of my Raspberry Pi on my x86_64 system and simply run the package manager compiled for ARM directly. (An x86_64 qemu binary for ARM needs to be placed within the chroot folder at the very same path given in the binfmt_misc config line for this to work.)

I'm confident that this approach - if supported by FreeBSD - would work to run 32 bit binary with the correctly ld.so.

miri64 · 2020-07-29T10:03:14Z

Sorry, it is binfmt_misc, not binfmt. You can provide the magic numbers of binary formats and give an interpreter that should be used to call an application. E.g. :DOSWin:M::MZ::/usr/bin/wine: will result in Windoofs .EXE applications being interpreted with wine, so that ./some_windoofs_prog.exe would work, as if it would be native.

I'm not really sure where I am supposed to put that :-/.

maribu · 2020-07-29T11:43:38Z

Either my internet search skills are poor, or FreeBSD does not support binfmt_misc. Sorry

miri64 · 2020-07-29T11:56:18Z

Either my internet search skills are poor, or FreeBSD does not support binfmt_misc. Sorry

I am actually also confused where to put that in Linux? Are they commands? Are those configurations? Where do those words you said go 😅 Maybe if I know what to do in Linux I can work from there in FreeBSD ;-).

maribu · 2020-07-29T13:36:45Z

Maybe if I know what to do in Linux I can work from there in FreeBSD ;-).

It is a configuration interface telling the kernel how to execute non-native binary executables. The details are described e.g. here.

I'm not 100% sure, but it seems that FreeBSD doesn't have this feature. (Or my internet-search-fu is to bad to find it.) So the easiest might just be to adapt the Makefiles to run /bin/ld-foo.so /path/to/app.elf instead of /path/to/app.elf.

stale · 2022-03-02T11:18:37Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

Cross-compiling C++ for 32-bit on a 64-bit FreeBSD is not as easy as installing the respective lib via package management as it is in most Linux distributions. This configures the build system to honor the `libstdcpp` feature for 64-bit FreeBSD systems.

miri64 · 2022-05-24T09:43:48Z

Rebased.

I'm not 100% sure, but it seems that FreeBSD doesn't have this feature. (Or my internet-search-fu is to bad to find it.) So the easiest might just be to adapt the Makefiles to run /bin/ld-foo.so /path/to/app.elf instead of /path/to/app.elf.

The problem is that the linking error occurs duirng make all, not make term

[vagrant@freebsd ~/RIOT]$ make -C tests/cpp_ctors/ flash
make: Entering directory '/home/vagrant/RIOT/tests/cpp_ctors'
-e \033[1;32mBuilding application "tests_cpp_ctors" for "native" with MCU "native".\033[0m
-e
"gmake" -C /home/vagrant/RIOT/boards/native
"gmake" -C /home/vagrant/RIOT/boards/native/drivers
"gmake" -C /home/vagrant/RIOT/core
"gmake" -C /home/vagrant/RIOT/cpu/native
"gmake" -C /home/vagrant/RIOT/cpu/native/periph
"gmake" -C /home/vagrant/RIOT/cpu/native/stdio_native
"gmake" -C /home/vagrant/RIOT/drivers
"gmake" -C /home/vagrant/RIOT/drivers/periph_common
"gmake" -C /home/vagrant/RIOT/sys
"gmake" -C /home/vagrant/RIOT/sys/auto_init
"gmake" -C /home/vagrant/RIOT/sys/embunit
"gmake" -C /home/vagrant/RIOT/sys/test_utils/interactive_sync
/usr/local/bin/ld: skipping incompatible /usr/local/lib/gcc9/gcc/x86_64-portbld-freebsd12.1/9.3.0/../../../libstdc++.so when searching for -lstdc++
/usr/local/bin/ld: skipping incompatible /usr/local/lib/gcc9/gcc/x86_64-portbld-freebsd12.1/9.3.0/../../../libstdc++.a when searching for -lstdc++
/usr/local/bin/ld: cannot find -lstdc++
collect2: error: ld returned 1 exit status
make: *** [/home/vagrant/RIOT/tests/cpp_ctors/../../Makefile.include:594: /home/vagrant/RIOT/tests/cpp_ctors/bin/native/tests_cpp_ctors.elf] Error 1
make: Leaving directory '/home/vagrant/RIOT/tests/cpp_ctors

Given that I do not really know how proceed with this and also somewhat lost track, I'd say: let's close it for now.

miri64 added this to the Release 2020.10 milestone Jul 28, 2020

miri64 requested review from maribu and fjmolinas July 28, 2020 09:37

maribu reviewed Jul 28, 2020

View reviewed changes

cpu/native/Makefile.dep Outdated Show resolved Hide resolved

miri64 added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Jul 28, 2020

miri64 force-pushed the native/fix/libstdcpp-in-freebsd branch from 7cc0ac0 to 5befda3 Compare July 28, 2020 13:35

fjmolinas reviewed Jul 28, 2020

View reviewed changes

miri64 mentioned this pull request Jul 28, 2020

make: only build MacOS-specific tool on MacOS #14644

Merged

gschorcht mentioned this pull request Jul 29, 2020

cpu/esp32: cleanup of C++ hacks #14649

Merged

jia200x modified the milestones: Release 2020.10, Release 2021.01, Release 2021.04 Jan 14, 2021

kaspar030 modified the milestones: Release 2021.04, Release 2021.07 Apr 23, 2021

MrKevinWeiss removed this from the Release 2021.07 milestone Jul 15, 2021

stale bot added the State: stale State: The issue / PR has no activity for >185 days label Mar 2, 2022

fjmolinas removed the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Mar 2, 2022

stale bot removed the State: stale State: The issue / PR has no activity for >185 days label Mar 2, 2022

miri64 added 2 commits May 24, 2022 11:20

native: fix C++ for FreeBSD

612a6c7

Cross-compiling C++ for 32-bit on a 64-bit FreeBSD is not as easy as installing the respective lib via package management as it is in most Linux distributions. This configures the build system to honor the `libstdcpp` feature for 64-bit FreeBSD systems.

fixup! native: fix C++ for FreeBSD

ad1348c

miri64 force-pushed the native/fix/libstdcpp-in-freebsd branch from 76b9a8c to ad1348c Compare May 24, 2022 09:43

github-actions bot added Area: cpu Area: CPU/MCU ports and removed Area: build system Area: Build system labels May 24, 2022

miri64 closed this May 24, 2022

miri64 mentioned this pull request Oct 13, 2022

boards/native: Remove macOS as native target #18735

Merged

native: fix C++ for FreeBSD #14631

native: fix C++ for FreeBSD #14631

Conversation

miri64 commented Jul 28, 2020 • edited Loading

Contribution description

Testing procedure

Issues/PRs references

maribu commented Jul 28, 2020

miri64 commented Jul 28, 2020

miri64 commented Jul 28, 2020

miri64 commented Jul 28, 2020

fjmolinas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjmolinas commented Jul 28, 2020

miri64 commented Jul 28, 2020

miri64 commented Jul 28, 2020

miri64 commented Jul 28, 2020

miri64 commented Jul 28, 2020

miri64 commented Jul 28, 2020

fjmolinas commented Jul 29, 2020

miri64 commented Jul 29, 2020

maribu commented Jul 29, 2020

miri64 commented Jul 29, 2020

maribu commented Jul 29, 2020 • edited Loading

maribu commented Jul 29, 2020

miri64 commented Jul 29, 2020

maribu commented Jul 29, 2020

miri64 commented Jul 29, 2020

maribu commented Jul 29, 2020

miri64 commented Jul 29, 2020

maribu commented Jul 29, 2020

stale bot commented Mar 2, 2022

miri64 commented May 24, 2022

miri64 commented Jul 28, 2020 •

edited

Loading

maribu commented Jul 29, 2020 •

edited

Loading