Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FreeBSD Support #312

Open
moonshine-cc opened this issue Apr 17, 2022 · 14 comments
Open

Add FreeBSD Support #312

moonshine-cc opened this issue Apr 17, 2022 · 14 comments

Comments

@moonshine-cc
Copy link

moonshine-cc commented Apr 17, 2022

Running FreeBSD 13.0-RELEASE with current versions of go, rust and gmake installed:

build-rust-release works and finishes without errors (only requires a modification of SHARED_LIB_EXT = "so" for FreeBSD OS detection in the Makefile).

build-go doesn't work. Here is the output:

$ gmake build-go
go build ./...
go: downloading github.com/stretchr/testify v1.7.0
go: downloading github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
# github.com/CosmWasm/wasmvm/api
ld: error: undefined symbol: analyze_code
>>> referenced by cgo-gcc-prolog:56
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_analyze_code)
>>> referenced by cgo-gcc-prolog:609
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_analyze_code)

ld: error: undefined symbol: execute
>>> referenced by cgo-gcc-prolog:90
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_execute)
>>> referenced by cgo-gcc-prolog:639
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_execute)

ld: error: undefined symbol: get_metrics
>>> referenced by cgo-gcc-prolog:113
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_get_metrics)
>>> referenced by cgo-gcc-prolog:658
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_get_metrics)

ld: error: undefined symbol: ibc_channel_close
>>> referenced by cgo-gcc-prolog:146
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_channel_close)
>>> referenced by cgo-gcc-prolog:687
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_channel_close)

ld: error: undefined symbol: ibc_channel_connect
>>> referenced by cgo-gcc-prolog:179
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_channel_connect)
>>> referenced by cgo-gcc-prolog:716
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_channel_connect)

ld: error: undefined symbol: ibc_channel_open
>>> referenced by cgo-gcc-prolog:212
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_channel_open)
>>> referenced by cgo-gcc-prolog:745
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_channel_open)

ld: error: undefined symbol: ibc_packet_ack
>>> referenced by cgo-gcc-prolog:245
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_packet_ack)
>>> referenced by cgo-gcc-prolog:774
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_packet_ack)

ld: error: undefined symbol: ibc_packet_receive
>>> referenced by cgo-gcc-prolog:278
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_packet_receive)
>>> referenced by cgo-gcc-prolog:803
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_packet_receive)

ld: error: undefined symbol: ibc_packet_timeout
>>> referenced by cgo-gcc-prolog:311
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_packet_timeout)
>>> referenced by cgo-gcc-prolog:832
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_packet_timeout)

ld: error: undefined symbol: init_cache
>>> referenced by cgo-gcc-prolog:337
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_init_cache)
>>> referenced by cgo-gcc-prolog:854
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_init_cache)

ld: error: undefined symbol: instantiate
>>> referenced by cgo-gcc-prolog:371
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_instantiate)
>>> referenced by cgo-gcc-prolog:884
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_instantiate)

ld: error: undefined symbol: load_wasm
>>> referenced by cgo-gcc-prolog:395
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_load_wasm)
>>> referenced by cgo-gcc-prolog:904
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_load_wasm)

ld: error: undefined symbol: migrate
>>> referenced by cgo-gcc-prolog:428
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_migrate)
>>> referenced by cgo-gcc-prolog:933
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_migrate)

ld: error: undefined symbol: pin
>>> referenced by cgo-gcc-prolog:449
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_pin)
>>> referenced by cgo-gcc-prolog:950
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_pin)

ld: error: undefined symbol: query
>>> referenced by cgo-gcc-prolog:479
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_query)
>>> referenced by cgo-gcc-prolog:976
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_query)

ld: error: undefined symbol: reply
>>> referenced by cgo-gcc-prolog:512
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_reply)
>>> referenced by cgo-gcc-prolog:1017
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_reply)

ld: error: undefined symbol: save_wasm
>>> referenced by cgo-gcc-prolog:536
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_save_wasm)
>>> referenced by cgo-gcc-prolog:1037
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_save_wasm)

ld: error: undefined symbol: sudo
>>> referenced by cgo-gcc-prolog:569
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_sudo)
>>> referenced by cgo-gcc-prolog:1066
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_sudo)

ld: error: undefined symbol: unpin
>>> referenced by cgo-gcc-prolog:590
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_unpin)
>>> referenced by cgo-gcc-prolog:1083
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_unpin)

ld: error: undefined symbol: release_cache
>>> referenced by cgo-gcc-prolog:991
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_release_cache)

ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
gmake: *** [Makefile:37: build-go] Error 2
@webmaster128
Copy link
Member

Have a look at api/link_*.go. There you see a build condition and different link commands depending on the system. I guess linux in both cases makes that both link_*.go are excluded and the linkage lines

// #cgo LDFLAGS: -Wl,-rpath,${SRCDIR} -L${SRCDIR} -lwasmvm
import "C"

is missing.

@moonshine-cc
Copy link
Author

@webmaster128 And clue on how to resolve this?

I should have mentioned that I'm not a developer/programmer and that I have no experience with go.

@webmaster128
Copy link
Member

@moonshine-cc if that is the case I see very little chance for FreeBSD support any time soon. We have to build the library for each target system individually which is quite hard for Linux/Mac/Windows on x86/ARM already and this is the first time I hear about the desire for building for FreeBSD.

What's your motivation for targetting this system?

@moonshine-cc
Copy link
Author

@webmaster128 With FreeBSD it's possible to use Linux binaries and libraries natively thanks to its Linux Binary Compatibility Layer, which makes use of CentOS 7 (if interested in the details, see Chapter 10. Linux Binary Compatibility of the FreeBSD handbook.

I'm running a set of validators on a server powered by FreeBSD, which is an extremely pleasant experience so far (way better then with the usual Linux distributions).

Thanks to the Linux Binary Compatibilty Layer and all the needed components (like go and rust) available natively, I could make almost all validator projects that I have tried run without problems so far, except the latest one that made use of CosmWasm.

@webmaster128
Copy link
Member

webmaster128 commented Apr 18, 2022

Oh, that's interesting. Our .so is built on a CentOS 7 system. So it might just work with this diff: #313. Can you try that?

@moonshine-cc
Copy link
Author

@webmaster128 Seems to have worked:

$ go build ./...
go: downloading github.com/tendermint/tm-db v0.6.7
go: downloading github.com/stretchr/testify v1.7.0
go: downloading github.com/google/btree v1.0.0
go: downloading github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/golang/snappy v0.0.1

No errors, but now I'm puzzled, what's the result?
(I've just cloned the repo, applied the diff and only ran go build ./... to check what's going to happen)

@moonshine-cc
Copy link
Author

@webmaster128 Wow, it really works! 😃

So I have just rebuilt said project with the necessary changes applied and everything indeed turned out fine (though there was some trickery involved, but that's that project's fault). This doesn't even require the FreeBSD Linux Compatibilty Layer, because with go and rust being available natively, libwasmvm.so is build natively for FreeBSD.

So all that needs to be done to make wasmvm compatible with FreeBSD is as follows:

  • Your patch for the two api/link_*.go files.
  • Make "so" the default in Makefile line 11 (instead of empty) or add freebsd to the OS detection for the "so" file extension in the following lines.

You than can add FreeBSD to the supported builds in the README.md.

This is just awesome, because I have been fiddling around with this several times including giving up in the meantime, only to come back and try again. The major catch with said project was, that it doesn't build the libwasmvm.so but uses the already present binary, which then doesn't work as it's linked to Linux binaries which aren't there on this FreeBSD system (at least not in the right place, seems to be a mixup between FreeBSD and Linuxinator binaries).

@webmaster128
Copy link
Member

Glad to hear that.

Now the question is how you want this deployed. The Linux .so is built by the CI and checked into git for every change. See https://github.com/CosmWasm/wasmvm/tree/main/api for the .so and .dylib. I would like to do that for FreeBSD too because it comsumes a lot of space in git and I don't know if I found a FreeBSD CI system. If you don't want to build everything manually and patch the chain for every upgrade using a local version of wasmvm, it's probably better to use the Linux build.

@moonshine-cc
Copy link
Author

moonshine-cc commented Apr 19, 2022

@webmaster128 Sorry, not really sure what you mean. The already present Linux libwasmvm.so can not be used on FreeBSD, so that library has to be built natively on a FreeBSD system, which is what I just did. I don't know if this can be solved in any better way, but as long as I'm able to build libwasmvm.so locally, just having the mentioned changes works fine by me.

@webmaster128
Copy link
Member

webmaster128 commented Apr 19, 2022

The already present Linux libwasmvm.so can not be used on FreeBSD

But didn't you say it can be used via CentOS 7 compatibility mode on FreeBSD?

@moonshine-cc
Copy link
Author

But didn't you say it can be used via CentOS 7 compatibility mode on FreeBSD?

That's what I thought as well, but it didn't work. This is what it looks like with the Linux libwasmvm.so:

$ ldd craftd 
craftd:
        libthr.so.3 => /lib/libthr.so.3 (0x803d3e000)
        libwasmvm.so => /home/craftnator/go/pkg/mod/github.com/!cosm!wasm/wasmvm@v1.0.0-soon2.0.20220223181252-4aba468c9bc6/api/libwasmvm.so (0x803e00000)
        libc.so.7 => /lib/libc.so.7 (0x8043c4000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x803d6b000)
        librt.so.1 => /usr/lib/librt.so.1 (0x803d84000)
        libpthread.so.0 => not found (0)
        libm.so.6 => not found (0)
        libdl.so.2 => not found (0)
        libc.so.6 => not found (0)
        ld-linux-x86-64.so.2 (0)

I'm not able to explain what the problem is with this (looks like the Linux binary can't find the needed libraries in the right place, and even with some manual symlinking, it then said something about a wrong glibc version).

This is what it looks like with the FreeBSD libwasmvm.so:

$ ldd craftd
craftd:
        libthr.so.3 => /lib/libthr.so.3 (0x803d3e000)
        libwasmvm.so => /home/craftnator/go/pkg/mod/github.com/!cosm!wasm/wasmvm@v1.0.0-soon2.0.20220223181252-4aba468c9bc6/api/libwasmvm.so (0x803d6b000)
        libc.so.7 => /lib/libc.so.7 (0x804076000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x804487000)
        libm.so.5 => /lib/libm.so.5 (0x8044a0000)

So the only way to go is to build libwasmvm.so natively for FreeBSD.

@moonshine-cc
Copy link
Author

@webmaster128 So it's working, but I'm not sure how you could implement the required patches to make it universally usable. The diff for the two api/link_*.go files is OK, but oddly the OS detection for determining the library extension isn't.

I have GNU Make installed on FreeBSD, so I'm compiling the project with gmake build and although it supposedly should work that way, I'm getting "Need an operator" errors on the used ifeq / elseif / endif in lines 12, 14 and 22, which is why I simply have commented the whole checks out and made "so" the default:

SHARED_LIB_EXT = "so" # File extension of the shared library
#ifeq ($(OS),Windows_NT)
#       SHARED_LIB_EXT = dll
#else
#       UNAME_S := $(shell uname -s)
#       ifeq ($(UNAME_S),Linux)
#               SHARED_LIB_EXT = so
#       endif
#       ifeq ($(UNAME_S),Darwin)
#               SHARED_LIB_EXT = dylib
#       endif
#endif

Any ideas?

@webmaster128
Copy link
Member

I think it should not be too hard to convert the main Makefile into a set of shell scripts. We hardly use any make feature there anyways. If you want to roll out make build, you can just do:

(cd libwasmvm && cargo build --release)
cp libwasmvm/target/release/libwasmvm.so api
	
cp libwasmvm/bindings.h api

go build ./...

@edjroz
Copy link

edjroz commented May 10, 2022

Manual build seems the way to go. you could also do a modification to the existing makefile with a BSD branch

SHARED_LIB_EXT = "" # File extension of the shared library
ifeq ($(OS),Windows_NT)
       SHARED_LIB_EXT = dll
else
       UNAME_S := $(shell uname -s)
       ifeq ($(UNAME_S),Linux)
               SHARED_LIB_EXT = so
       endif
      ifeq ($(UNAME_S), FreeBSD) # since the library works for both OS
               SHARED_LIB_EXT = so 
       endif
       ifeq ($(UNAME_S),Darwin)
               SHARED_LIB_EXT = dylib
       endif
endif

@moonshine-cc This might lead to underlying issues so thread carefully e.g:
Wasmd offers support for OpenBSD but has to disable ledger support

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants