problem compiling and running with Apple Silicon processor

[nicocaille]

Hello,

I have erlang compiled on macOS Big Sur with arm64 arch:

Erlang/OTP 23 [erts-11.1.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [dtrace]

Interactive Elixir (1.11.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :erlang.system_info :system_version
'Erlang/OTP 23 [erts-11.1.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [dtrace]\n'
iex(2)> :erlang.system_info :system_architecture
'arm-apple-darwin20.1.0'

I changed the rebar.config file to take into account the CFLAGS/CXXFLAGS/LDFLAGS which i think were adapted to Apple Silicon framework:

{"apple-darwin-arm", "CFLAGS", "$CFLAGS -m64 -arch arm64 -fPIC -O3 -std=c99 -finline-functions -Wall -Wmissing-prototypes"},
{"apple-darwin-arm", "CXXFLAGS", "$CXXFLAGS -m64 -arch arm64 -fPIC -O3 -finline-functions -Wall"},
{"apple-darwin-arm", "LDFLAGS", "$LDFLAGS -m64 -flat_namespace -undefined suppress -lsodium"},

I checked that the file is compiled for the arm64 with the following:

lipo _build/dev/lib/enacl/priv/enacl_nif.so -info
Non-fat file: _build/dev/lib/enacl/priv/enacl_nif.so is architecture: arm64

I tried to call the sign_detached function as an example to see if it is working, getting the following warn/error message:

12:17:43.957 [warn]  The on_load function for module enacl_nif returned:
{:error,
 {:bad_lib,
  'Failed to find library init function: \'dlsym(0x134e18090, _nif_init): symbol not found\''}}

iex(3)> Sign.hello
** (UndefinedFunctionError) function :enacl_nif.crypto_sign_detached/2 is undefined (module :enacl_nif is not available)

iex(4)> Application.loaded_applications
[
  {:crypto, 'CRYPTO', '4.8'},
  {:logger, 'logger', '1.11.2'},
  {:stdlib, 'ERTS  CXC 138 10', '3.13.2'},
  {:sign, 'sign', '0.1.0'},
  {:iex, 'iex', '1.11.2'},
  {:enacl, 'Erlang libsodium (NaCl) bindings', '1.1.1'},
  {:hex, 'hex', '0.20.6'},
  {:compiler, 'ERTS  CXC 138 10', '7.6.5'},
  {:mix, 'mix', '1.11.2'},
  {:ssl, 'Erlang/OTP SSL application', '10.1'},
  {:inets, 'INETS  CXC 138 49', '7.3'},
  {:ex_unit, 'ex_unit', '1.11.2'},
  {:public_key, 'Public key infrastructure', '1.9.1'},
  {:kernel, 'ERTS  CXC 138 10', '7.1'},
  {:elixir, 'elixir', '1.11.2'},
  {:asn1, 'The Erlang ASN1 compiler version 5.0.14', '5.0.14'}
]

I don't know if i missed some fields in the compile arguments or if it is simply not compatible at the moment.

Thanks for your help

[jlouis]

Hi!

The particular bug is because for some reason, the compiled .so doesn't seem to have the right symbols. I don't understand why. MacOS somewhat eludes me, but in real operating systems used by real people in production you can use nm(1) to list symbols and figure out if _nit_init is present in the library. My guess, and this guess is somewhat weak, is that the ERL_NIF_INIT macro isn't expanded correctly for some reason. Or that the linker didn't include all files. I'm as clueless as you here.

One thing I wonder is if this only pertains to these bindings, or if a small nif example fails in general with Apple's compiler. It might also be you need to add some special flags to the compiler for this to work.

The underlying libsodium should have arm64 support, though IIRC some of the implementations fall back to C rather than using hand-optimized assembly. So it should work, but also test performance if it's important.

[jlouis]

actions/runner-images#2187 would be really nice to have here. Then I could go enable building for that platform.

[jlouis]

Another data point is that setup-beam as an action must support macos-latest too!

[dabaer]

Hi @nicocaille, I ran into this exact same issue on a different library (as well as this one) and fixed both by running:

export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

For some reason the compiler gets confused when you have x86 libraries installed so you just have to point it in the right direction.

This is alongside removing the arch -x86_64 from the makefile altogether. Your mileage may vary if you have x86 versions of the libraries installed as well, but I only tested it with a pure ARM homebrew installation and libraries.

[matthewcarlreetz]

#63

enacl/rebar.config

Lines 21 to 23 in e02325d

	{"darwin", "CFLAGS", "$CFLAGS -fPIC -O3 -std=c99 -finline-functions -Wall -Wmissing-prototypes"},
	{"darwin", "CXXFLAGS", "$CXXFLAGS -fPIC -O3 -finline-functions -Wall"},
	{"darwin", "LDFLAGS", "$LDFLAGS -flat_namespace -undefined suppress -lsodium"},

[doawoo]

For those who are still waiting for the PRs there's a quick workaround you can do in your environment that requires no modification to the library:

Export the paths for homebrew like @dabaer suggested:

export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

Now, also specify the CFLAGS, CXXFLAGS and LDFLAGS to include -arch arm64

export CFLAGS="-arch arm64"
export CXXFLAGS="-arch arm64"
export LDFLAGS="-arch arm64"

Now clean and rebuild your enacl dep and it will actually produce a dual-architecture library.

[midigofrank]

Hello,
anyone with an idea as to what might be going on/missing here? All the env vars have been set as specified by @doawoo

The libsodium library was installed via homebrew, version 1.0.19

===> Fetching pc v1.14.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/aead.c
===> Compiling c_src/enacl.c
===> Compiling c_src/enacl_ext.c
===> Compiling /Users/taylor/build/lightning/deps/enacl/c_src/enacl_nif.c
===> /Users/taylor/build/lightning/deps/enacl/c_src/enacl_nif.c:435:83: error: incompatible function pointer types initializing 'void (*)(ErlNifEnv *, void *)' (aka 'void (*)(struct enif_environment_t *, void *)') with an expression of type 'int (ErlNifEnv *, void **, ERL_NIF_TERM)' (aka 'int (struct enif_environment_t *, void **, unsigned long)') [-Wincompatible-function-pointer-types]
ERL_NIF_INIT(enacl_nif, nif_funcs, enacl_crypto_load, NULL, enacl_crypto_upgrade, enacl_crypto_unload);
                                                                                  ^~~~~~~~~~~~~~~~~~~
/Users/taylor/.asdf/installs/erlang/26.0.2/erts-14.0.2/include/erl_nif.h:432:25: note: expanded from macro 'ERL_NIF_INIT'
        LOAD, RELOAD, UPGRADE, UNLOAD,  \
                               ^~~~~~
1 error generated.

** (Mix) Could not compile dependency :enacl, "/Users/taylor/.asdf/installs/elixir/1.16.2-otp-26/.mix/elixir/1-16/rebar3 bare compile --paths /Users/taylor/build/lightning/_build/dev/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile enacl --force", update it with "mix deps.update enacl" or clean it with "mix deps.clean enacl"

[dabaer]

@midigofrank I ran into the same issue recently, the fixes I initially reported here no longer seem to work on Apple Silicon.

I haven't been able to find a solution and have since migrated to using Erlang's :crypto module for random bytes, and :argon2_elixir for hashing (which is all I used libsodium for).

I'm not sure of any alternatives for the Box type functions for encryption and signatures unfortunately.

[arpunk]

For those who are having issues with M1, the aeternity fork compiles and loads just fine: https://github.com/aeternity/enacl

[midigofrank]

Niice, the aeternity fork works fine . Thanks for mentioning this @arpunk 🙌

[zacksiri]

I ran into this issue with my x86_64 cpu. Not sure how to fix this. I tried the fork mentioned here and it worked. But ideally would like to use the official published hex.