Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

building static libraries with shared library wrapper #105

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

nsec1
Copy link

@nsec1 nsec1 commented Jun 26, 2024

Ref. #74 haveno-dex/haveno#795

Now, there is only monero-java.so. It wraps monero, monero-cpp, boost etc.
Working with haveno on linux (--useNativeXmrWallet=true).
Depends woodser/monero-cpp#64

CMakeLists.txt Outdated
############
# Boost
############

set(Boost_NO_BOOST_CMAKE 1)
set(Boost_USE_MULTITHREADED ON)
set(Boost_USE_STATIC_LIBS ON)
Copy link
Owner

@woodser woodser Jun 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Ubuntu 20.0 and 22.04, I get this error with this field set to ON:

[100%] Linking CXX shared library libmonero-java.so
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libboost_serialization.a(basic_iarchive.o): warning: relocation against `_ZTVN5boost7archive6detail14basic_iarchiveE' in read-only section `.text'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libboost_chrono.a(chrono.o): relocation R_X86_64_PC32 against symbol `_ZN5boost6system6detail10cat_holderIvE24system_category_instanceE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value

But it works when set to OFF. Anything wrong with leaving it off?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, if I set it to OFF, it builds successfully, but the libraries built on Ubuntu 20.04 still can't be run on Ubuntu 22.04, showing an error from libboost_chrono.so.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think boost is compiled without -fPIC on ubuntu. I will look into libboost_chrono.so error.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, maybe it's not possible to build libs compatible with both versions then.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@woodser , does the actual build only run on Ubuntu 20.04?

Shared libs have different versions on Ubuntu 20.04 and Ubuntu 22.04.
Ex:

ubuntu-20-04:~$ ls /lib/x86_64-linux-gnu/libssl.*
/lib/x86_64-linux-gnu/libssl.a  /lib/x86_64-linux-gnu/libssl.so  /lib/x86_64-linux-gnu/libssl.so.1.1

ubuntu-22-04:~$ ls /lib/x86_64-linux-gnu/libssl.*
/lib/x86_64-linux-gnu/libssl.a  /lib/x86_64-linux-gnu/libssl.so  /lib/x86_64-linux-gnu/libssl.so.3

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know that the same build has ever worked on both platforms.

Was hoping the static libs with a shared library wrapper (for JNI) would be cross compatible.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @woodser .

I got haveno working with native wallet on ubuntu 20 and 22 but I have had to:

  • use openssl and unbound static lib
  • clone boost (1.71.0) and build it static with -fPIC (maybe it will become a module in monero-java)
  • disable Trezor hardware wallet support on monero build (USE_DEVICE_TREZOR=OFF). Because I have had problems to run haveno with protobuf built static with -fPIC. Is it a problem?

OBS: I built monero/monero-cpp/monero-java on ubuntu 20.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its ok to go in this way?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its ok to go in this way?

Yes I think we may have to go this way if we want this static/shared library wrapper idea to work, since the goal is to make the libraries standalone and supported in JNI. Maybe this is the best way?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think we may have to go this way if we want this static/shared library wrapper idea to work, since the goal is to make the libraries standalone and supported in JNI. Maybe this is the best way?

I think so. This is done with the last commit.
PS. I only tested this on linux. I dont have how testing on others platforms.

@woodser
Copy link
Owner

woodser commented Aug 10, 2024

I've applied this PR and woodser/monero-cpp#64.

Looks promising, but I'm having trouble building it.

On Ubuntu 20.04, I first get this error:

[100%] Built target monero-cpp
-- MONERO_CPP:/home/woodser/git/monero-java/external/monero-cpp
-- EXTRA_LIBRARIES:
-- Using Boost include dir at /home/woodser/git/monero-java/external/boost/boost
-- Found boost library: boost_chrono;boost_date_time;boost_filesystem;boost_program_options;boost_regex;boost_serialization;boost_wserialization;boost_system;boost_thread
-- Using OpenSSL include dir at /usr/include
-- Using OpenSSL libs: /usr/lib/aarch64-linux-gnu/libssl.a;/usr/lib/aarch64-linux-gnu/libcrypto.a;-lpthread;dl;dl;-lpthread
-- Using libsodium library at /usr/lib/aarch64-linux-gnu/libsodium.so
-- Using monero-project build: /home/woodser/git/monero-java/external/monero-cpp/external/monero-project/build/release
-- Found libunbound library: /usr/local/lib/libunbound.a
-- Configuring done
-- Generating done
-- Build files have been written to: /home/woodser/git/monero-java/build
make[2]: *** No rule to make target 'libmonero-cpp.a', needed by 'libmonero-java.so'.  Stop.
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/monero-java.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Setting BUILD_SHARED_LIBS to OFF in monero-cpp and deleting the build directories creates the needed static monero-cpp.a.

Then the build fails during linking:

[ 50%] Building CXX object CMakeFiles/monero-java.dir/src/main/cpp/monero_jni_bridge.cpp.o
[100%] Linking CXX shared library libmonero-java.so
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/libcrypto.a(sha1-armv8.o): relocation R_AARCH64_PREL64 against symbol `OPENSSL_armcap_P' which may bind externally can not be used when making a shared object; recompile with -fPIC
/usr/lib/aarch64-linux-gnu/libcrypto.a(sha1-armv8.o): in function `sha1_block_armv8':
(.text+0x1240): dangerous relocation: unsupported relocation
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/libcrypto.a(chacha-armv8.o): relocation R_AARCH64_PREL64 against symbol `OPENSSL_armcap_P' which may bind externally can not be used when making a shared object; recompile with -fPIC
/usr/lib/aarch64-linux-gnu/libcrypto.a(chacha-armv8.o):(.text+0x20): dangerous relocation: unsupported relocation
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/libcrypto.a(poly1305-armv8.o): relocation R_AARCH64_ADR_PREL_LO21 against symbol `poly1305_blocks' which may bind externally can not be used when making a shared object; recompile with -fPIC
/usr/lib/aarch64-linux-gnu/libcrypto.a(poly1305-armv8.o): in function `poly1305_init':
(.text+0x40): dangerous relocation: unsupported relocation
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/libcrypto.a(poly1305-armv8.o): relocation R_AARCH64_ADR_PREL_LO21 against symbol `poly1305_emit' which may bind externally can not be used when making a shared object; recompile with -fPIC
(.text+0x48): dangerous relocation: unsupported relocation
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/libcrypto.a(poly1305-armv8.o): relocation R_AARCH64_PREL64 against symbol `OPENSSL_armcap_P' which may bind externally can not be used when making a shared object; recompile with -fPIC
/usr/lib/aarch64-linux-gnu/libcrypto.a(poly1305-armv8.o): in function `poly1305_emit_neon':
(.text+0x9a0): dangerous relocation: unsupported relocation
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/libcrypto.a(sha256-armv8.o): relocation R_AARCH64_PREL64 against symbol `OPENSSL_armcap_P' which may bind externally can not be used when making a shared object; recompile with -fPIC
/usr/lib/aarch64-linux-gnu/libcrypto.a(sha256-armv8.o): in function `sha256_block_data_order':
(.text+0xf88): dangerous relocation: unsupported relocation
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/libcrypto.a(sha512-armv8.o): relocation R_AARCH64_PREL64 against symbol `OPENSSL_armcap_P' which may bind externally can not be used when making a shared object; recompile with -fPIC
/usr/lib/aarch64-linux-gnu/libcrypto.a(sha512-armv8.o): in function `sha512_block_data_order':
(.text+0x1108): dangerous relocation: unsupported relocation
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/monero-java.dir/build.make:122: libmonero-java.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/monero-java.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

On macOS, I get this error:

[ 50%] Building CXX object CMakeFiles/monero-java.dir/src/main/cpp/monero_jni_bridge.cpp.o
In file included from /Users/woodser/git/monero-java/src/main/cpp/monero_jni_bridge.cpp:39:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_full.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_model.h:55:
/Users/woodser/git/monero-java/external/monero-cpp/src/daemon/monero_daemon_model.h:55:10: fatal error: 'boost/property_tree/ptree.hpp' file not found
#include <boost/property_tree/ptree.hpp>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Changing set(Boost_INCLUDE_DIR "${BOOST_PATH}/boost") to set(Boost_INCLUDE_DIR "${BOOST_PATH}") seems to get further, but fails on the error:

In file included from /Users/woodser/git/monero-java/src/main/cpp/monero_jni_bridge.cpp:39:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_full.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_model.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/daemon/monero_daemon_model.h:55:
In file included from /Users/woodser/git/monero-java/external/boost/boost/property_tree/ptree.hpp:21:
In file included from /Users/woodser/git/monero-java/external/boost/boost/multi_index_container.hpp:25:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/at.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/at_impl.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/advance.hpp:19:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/negate.hpp:17:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/integral_c.hpp:32:
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
    typedef AUX_WRAPPER_INST( BOOST_MPL_AUX_STATIC_CAST(AUX_WRAPPER_VALUE_TYPE, (value - 1)) ) prior;
                              ^
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
#   define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)
                                              ^
In file included from /Users/woodser/git/monero-java/src/main/cpp/monero_jni_bridge.cpp:39:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_full.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_model.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/daemon/monero_daemon_model.h:55:
In file included from /Users/woodser/git/monero-java/external/boost/boost/property_tree/ptree.hpp:21:
In file included from /Users/woodser/git/monero-java/external/boost/boost/multi_index_container.hpp:25:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/at.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/at_impl.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/advance.hpp:19:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/negate.hpp:17:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/integral_c.hpp:32:
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
#   define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)

@woodser
Copy link
Owner

woodser commented Aug 11, 2024

Hoping the errors can be resolved because it would be really great to build to portable, static libraries. :)

@nsec1
Copy link
Author

nsec1 commented Aug 11, 2024

Setting BUILD_SHARED_LIBS to OFF in monero-cpp and deleting the build directories creates the needed static monero-cpp.a.

This is done in bin/build_libmonero_java.sh

Then the build fails during linking:

OpenSSL will have to become a module built with -fPIC. I will look into this.

Changing set(Boost_INCLUDE_DIR "${BOOST_PATH}/boost") to set(Boost_INCLUDE_DIR "${BOOST_PATH}") seems to get further, but fails on the error:

I will look into this

@nsec1
Copy link
Author

nsec1 commented Aug 14, 2024

Then the build fails during linking:

@woodser , could you try to build again on Ubuntu 20.04 aarch64-linux-gnu?

@woodser
Copy link
Owner

woodser commented Aug 17, 2024

I was able to build successfully on Ubuntu 20.04 aarch64-linux-gnu, but I'm still having trouble building on macOS, first failing on this error:

[ 50%] Building CXX object CMakeFiles/monero-java.dir/src/main/cpp/monero_jni_bridge.cpp.o
In file included from /Users/woodser/git/monero-java/src/main/cpp/monero_jni_bridge.cpp:39:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_full.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_model.h:55:
/Users/woodser/git/monero-java/external/monero-cpp/src/daemon/monero_daemon_model.h:55:10: fatal error: 'boost/property_tree/ptree.hpp' file not found
#include <boost/property_tree/ptree.hpp>

I'm confused why the build should work at all, including on Ubuntu, because set(Boost_INCLUDE_DIR "${BOOST_PATH}/boost") would use the full include path of ./external/boost/boost/property_tree/ptree.hpp, which doesn't exist.

By modifying CMakeLists.txt to set(Boost_INCLUDE_DIR "${BOOST_PATH}"), it's able to find the include path to the property tree, but then later fails on this error:

In file included from /Users/woodser/git/monero-java/src/main/cpp/monero_jni_bridge.cpp:39:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_full.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_model.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/daemon/monero_daemon_model.h:55:
In file included from /Users/woodser/git/monero-java/external/boost/boost/property_tree/ptree.hpp:21:
In file included from /Users/woodser/git/monero-java/external/boost/boost/multi_index_container.hpp:25:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/at.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/at_impl.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/advance.hpp:19:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/negate.hpp:17:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/integral_c.hpp:32:
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
    typedef AUX_WRAPPER_INST( BOOST_MPL_AUX_STATIC_CAST(AUX_WRAPPER_VALUE_TYPE, (value - 1)) ) prior;
                              ^
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
#   define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)
                                              ^
In file included from /Users/woodser/git/monero-java/src/main/cpp/monero_jni_bridge.cpp:39:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_full.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/wallet/monero_wallet_model.h:55:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/daemon/monero_daemon_model.h:55:
In file included from /Users/woodser/git/monero-java/external/boost/boost/property_tree/ptree.hpp:21:
In file included from /Users/woodser/git/monero-java/external/boost/boost/multi_index_container.hpp:25:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/at.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/at_impl.hpp:18:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/advance.hpp:19:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/negate.hpp:17:
In file included from /Users/woodser/git/monero-java/external/boost/boost/mpl/integral_c.hpp:32:
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
/Users/woodser/git/monero-java/external/boost/boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
#   define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)

@woodser
Copy link
Owner

woodser commented Aug 17, 2024

I see that monero-cpp is using boost and openssl with these paths instead of the new submodules, wondering if that could be a problem:

-- Using Boost include dir at /opt/homebrew/include
-- Using OpenSSL found at /opt/homebrew/opt/openssl@3
-- Using OpenSSL include dir at /opt/homebrew/opt/openssl@3/include
-- Using libsodium library at /opt/homebrew/lib/libsodium.dylib
-- Using HIDAPI include dir at /opt/homebrew/include/hidapi

Perhaps boost and openssl should be included within ./external/monero-cpp/external instead and used to build monero-cpp as well?

@nsec1
Copy link
Author

nsec1 commented Aug 18, 2024

I'm confused why the build should work at all, including on Ubuntu, because set(Boost_INCLUDE_DIR "${BOOST_PATH}/boost") would use the full include path of ./external/boost/boost/property_tree/ptree.hpp, which doesn't exist.

It works because you have boost installed. You are write, I have fixed boost include path.

@nsec1
Copy link
Author

nsec1 commented Aug 18, 2024

I see that monero-cpp is using boost and openssl with these paths instead of the new submodules, wondering if that could be a problem:

Whats the boost version installed on macOS?
monero-cpp and monero-project require boost >= 1.58.
monero-java, monero-cpp and monero-project should be built with compatible versions of boost.
I chose to use boost 1.71 because Ubuntu 20 uses this version.
I think yours macoS is using boost < 1.71.
If its true, We can force monero-cpp to require 1.71.

@woodser
Copy link
Owner

woodser commented Aug 18, 2024

Whats the boost version installed on macOS?

I have boost 1.84 installed through brew.

monero-java, monero-cpp and monero-project should be built with compatible versions of boost

Wondering again then if boost and openssl should be a submodule within external/monero-cpp/external, which both monero-java and monero-cpp build from?

This would be consistent with how monero-ts works, where boost [1] and openssl [2] are downloaded to external/monero-cpp/external.

@nsec1
Copy link
Author

nsec1 commented Aug 18, 2024

Wondering again then if boost and openssl should be a submodule within external/monero-cpp/external, which both monero-java and monero-cpp build from?

Ok. I will try. But monero-project also use boost and openssl and maybe we dont have control of it.

@woodser
Copy link
Owner

woodser commented Aug 18, 2024

Ok. I will try.

Maybe it's not necessary, and the boost versions merely need adjusted to be compatible, but I would think monero-cpp should be built with the same version as monero-java.

But monero-project also use boost and openssl and maybe we dont have control of it.

Don't know if the -fPIC difference could be a problem, but I'm not so well versed in this build stuff.

@nsec1
Copy link
Author

nsec1 commented Aug 21, 2024

@woodser could you try to build on macos again?
You have to apply the patch woodser/monero-cpp#64

@woodser
Copy link
Owner

woodser commented Aug 22, 2024

Hm, I still end up with these errors building monero-java with the new changes to this and monero-cpp:

In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/utils/gen_utils.cpp:53:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/utils/gen_utils.h:57:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/lexical_cast.hpp:32:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/lexical_cast/try_lexical_convert.hpp:43:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/lexical_cast/detail/converter_numeric.hpp:36:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/cast.hpp:33:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/converter.hpp:13:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/conversion_traits.hpp:13:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/detail/conversion_traits.hpp:18:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/detail/int_float_mixture.hpp:19:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/mpl/integral_c.hpp:32:
/Users/woodser/git/monero-java/build/install/include/boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
    typedef AUX_WRAPPER_INST( BOOST_MPL_AUX_STATIC_CAST(AUX_WRAPPER_VALUE_TYPE, (value - 1)) ) prior;
                              ^
/Users/woodser/git/monero-java/build/install/include/boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
#   define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)
                                              ^
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/utils/gen_utils.cpp:53:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/src/utils/gen_utils.h:57:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/lexical_cast.hpp:32:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/lexical_cast/try_lexical_convert.hpp:43:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/lexical_cast/detail/converter_numeric.hpp:36:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/cast.hpp:33:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/converter.hpp:13:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/conversion_traits.hpp:13:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/detail/conversion_traits.hpp:18:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/numeric/conversion/detail/int_float_mixture.hpp:19:
In file included from /Users/woodser/git/monero-java/build/install/include/boost/mpl/integral_c.hpp:32:
/Users/woodser/git/monero-java/build/install/include/boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
/Users/woodser/git/monero-java/build/install/include/boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
#   define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)

I confirmed that the build succeeds from the master branches.

@nsec1
Copy link
Author

nsec1 commented Aug 22, 2024

Could you past here or send me the full build output? I want to know if build is using boost lib from submodule or system.

@woodser
Copy link
Owner

woodser commented Aug 22, 2024

Could you past here or send me the full build output? I want to know if build is using boost lib from submodule or system.

build.log

@nsec1
Copy link
Author

nsec1 commented Aug 22, 2024

@woodser could you try again on macos and put the log here please?
You have to apply the patch woodser/monero-cpp#64

@woodser
Copy link
Owner

woodser commented Aug 23, 2024

It's failing to build boost with the latest changes to monero-java and monero-cpp:

build.log

@woodser
Copy link
Owner

woodser commented Aug 23, 2024

This is the error:

./boost/unordered/detail/fwd.hpp:53:16: error: no member named 'piecewise_construct' in namespace 'std'; did you mean 'piecewise_construct_t'?
    using std::piecewise_construct;
          ~~~~~^~~~~~~~~~~~~~~~~~~
               piecewise_construct_t

@nsec1
Copy link
Author

nsec1 commented Aug 23, 2024

Its related with the newer macOS plus boost 1.7x. See bambulab/BambuStudio#3957
I will try to use 1.8x.
Anyway, I need to emulate macos env on linux. Any suggestion?

@woodser
Copy link
Owner

woodser commented Aug 23, 2024

Yeah I saw that. I just updated my OS to the latest version, but it didn't help.

Using 1.8x sounds good, it's what I've been using in the libraries anyway.

I don't have any experience emulating mac in linux unfortunately.

@nsec1
Copy link
Author

nsec1 commented Aug 23, 2024

Build failed on ubuntu-20.04 with boost 1.86. We need to update monero-project. See monero-project/monero#9313
Can I do this in woodser/monero-cpp#64?

@woodser
Copy link
Owner

woodser commented Aug 23, 2024

Ok, I'm in the process of updating to monero-project v0.18.3.4 now.

@woodser
Copy link
Owner

woodser commented Aug 23, 2024

Ok it's all updated to monero-project v0.18.3.4.

@nsec1
Copy link
Author

nsec1 commented Aug 23, 2024

@woodser , build ok on ubuntu-20.04. Could you try again on macos?

@woodser
Copy link
Owner

woodser commented Aug 24, 2024

Sure, please resolve the merge conflict with master. :)

@woodser
Copy link
Owner

woodser commented Aug 24, 2024

With the PRs applied to monero-java and monero-cpp (and the merge conflict resolved), and using the latest monero-project v0.18.3.4 (pushed to my master at https://github.com/woodser/monero), I get these errors on building monero-project:

[ 44%] Building CXX object contrib/epee/src/CMakeFiles/obj_epee.dir/int-util.cpp.o
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/cryptonote_basic/cryptonote_format_utils_basic.cpp:31:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/cryptonote_basic/cryptonote_format_utils.h:33:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/cryptonote_basic/cryptonote_basic_impl.h:33:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/cryptonote_basic/cryptonote_basic.h:44:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/serialization/crypto.h:37:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/crypto/chacha.h:44:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/crypto/hash.h:37:
/Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/crypto/generic-ops.h:36:10: fatal error: 'sodium/crypto_verify_32.h' file not found
#include <sodium/crypto_verify_32.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 46%] Building CXX object contrib/epee/src/CMakeFiles/obj_epee.dir/portable_storage.cpp.o
1 error generated.
make[3]: *** [src/device/CMakeFiles/obj_device.dir/device.cpp.o] Error 1
make[3]: *** Waiting for unfinished jobs....
[ 46%] Building CXX object contrib/epee/src/CMakeFiles/obj_epee.dir/misc_language.cpp.o
[ 46%] Building CXX object src/cryptonote_basic/CMakeFiles/obj_cryptonote_basic.dir/cryptonote_basic_impl.cpp.o
1 error generated.
[100%] Built target generate_translations_header
make[3]: *** [src/device_trezor/CMakeFiles/obj_device_trezor.dir/device_trezor.cpp.o] Error 1
make[2]: *** [src/device_trezor/CMakeFiles/obj_device_trezor.dir/all] Error 2
[ 46%] Building CXX object src/blockchain_db/CMakeFiles/obj_blockchain_db.dir/lmdb/db_lmdb.cpp.o
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/ringct/rctSigs.cpp:35:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/common/util.h:48:
In file included from /Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/crypto/hash.h:37:
/Users/woodser/git/monero-java/external/monero-cpp/external/monero-project/src/crypto/generic-ops.h:36:10: fatal error: 'sodium/crypto_verify_32.h' file not found
#include <sodium/crypto_verify_32.h>

Here's the full build log:

build.log

monero-project succeeds to build if I run make release-static directly within the repo, so it must be related to building the full stack from ./bin/build_libmonero_java.sh .

I confirmed that the build succeeds by switching to the latest master branch of all 3 projects. To ensure the build is completely fresh, I first deleted the build directory of all 3 projects.

@woodser
Copy link
Owner

woodser commented Sep 12, 2024

Ideally we can keep existing hardware wallet dependencies, unless they are not already included? In which case, no need to add them as part of this PR.

@nsec1
Copy link
Author

nsec1 commented Sep 18, 2024

Ideally we can keep existing hardware wallet dependencies, unless they are not already included? In which case, no need to add them as part of this PR.

No. if we keep existing hardware wallet dependencies (libs linked to libmonero-java.so), the host running monero-java application must have dependencies installed on.

For example, haveno running on host with no hidapi lib installed and using monero-java (this PR) build with hidapi linked (your patch) gets this error:

[ STARTING] WARN  haveno.core.xmr.wallet.XmrWalletService: Failed to load Monero native libraries: java.lang.UnsatisfiedLinkError: /tmp/libmonero8612716650094032596/libmonero-java.so: libhidapi-libusb.so.0: cannot open shared object file: No such file or directory 
[ STARTING] WARN  haveno.core.xmr.wallet.XmrWalletService: Monero native libraries applied: false

At the moment there is just this problem, mac build needs hidapi. I didnt understand why because -D USE_DEVICE_TREZOR=OFF should do the job for all related dependencies.

@woodser
Copy link
Owner

woodser commented Sep 18, 2024

I've released updates to monero-java and monero-cpp, and I'd like to retest this PR with the new updates. Maybe it could have fixed the previous error we saw? But can you rebase your PRs on master, please?

@nsec1
Copy link
Author

nsec1 commented Sep 18, 2024

Ok, I will do this and move openssl, boost and sodium build to monero-cpp.

@nsec1 nsec1 force-pushed the sharedLibraryWrapper branch from 99925e5 to bca404b Compare September 18, 2024 20:23
@nsec1
Copy link
Author

nsec1 commented Sep 18, 2024

Done.

git clone --recurse-submodules -b sharedLibraryWrapper https://github.com/nsec1/monero-java.git

pom.xml Outdated
@@ -4,7 +4,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>io.github.woodser</groupId>
<artifactId>monero-java</artifactId>
<version>0.8.33</version>
<version>0.8.33x</version>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to change the version number now? Normally I bump it just before release.

Copy link
Author

@nsec1 nsec1 Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My fault. Just fixed

@nsec1 nsec1 force-pushed the sharedLibraryWrapper branch from 1ad252e to 1e5b6c9 Compare September 19, 2024 13:01
@woodser
Copy link
Owner

woodser commented Sep 19, 2024

I have your latest changes applied to monero-java, monero-cpp, and monero-project, but I'm getting this linker error at the end of the build process:

[ 50%] Building CXX object CMakeFiles/monero-java.dir/src/main/cpp/monero_jni_bridge.cpp.o
[100%] Linking CXX shared library libmonero-java.dylib
Undefined symbols for architecture arm64:
  "boost::filesystem::detail::canonical_v3(boost::filesystem::path const&, boost::filesystem::path const&, boost::system::error_code*)", referenced from:
      tools::wallet2::store_to(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, epee::wipeable_string const&, bool) in libwallet.a[2](wallet2.cpp.o)
  "boost::filesystem::detail::weakly_canonical_v3(boost::filesystem::path const&, boost::filesystem::path const&, boost::system::error_code*)", referenced from:
      tools::wallet2::store_to(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, epee::wipeable_string const&, bool) in libwallet.a[2](wallet2.cpp.o)
  "boost::filesystem::detail::directory_iterator_construct(boost::filesystem::directory_iterator&, boost::filesystem::path const&, boost::filesystem::directory_options, boost::filesystem::detail::directory_iterator_params*, boost::system::error_code*)", referenced from:
      std::__1::__function::__func<mlog_configure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool, unsigned long, unsigned long)::$_0, std::__1::allocator<mlog_configure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool, unsigned long, unsigned long)::$_0>, void (char const*, unsigned long)>::operator()(char const*&&, unsigned long&&) in libepee.a[7](mlog.cpp.o)
  "boost::filesystem::detail::copy_file(boost::filesystem::path const&, boost::filesystem::path const&, boost::filesystem::copy_options, boost::system::error_code*)", referenced from:
      tools::copy_file(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in libcommon.a[8](util.cpp.o)
  "_hid_close", referenced from:
      hw::io::device_io_hid::disconnect() in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_enumerate", referenced from:
      hw::io::device_io_hid::connect(unsigned int, unsigned int, boost::optional<int>, boost::optional<unsigned short>) in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_error", referenced from:
      hw::io::safe_hid_error(hid_device_*) in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_free_enumeration", referenced from:
      hw::io::device_io_hid::connect(unsigned int, unsigned int, boost::optional<int>, boost::optional<unsigned short>) in libdevice.a[6](device_io_hid.cpp.o)
      hw::io::device_io_hid::connect(unsigned int, unsigned int, boost::optional<int>, boost::optional<unsigned short>) in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_init", referenced from:
      hw::io::device_io_hid::init() in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_open_path", referenced from:
      hw::io::device_io_hid::connect(unsigned int, unsigned int, boost::optional<int>, boost::optional<unsigned short>) in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_read", referenced from:
      hw::io::device_io_hid::exchange(unsigned char*, unsigned int, unsigned char*, unsigned int, bool) in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_read_timeout", referenced from:
      hw::io::device_io_hid::exchange(unsigned char*, unsigned int, unsigned char*, unsigned int, bool) in libdevice.a[6](device_io_hid.cpp.o)
      hw::io::device_io_hid::exchange(unsigned char*, unsigned int, unsigned char*, unsigned int, bool) in libdevice.a[6](device_io_hid.cpp.o)
  "_hid_write", referenced from:
      hw::io::device_io_hid::exchange(unsigned char*, unsigned int, unsigned char*, unsigned int, bool) in libdevice.a[6](device_io_hid.cpp.o)
ld: symbol(s) not found for architecture arm64

Here's the full build.log: build.log

I confirmed that the build is working from the master branches and my release-v0.18 branch in monero-project, which includes a couple of custom commits.

Confirming this change should still be applied to monero-project?

commit ef2922ec64ca510027a0773ca54f79ba6e7acc4c (HEAD -> release-v0.18)
Author: nsec1 <167650977+nsec1@users.noreply.github.com>
Date:   Fri Sep 6 06:57:55 2024 -0300

    fix zmq and sodium include dirs search

Also I was wondering, is it necessary to checkout all of boost's submodules? There are so many of them :)

@nsec1
Copy link
Author

nsec1 commented Sep 19, 2024

Here's the full build.log: build.log

It's the include path conflict. Monero project is using /opt/homebrew/include for boost and maybe /opt/homebrew/lib too.

Confirming this change should still be applied to monero-project?

Yes. It resolves include path conflict for zmq and sodium

Could you apply this patch to monero-project and build again? It is for printing INCLUDE_DIRECTORIES list.

I confirmed that the build is working from the master branches and my release-v0.18 branch in monero-project, which includes a couple of custom commits.

Did you use my monero fork? I can fork yours and apply my change.

Also I was wondering, is it necessary to checkout all of boost's submodules? There are so many of them :)

It is out of my control, comes from git module update. I would have to track each module with its version. it doesn't worth.

@woodser
Copy link
Owner

woodser commented Sep 19, 2024

I can fork yours and apply my change.

Yes please, if you can please submit a PR to my release branch here: https://github.com/woodser/monero/tree/release-v0.18. That's now the default for the libraries, since that's what they're based on.

@woodser
Copy link
Owner

woodser commented Sep 20, 2024

I applied your patch manually (it wasn't included in your PR).

Here's the resulting log file: build.log

@nsec1
Copy link
Author

nsec1 commented Sep 20, 2024

I applied your patch manually (it wasn't included in your PR).

It is for debug only, can be ignored.

I make a fix on monero-cpp build script. -D BOOST_IGNORE_SYSTEM_PATHS=ON can resolve include/libs conflicts on macos+homebrew. Could you try again?

@woodser
Copy link
Owner

woodser commented Sep 20, 2024

I pulled in your commit to monero-cpp. Looks like I'm getting the same error, here's the full build log: build.log

@nsec1
Copy link
Author

nsec1 commented Sep 20, 2024

I don't know what to do. Monero project is using boost from /opt/homebrew/include no matter what i do. Looks like homebrew has priority over other libs on cmake. Remove boost from homebrew is a workaround. Any suggestion??

@woodser
Copy link
Owner

woodser commented Oct 4, 2024

I was able to build and run the libs, by uninstalling boost through brew, because as you said, it overrides and conflicts with the build boost.

I find that I must build with USE_DEVICE_TREZOR=ON or I get linker errors related to HIDAPI, e.g. "_hid_close". I'd like to make this the default and remove this option altogether, but you encounter build problems with it enabled?

@@ -1,3 +1,4 @@
[submodule "external/monero-cpp"]
path = external/monero-cpp
url = https://github.com/woodser/monero-cpp
url = https://github.com/nsec1/monero-cpp.git
branch = buildStaticOption
Copy link
Owner

@woodser woodser Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this branch be removed? We can assume your PR to monero-cpp is applied during the build process (as all PRs will be accepted at once).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can. It just makes tests easier. I will do this.

if(STATIC AND NOT IOS)
if(UNIX)
set(OPENSSL_LIBRARIES "${OPENSSL_LIBRARIES};${CMAKE_DL_LIBS};${CMAKE_THREAD_LIBS_INIT}")
option(USE_DEVICE_TREZOR "Trezor hardware wallet suport" ON)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get linker errors building with USE_DEVICE_TREZOR=OFF related to HIDAPI.

Can we remove this conditional and always find the package?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nsec1 any response?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@woodser other than this comment, is this PR ready to merge?

export HOST_NCORES=${HOST_NCORES-$(nproc 2>/dev/null || shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)}
BUILD_DIR="$(pwd)/build"
export CMAKE_PREFIX_PATH="$(pwd)/external/monero-cpp/build/install"${CMAKE_PREFIX_PATH+:$CMAKE_PREFIX_PATH}
export USE_DEVICE_TREZOR=OFF
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this or does it cause a problem on your system?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nsec1
Copy link
Author

nsec1 commented Oct 13, 2024

@woodser , just thoughts, why not make monero-java by passing monero-cpp and wallet2, and connect directly with monero daemon rpc? It would be java only, no jni, no portable builds.

@woodser
Copy link
Owner

woodser commented Oct 13, 2024

@woodser , just thoughts, why not make monero-java by passing monero-cpp and wallet2, and connect directly with monero daemon rpc? It would be java only, no jni, no portable builds.

monero-cpp and wallet2 would still need to be built native for the OS, and monero-java communicates with monero-cpp over JNI which also uses C++, so I don't see how this would circumvent native builds.

@nsec1
Copy link
Author

nsec1 commented Oct 13, 2024

@woodser , just thoughts, why not make monero-java by passing monero-cpp and wallet2, and connect directly with monero daemon rpc? It would be java only, no jni, no portable builds.

monero-cpp and wallet2 would still need to be built native for the OS, and monero-java communicates with monero-cpp over JNI which also uses C++, so I don't see how this would circumvent native builds.

monero-java would not use monero-cpp nor wallet2, no C++, no jni, just java communicates directly to monero daemon rpc using http. Maybe have to re-implement some wallet2 features but maybe its worth.

@NorrinRadd
Copy link

@nsec1 the concern is speed. Statically linked binaries run far faster than any RPC.

@nsec1
Copy link
Author

nsec1 commented Oct 16, 2024

@nsec1 the concern is speed. Statically linked binaries run far faster than any RPC.

Ok but wallet2 communicates to node daemon over rpc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants