Add Depends to monero #3430

TheCharlatan · 2018-03-18T16:28:11Z

Depends is a localized cross compilation environment originally developed by Cory Fields. It allows its users to create cross compiled binaries for 32 and 64 bit linux and windows, arm hf, aarch64 and macos.

This pull request places depends into contrib/depends. The required monero dependencies are generated for each platform triplet (for example x86_64-w64-mingw32) and saved in their own directory in depends. Recipes for each package can be found in contrib/depends/packages, packages.mk defines the packages used for each platform triplet. The allowed ${triplet}'s are: i686-w64-mingw32, x86_64-w64-mingw32, x86_64-apple-darwin11, x86_64-linux-gnu, i686-linux-gnu, arm-linux-gnueabihf and aarch64-linux-gnu.

Libraries and tools are generated in depends by running make HOST=${triplet}. This step first downloads tar archives of the required dependency source files and places them in sources/. These archives are then checked for their sha256sum hash and compared with a hardcoded value in the package recipe. They are then extracted and placed into work/build during a build process. After building the package binaries/libraries are installed in work/staging . Once completed their staging directory is compressed and placed into built/ , after which their content in work/ is deleted. When every package is in built/ their archives are decompressed into ${triple}/ . This is where the monero build system will pick them up from.

A cmake toolchain file is uniquely generated for ${triple} and placed in contrib/depends/${triplet}/share/toolchain.cmake. They can then be accessed by the monero build system by executing for example: cmake -DCMAKE_TOOLCHAIN_FILE=`pwd`/contrib/depends/${triplet}/share/toolchain.cmake . The toolchain file also sets the DEPENDS variable to identify a depends build to the monero build system. This is used to exclude some additions from external as well as correcting some make flags.

General usage is also briefly reflected in the Readme.md under "Building portable statically linked binaries".

Example of a depends build for 64 bit windows:

Install build essentials, nsis and g++-mingw-w64-x86-64 on your host system. Then:

    cd contrib/depends
    make HOST=x86_64-w64-mingw32

Once it is finished building go back to the root directory and execute:

    cmake -DCMAKE_TOOLCHAIN_FILE=`pwd`/contrib/depends/x86_64-w64-mingw32/share/toolchain.cmake

Advantages of using depends:

Serves as a basis for reproducible builds on all plattforms
Allows cross compilation of windows and macos monero binaries
Having a standardized and well defined list of dependencies used in the project.
Vendored dependencies can now be source controlled from their original repository

One of the things changed that also affect builds other than with the depends system is lowercasing the winsock header include.
Another is that in this pull request the usage of unwind on windows is disable in the source code. Some discussion needs to be had about it. In its current form it only allows builds from depends.

Further topics that need to be discussed are the combinations of system and processor architecture pairs that need to be able to be passed to the monero build system for armv6, v7 and v8.

I realize this is hard to review, and if there is anything I can do to make it easier, do not hesitate to leave suggestions. So far I tested binaries for linux and windows 64bit.

moneromooo-monero

I have to say, my eyes glazed over quick :)

moneromooo-monero · 2018-03-19T22:09:20Z

.travis.yml

+  - $HOME/.ccache
+env:
+  global:
+    - MAKEJOBS=-j3


That should not be overridden (or there should be a user override).

This is a travis CI instruction set. Why is the override problematic? I only included it, because it is quite useful, but basically it does not much more than the current build bot.

Because building monero gobbles up a lot of RAM and 3 jobs means at least 6 GB.

Removed MAKEJOBS.

moneromooo-monero · 2018-03-19T22:13:31Z

cmake/FindLibunwind.cmake

@@ -18,9 +18,10 @@ find_path(LIBUNWIND_INCLUDE_DIR libunwind.h

 find_library(LIBUNWIND_LIBRARIES NAMES unwind )
 if(NOT LIBUNWIND_LIBRARIES STREQUAL "LIBUNWIND_LIBRARIES-NOTFOUND")
-  if (CMAKE_COMPILER_IS_GNUCC)
+  if (CMAKE_COMPILER_IS_GNUCC AND NOT DEPENDS)


That might be better off done in the caller, since it's find macro ?

This was actually not needed anymore. Removed.

moneromooo-monero · 2018-03-19T22:23:32Z

src/common/stack_trace.cpp

@@ -119,7 +119,6 @@ void log_stack_trace(const char *msg)
  int status;
  const char *log = stack_trace_log.empty() ? NULL : stack_trace_log.c_str();
 #endif
-


Please remove :)

TheCharlatan · 2018-03-20T03:04:01Z

I removed the travis yml from the pr. A lot of packages are required (like gcc > v5) that are not available on the standard travis distro (ubuntu trusty). This could be solved by starting a docker container, that then runs a more up to date distro, but this goes over the scope of this pr. In general using travis with depends should be encouraged, since it allows cross platform CI builds on all repository forks.

moneromooo-monero · 2018-03-20T10:28:48Z

Is it worth removing it just to fix the -j setting ?

TheCharlatan · 2018-03-20T11:05:10Z

It's not just the -j setting. The entire CI needs to be able to run in a docker container, since the current travis distro is out of date (cmake and gcc are too old).

TheCharlatan · 2018-04-25T16:38:17Z

Rebased. I removed the depends recipes for unbound and miniupnp, instead the build flags are passed to the submodules. Windows builds will work, as soon as the unbound cmake does not check for the winpthread dll anymore.

TheCharlatan · 2018-04-26T14:38:53Z

Opened #3 on the unbound fork to remove the winpthread check.

erciccione · 2018-07-01T12:33:55Z

HI @TheCharlatan could you give an ETA for this PR?

TheCharlatan · 2018-07-09T21:51:40Z

@erciccione still working on it. Currently the following issues are still remaining, and the reasons why my local travis builds are failing:

Pcsc is not easily statically cross compiled. While the headers are available in both mingw (for windows) and the macOS SDK, the actual library does not seem to be there. Pcsc-lite itself does not support mac cross compilation as is, so I am a bit stuck there. I will probably take some time end of July to work on this by dropping pcsc and instead use hid for the hardware communication. I already talked to the ledger devs about this and they all agreed to it.
This one should be easy, but I can't seem to figure out why the arm build does not want to shadow build anymore. If you look at the travis logs I posted above, cmake does not find any of the included subdirectories.
A bunch of smaller stuff, like inconsistent linking of icu etc. is also still in, but should be easy / can be fixed later.

moneromooo-monero · 2018-09-01T11:54:48Z

It would be very, very nice if we could have this in time for the 0.13 builds at least for x86_64. Even just Linux would be better than nothing if windows is being a pain.

Since libpcsc is being replaced, that just leaves the icu stuff. Any chance this can be done soon ?

Depends cross compiles project dependencies for linux, mac and windows and multiple architectures. Depends is original work by Cory Fields and used in bitcoin and a wide range of bitcoin related projects.

Add readline, ldns, graphviz, unbound to depends packages Add a cmake toolchain file to depends that is uniquely created for every build and placed in triple/share/toolchain.cmake This file is then passed to cmake with -DCMAKE_TOOLCHAIN_FILE=/path/to/triple/share/toolchain.cmake Add the boost locale package to depends In the depends cmake toolchain file, a DEPENDS flag is added to exclude, or change cmake checks done that are required for depends Link miniupnpc and unwind from depends and not external Add libiconv and icu4c to depends, required for mingw32 builds. Headers (winsock) need to be lower case in order to compile on unix systems. This should not affect building on windows.

Add pcsc-lite to linux builds Fixup windows icu4c linking with depends, the static libraries have an 's' appended to them Compiling depends arm-linux-gnueabihf will allow you to compile armv6zk monero binaries

Explain the role of the SDK in the darwin build. Add instructions to compile depends to the basic readme.

Fix builds for native linux and windows The architecture flag was set incorrectly. It needs to be set only when compiling arm6.

Drop miniupnp and unbound depends builds. Make sure that build variables are propageted properly to unbound and miniupnp. Rebase to after the v0.12 release

This includes a minimal qt build without gui

TheCharlatan · 2018-09-10T21:36:03Z

Rebased.
Current state looks like this (ticks indicate that the pr behaves as intended):

Linux x86_64 (looks like the compilation problem I am getting here is from master, it builds the daemon fine)
32-bit arm (same as linux)
64-bit arm (need source for librt)
64-bit windows (needs hid)
64-bit mac (needs hid)

moneromooo-monero · 2018-09-11T08:40:22Z

CMakeLists.txt

+  STAMP_DIR ${LRELEASE_PATH}
+  CMAKE_ARGS -DLRELEASE_PATH=${LRELEASE_PATH}
+  INSTALL_COMMAND cmake -E echo "")
+#>>>>>>> b1f43170... Add lrelease to the depends


Conflict markers

moneromooo-monero · 2018-09-11T08:41:09Z

CMakeLists.txt

@@ -812,8 +832,12 @@ endif()
 include_directories(SYSTEM ${Boost_INCLUDE_DIRS})
 if(MINGW)
  set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Wa,-mbig-obj")
-  set(EXTRA_LIBRARIES mswsock;ws2_32;iphlpapi;crypt32)
-  set(ICU_LIBRARIES ${Boost_LOCALE_LIBRARY} icuio icuin icuuc icudt icutu iconv)
+  set(EXTRA_LIBRARIES mswsock;ws2_32;iphlpapi)


crypt32 is needed on Windows, yes.

moneromooo-monero · 2018-09-11T08:42:45Z

contrib/depends/Makefile

+NO_QT ?=
+NO_WALLET ?=
+NO_UPNP ?=
+FALLBACK_DOWNLOAD_PATH ?= https://bitcoincore.org/depends-sources


Does this mean the build process actually downloads stuff ? If it does, does it actually execute it ?

getmonero should make the sources available as well, so we can change the download path from bitcoincore.org to getmonero.org .

moneromooo-monero · 2018-09-11T08:46:43Z

contrib/depends/description.md

+Each package is configured and patched so that it will yield the same
+build-results with each consequent build, within a reasonable set of
+constraints. Some things like timestamp insertion are unavoidable, and are
+beyond the scope of this system. Additionally, the toolchain itself must be


Does this not defeat the point or being able to check a hash, rather than diff binaries and work out that all differences are down to timestamps (which may be pretty hard for a non expert) ?

moneromooo-monero · 2018-09-11T08:47:08Z

contrib/depends/description.md

+
+Each package must define its source location and checksum. The build will fail
+if the fetched source does not match. Sources may be pre-seeded and/or cached
+as desired.


That probably answers my question above, seems good.

iDunk5400 · 2018-09-16T14:55:50Z

For 64-bit Windows, you need mingw-w64-x86_64-hidapi in MSYS2.

/Edit: Err, i just realised what I wrote... all the HID stuff is already included in mingw crosscompiler. Not sure at this point if source code changes are necessary.

iDunk5400 · 2018-09-16T15:36:10Z

Nevermind my last comment, it was wrong. Looks like MSYS2 also uses libhidapi from here.
(Meaning libhidapi is not included in the crosscompiler)

Fixup arm toolchain file.

icu tex support is not required, so just disable it. Re-add mistakingly removed crypt32 lib.

Set the architecture in the toolchain file correctly

moneromooo-monero

Thanks :)

4239735 Fixup 32bit arm build (TheCharlatan) a06d258 Fix Windows build (TheCharlatan) ecaf5b3 Add libsodium to the packages, the arm build was complaining about it. (TheCharlatan) cbbf4d2 Adapt translations to upstream changes (TheCharlatan) db57154 Updated pcsc url (TheCharlatan) f0ba19f Add lrelease to the depends (TheCharlatan) cfb3046 Add Miniupnp submodule (TheCharlatan) 5f7da00 Unbound is now a submodule. Adapt depends for this. (TheCharlatan) d6b9bdd Update readmes to reflect the usage of depends (TheCharlatan) 56b6e41 Add support for apple and arm building (TheCharlatan) 29311fd Disable stack unwinding for mingw32 depends build. (TheCharlatan) 8db3d57 Modify depends for monero's dependencies (TheCharlatan) 0806a23 Initial depends addition (TheCharlatan)

fluffypony

Reviewed

Technohacker · 2018-10-02T10:53:43Z

README.md

+The required packages are the names for each toolchain on apt. Depending on your distro, they may have different names.
+Then go back to the source dir and type for example for windows 64bit: 
+
+* ```cmake -DCMAKE_TOOLCHAIN_FILE=`pwd`/contrib/depends/x86_64-w64-mingw32```


This part seems incomplete, I think it's supposed to be cmake -DCMAKE_TOOLCHAIN_FILE=`pwd`/contrib/depends/x86_64-w64-mingw32/share/toolchain.cmake

Yes, this is wrong. Do you want to correct it? If you do, please move the section back up to ### Building portable statically linked binaries (Cross Compiling) . It seemed to have slipped down on my final rebase. Bit annoyed that the pr was still not in the best of states when it got merged.

I'll get it done tomorrow :)

As per monero-project#3430

TheCharlatan mentioned this pull request Mar 18, 2018

Deterministic/Reproducible builds #2641

Closed

TheCharlatan force-pushed the depends branch 6 times, most recently from 2afad21 to 4886588 Compare March 19, 2018 20:00

moneromooo-monero approved these changes Mar 19, 2018

View reviewed changes

TheCharlatan force-pushed the depends branch 3 times, most recently from 5adf3a1 to ac66997 Compare March 20, 2018 02:58

moneromooo-monero mentioned this pull request Apr 4, 2018

Compile errors and exit on Ubuntu 16.04 #3541

Closed

TheCharlatan force-pushed the depends branch from e0a4bbc to 1142548 Compare April 25, 2018 16:32

TheCharlatan force-pushed the depends branch from a1328c4 to 1b4cb63 Compare May 6, 2018 11:46

TheCharlatan force-pushed the depends branch from 1b4cb63 to b1f4317 Compare July 6, 2018 13:52

TheCharlatan added 7 commits September 10, 2018 22:03

Initial depends addition

0806a23

Depends cross compiles project dependencies for linux, mac and windows and multiple architectures. Depends is original work by Cory Fields and used in bitcoin and a wide range of bitcoin related projects.

Disable stack unwinding for mingw32 depends build.

29311fd

Add support for apple and arm building

56b6e41

Add pcsc-lite to linux builds Fixup windows icu4c linking with depends, the static libraries have an 's' appended to them Compiling depends arm-linux-gnueabihf will allow you to compile armv6zk monero binaries

Update readmes to reflect the usage of depends

d6b9bdd

Explain the role of the SDK in the darwin build. Add instructions to compile depends to the basic readme.

Unbound is now a submodule. Adapt depends for this.

5f7da00

Fix builds for native linux and windows The architecture flag was set incorrectly. It needs to be set only when compiling arm6.

Add Miniupnp submodule

cfb3046

Drop miniupnp and unbound depends builds. Make sure that build variables are propageted properly to unbound and miniupnp. Rebase to after the v0.12 release

TheCharlatan added 3 commits September 10, 2018 22:08

Add lrelease to the depends

f0ba19f

This includes a minimal qt build without gui

Updated pcsc url

db57154

Adapt translations to upstream changes

cbbf4d2

TheCharlatan force-pushed the depends branch from b1f4317 to cbbf4d2 Compare September 10, 2018 20:18

moneromooo-monero reviewed Sep 11, 2018

View reviewed changes

Add libsodium to the packages, the arm build was complaining about it.

ecaf5b3

Fixup arm toolchain file.

TheCharlatan force-pushed the depends branch from d51546c to ecaf5b3 Compare September 17, 2018 14:10

TheCharlatan added 2 commits September 17, 2018 23:59

Fix Windows build

a06d258

icu tex support is not required, so just disable it. Re-add mistakingly removed crypt32 lib.

Fixup 32bit arm build

4239735

Set the architecture in the toolchain file correctly

moneromooo-monero approved these changes Sep 18, 2018

View reviewed changes

fluffypony merged commit 4239735 into monero-project:master Sep 18, 2018

fluffypony reviewed Sep 18, 2018

View reviewed changes

Technohacker reviewed Oct 2, 2018

View reviewed changes

TheCharlatan deleted the depends branch October 17, 2018 12:00

HarrisonHesslink pushed a commit to EquilibriaCC/Equilibria that referenced this pull request Oct 30, 2018

Move cross compiling steps into proper heading

fc0eb8b

As per monero-project#3430

stoffu mentioned this pull request Dec 17, 2018

Build-related upstream patches 1 aeonix/aeon#88

Merged

stoffu mentioned this pull request Jun 12, 2020

Aeon upstream 8 aeonix/aeon#181

Merged

stoffu pushed a commit to stoffu/monero that referenced this pull request Jun 12, 2020

Move cross compiling steps into proper heading /monero#4494

a96535a

As per monero-project#3430

stoffu mentioned this pull request Jun 19, 2020

Aeon upstream 9 aeonix/aeon#183

Merged

iamamyth mentioned this pull request Mar 7, 2021

Mac: Enable on ARM, by not enforcing x86_64 #7435

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Depends to monero #3430

Add Depends to monero #3430

TheCharlatan commented Mar 18, 2018

moneromooo-monero left a comment

moneromooo-monero Mar 19, 2018

TheCharlatan Mar 19, 2018

moneromooo-monero Mar 20, 2018

TheCharlatan Mar 20, 2018

moneromooo-monero Mar 19, 2018

TheCharlatan Mar 20, 2018

moneromooo-monero Mar 19, 2018

TheCharlatan commented Mar 20, 2018

moneromooo-monero commented Mar 20, 2018

TheCharlatan commented Mar 20, 2018

TheCharlatan commented Apr 25, 2018

TheCharlatan commented Apr 26, 2018 •

edited

Loading

erciccione commented Jul 1, 2018

TheCharlatan commented Jul 9, 2018

moneromooo-monero commented Sep 1, 2018 •

edited

Loading

TheCharlatan commented Sep 10, 2018 •

edited

Loading

moneromooo-monero Sep 11, 2018

moneromooo-monero Sep 11, 2018

iDunk5400 Sep 16, 2018

moneromooo-monero Sep 11, 2018

TheCharlatan Sep 18, 2018

moneromooo-monero Sep 11, 2018

moneromooo-monero Sep 11, 2018

iDunk5400 commented Sep 16, 2018 •

edited

Loading

iDunk5400 commented Sep 16, 2018 •

edited

Loading

moneromooo-monero left a comment

fluffypony left a comment

Technohacker Oct 2, 2018 •

edited

Loading

TheCharlatan Oct 2, 2018

Technohacker Oct 2, 2018

Add Depends to monero #3430

Add Depends to monero #3430

Conversation

TheCharlatan commented Mar 18, 2018

Example of a depends build for 64 bit windows:

Advantages of using depends:

moneromooo-monero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheCharlatan commented Mar 20, 2018

moneromooo-monero commented Mar 20, 2018

TheCharlatan commented Mar 20, 2018

TheCharlatan commented Apr 25, 2018

TheCharlatan commented Apr 26, 2018 • edited Loading

erciccione commented Jul 1, 2018

TheCharlatan commented Jul 9, 2018

moneromooo-monero commented Sep 1, 2018 • edited Loading

TheCharlatan commented Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iDunk5400 commented Sep 16, 2018 • edited Loading

iDunk5400 commented Sep 16, 2018 • edited Loading

moneromooo-monero left a comment

Choose a reason for hiding this comment

fluffypony left a comment

Choose a reason for hiding this comment

Technohacker Oct 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheCharlatan commented Apr 26, 2018 •

edited

Loading

moneromooo-monero commented Sep 1, 2018 •

edited

Loading

TheCharlatan commented Sep 10, 2018 •

edited

Loading

iDunk5400 commented Sep 16, 2018 •

edited

Loading

iDunk5400 commented Sep 16, 2018 •

edited

Loading

Technohacker Oct 2, 2018 •

edited

Loading