Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

additional Build-System features & fixes #327

Merged
merged 14 commits into from
Aug 29, 2017
Merged

additional Build-System features & fixes #327

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
4dc98cc
refactoring for multiple build-environments per platform
drywolf Jun 30, 2017
4e6c541
add "j2v8optimize" buildstep (performs execstack & strip on linux)
drywolf Jul 19, 2017
07fea24
additional build-system features & some j2v8 fixes
drywolf Jul 30, 2017
74221c9
set variables in pom.xml directly (no env-vars)
drywolf Jul 30, 2017
138a693
show progress percentage on `nodejs.py pkg`
drywolf Aug 1, 2017
2520075
extended documentation & code polishing
drywolf Aug 2, 2017
6092084
fix for #292
drywolf Aug 2, 2017
d372158
build-system unit-testing & other extras
drywolf Aug 14, 2017
7ca8618
more CLI documentation & build-step args
drywolf Aug 15, 2017
e5e85bc
docs cleanup & link to BUILDING.md
drywolf Aug 15, 2017
01fe088
additional CLI examples & `build -i` step-parsing fix
drywolf Aug 16, 2017
25b7b5c
atexit fix & prettier `build -i` listing
drywolf Aug 16, 2017
c44946c
show V8 & Node.js version on build
drywolf Aug 19, 2017
95d4543
some minor fixes
drywolf Aug 28, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,17 @@ hs_err*.log
*.iws
.idea

# python binaries
# Python binaries
*.pyc

# Build input/output.
# build input/output
node
build.out
cmake.out
node.out

# test input/output
test-mockup-os-release

# generated dependency packages
*.tar.*
190 changes: 172 additions & 18 deletions BUILDING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,114 @@
# Getting started / building from source

1. clone the source code from the [J2V8 GitHub repository](https://github.com/eclipsesource/J2V8)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously I added #327 (review) and @irbull found downloading a node version not a good separation of concerns. But I don't see what is the problem, actually.

Now we have all build options that include node already in a hard-coded version. Node yes/no could be an additional question at startup, followed by typing a version after 'yes'. This can of course be added in a follow-up PR, doesn't need to be in this one.

Copy link
Contributor Author

@drywolf drywolf Aug 15, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put this point already into my backlog/roadmap for future additions based on your previous comments ... but I don't want to blow up this PR even further. Let's take one leap at a time, so to speak 😆

2. run `j2v8-cli.cmd` (on Win32) or `source j2v8-cli.sh` on MacOS / Linux
3. `nodejs git clone` to clone the Node.js/V8 source code
4. `nodejs diff apply` to apply the required modifications to the Node.js source code
5. start the desired J2V8 build either via `build -i` or `build ...args` (see below for details)

# Build-System CLI

## Interactive
```shell
build --i, --interactive
# or
python build.py --i, --interactive

entering interactive mode...

[0] Docker >> android-x86 >> NODE_ENABLED
[1] Docker >> android-arm >> NODE_ENABLED
[2] Docker >> alpine-linux-x64 >> NODE_ENABLED
[3] Docker >> linux-x64 >> NODE_ENABLED
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you choose these ones as suggestions? Is it possible to add suggestions? I though interactive would be that I would pick target, vendor, platform, etc, interactively.. also I found a tiny bug, it is -i or --interactive, --i doesn't work

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you choose these ones as suggestions?
I though interactive would be that I would pick target, vendor, platform, etc, interactively..

Because not all combinations of platforms * architectures * vendors * virtualization-tech * feature-variations[N] are currently technically possible nor would it be sensible to try and support them all.

For the current suggestions I added the ones that are working and that were important for me to test the cross-platform build.

Is it possible to add suggestions?

Yes, have a look at build_system/build_configs.py

also I found a tiny bug, it is -i or --interactive, --i doesn't work

You can use -i or --interactive ... they are both doing the same thing (they are aliases)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, I also can imagine many ways to improve upon this feature and see what would be the best DX for the interactive CLI for devs that want to build J2V8 themselves.
But that would be stuff for a next PR ... this one is already bloated enough 😆

[4] Docker >> linux-x86 >> NODE_ENABLED
[5] Vagrant >> macosx-x64 >> NODE_ENABLED
[6] Vagrant >> macosx-x86 >> NODE_ENABLED
[7] Native >> windows-x64 >> NODE_ENABLED
[8] Docker >> windows-x64 >> NODE_ENABLED
[9] Vagrant >> windows-x64 >> NODE_ENABLED

Select a predefined build-configuration to run: 2
Building: Docker >> alpine-linux-x64 >> NODE_ENABLED

Override build-steps ? (leave empty to run pre-configured steps): nodejs j2v8 test
```

## Non-interactive
```shell
build -h, --help
# or
python build.py -h, --help

usage: build [-h] --target {android,linux,macos,win32} --arch {x86,x64,arm}
[--vendor VENDOR] [--keep-native-libs] [--node-enabled]
[--docker] [--vagrant] [--sys-image SYS_IMAGE] [--no-shutdown]
[--redirect-stdout] [--interactive]
[build-steps [build-steps ...]]
```

### Basic Examples

Build for Debian/Ubuntu Linux x64 on the host-system:<br/>
`build -t linux -a x64`

Build for Debian/Ubuntu Linux x64 using Docker:<br/>
`build -t linux -a x64 -dkr`

Build for Alpine-Linux x64 using Docker and Node.js features included:<br/>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alpine is kinda exotic, I would put a documentation for a typical ubuntu user as first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I will extend / update this part

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

`build -v alpine -t linux -a x64 -dkr -ne`

Build for MacOSX x64 using Vagrant excluding Node.js features:<br/>
`build -t macos -a x64 -vgr`

Build for Windows x64 directly on the host-system, Node.js features included:<br/>
`build -t win32 -a x64 -ne`

### Build-Step syntax

If no build-steps are specified, then the CLI will run `all` available build-steps by default.
To see a list of available build-steps run `build --help` or see the ***Build-Steps*** section below.

For ease of use, there are also some advanced build-step aliases that when specified will run a collection of some of the base-steps:

- `all` ... is the default, and will run all known build-steps
- `native` ... will run only the build-steps that are relevant for building **native** artifacts
- `node_js`, `j2v8_cmake`, `j2v8_jni`, `j2v8_cpp`, `j2v8_optimize`
- `j2v8` ... runs all build-steps, except for `nodejs` and `j2v8test`
- `java` ... alias for the single `j2v8java` step
- `test` ... alias for the single `j2v8test` step

#### Anti-Steps
Anti-steps provide a way to exclude a particular step, or a step-alias from the set of build-steps that should be run. To use such an anti-step, just prefix any of the available build-steps with the "~" symbol.

Build everything but do not optimize and do not run J2V8 unit tests:<br/>
`build <...other-args> all ~j2v8optimize ~test`

Build only the Java parts and also run tests:<br/>
`build <...other-args> all ~native`

#### Step-Arguments

For some of the build-steps, you can pass additional command-line parameters that will be added as arguments when the CLI build-tool of this particular build-step is run.

Run the `j2v8test` step with additional args that will be passed to maven:<br/>
(e.g. run only the `LibraryLoaderTest`)<br/>
`build -t linux -a x64 --j2v8test="-Dtest=LibraryLoaderTest"`



# Build-Steps

The J2V8 build-system performs several build steps in a fixed order to produce the final J2V8 packages for usage on the designated target platforms. What follows is a short summary for what each of the executed build-steps does and what output artifacts are produced by each step.

```
Node.js --> CMake --> JNI --> C++ --> Optimize --> Java/Android Build --> Java/Android Test
```
---
## Node.js
CLI name: `nodejs`

Builds the [Node.js](https://nodejs.org/en/) & [V8](https://developers.google.com/v8/) dependency artifacts that are later linked against by the J2V8 native bridge code.
Builds the [Node.js](https://nodejs.org/en/) & [V8](https://developers.google.com/v8/) dependency artifacts that are later linked into the J2V8 native bridge code.
Copy link
Contributor

@matiwinnetou matiwinnetou Aug 2, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel guilty about commenting that you did so much work but I have to say I didn't learn much from description from nodejs.py:

➜  J2V8 git:(drywolf-build-sys-features) ✗ python nodejs.py -h
usage: nodejs.py [-h] command

positional arguments:
  command

optional arguments:
  -h, --help  show this help message and exit

I still don't know how checkout nodejs source, it used to be python prepare_build.py

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to breat something to get commands:

➜  J2V8 git:(drywolf-build-sys-features) ✗ python nodejs.py 7.4.0
usage: nodejs.py [-h] command
nodejs.py: error: argument command: invalid choice: '7.4.0' (choose from 'flush-cache', 'fc', 'git-init', 'gi', 'package', 'pkg', 'store-diff', 'sd', 'apply-diff', 'ad')

Copy link
Contributor Author

@drywolf drywolf Aug 2, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script is far from finished, I need to first get a better picture of how this would be used in a CI build before I start this kind of polishing 😉
But I agree that this help output is pretty useless even at this point, I will add some preliminary help for now.

PS: Thanks for reviewing ... once you are done I will start to see what needs fixing and how to prioritze 👍

PPS: looks like there is a feature of the args parser that I could use to improve upon that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that after my successful build I only have a ./node/out directory. The other ones listed here are non-existing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not all of those directories will be created for all platforms ... the latter three ones are only created on Win32 builds of Node.js IIRC
I will add some remarks about that to the documentation 👍

(only works if the Node.js source was checked out into the J2V8 `./node` directory)

__Inputs:__
- Node.js source code
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking out the complete node project is about 1.9GB. Maybe you can advise to checkout only required parts, like:

git clone --depth=2 -b v8.x https://github.com/nodejs/node.git

This yields 236MB

Expand All @@ -16,13 +119,15 @@ __Inputs:__
__Artifacts:__
- Node.js & V8 static link libraries
- `./node/out/`
- `./node/build/`
- `./node/Debug/`
- `./node/Release/`
- *win32 specific*
- `./node/build/`
- `./node/Debug/`
- `./node/Release/`
---
## CMake
CLI name: `j2v8cmake`

Uses [CMake](https://cmake.org/) to generate the native Makefiles / IDE project files to later build the J2V8 C++ native bridge shared libraries (.so/.dylib/.dll)
Uses [CMake](https://cmake.org/) to generate the native Makefiles / IDE project files to later build the J2V8 C++ native bridge shared libraries.

__Inputs__:
- Node.js / V8 static link libraries
Expand All @@ -35,42 +140,81 @@ __Artifacts:__
- CMake generated Makefiles / IDE Project-files
- `./cmake.out/{platform}.{architecture}/`
---
## JNI
## JNI Header Generation
CLI name: `j2v8jni`

The previously generated Makefiles / IDE project files are used to compile and link the J2V8 C++ source code, which provides the JNI bridge to interop between the Java code and the C++ code of Node.js / V8.
Generate the JNI glue header file from the native method definitions of the Java `V8` class.

__Inputs__:
- Java V8.class file
- `./target/classes/com/eclipsesource/v8/V8.class`

__Artifacts:__
- J2V8 C++ JNI header file
- `./jni/com_eclipsesource_v8_V8Impl.h`
---
## C++
CLI name: `j2v8cpp`

Compile and link the J2V8 native shared libraries (.so/.dylib/.dll), which contain the C++ JNI bridge code to interop with the embedded Node.js / V8 parts.

__Inputs__:
- CMake generated Makefiles / IDE Project-files
- Node.js / V8 static link libraries & C++ header files
- J2V8 C++ JNI source code
- `./jni/com_eclipsesource_v8_V8Impl.h`
- `./jni/com_eclipsesource_v8_V8Impl.cpp`

__Artifacts:__
- J2V8 native shared libraries
- `./cmake.out/{platform}.{architecture}/libj2v8_{platform}_{abi}.{ext}`
- e.g. `./cmake.out/linux.x64/libj2v8_linux_x86_64.so`
- The built shared libraries will also be automatically copied to the required Java / Android project directories to be included in the .jar/.aar packages that will be built later.
- `./src/main/resources/` (Java)
- `./src/main/jniLibs/{abi}/libj2v8.so` (Android)
- `./cmake.out/{platform}.{architecture}/libj2v8-[vendor-]{platform}-{abi}.{ext}`
- e.g. `./cmake.out/linux.x64/libj2v8-alpine-linux-x86_64.so`
---
## Optimize
CLI name: `j2v8optimize`

The native J2V8 libraries are optimized for performance and/or filesize by using the available tools of the target-platform / compiler-toolchain.

__Inputs__:
- <u>unoptimized</u> J2V8 native shared libraries
- `./cmake.out/{platform}.{architecture}/libj2v8-[vendor-]{platform}-{abi}.{ext}`
- e.g. `./cmake.out/linux.x64/libj2v8-alpine-linux-x86_64.so`
- platform-specific optimization tools:
- Android: -
- Linux: `execstack`, `strip`
- MacOSX: -
- Windows: -

__Artifacts:__
- <u>optimized</u> J2V8 native shared libraries
- `./cmake.out/{platform}.{architecture}/libj2v8-[vendor-]{platform}-{abi}.{ext}`
- e.g. `./cmake.out/linux.x64/libj2v8-alpine-linux-x86_64.so`
---
## Java / Android
CLI name: `j2v8java` / `java`

Compiles the Java source code and packages it, including the previously built native libraries, into the final package artifacts. For the execution of this build-step [Maven](https://maven.apache.org/) (Java) or [Gradle](https://gradle.org/) (Android) are used for the respective target platforms.

__Inputs__:
- J2V8 native shared libraries
- J2V8 native shared libraries (will be automatically copied to the required Java / Android project directories to be included in the .jar/.aar packages)
- `./src/main/resources/` (Java)
- `./src/main/jniLibs/{abi}/libj2v8.so` (Android)
- J2V8 Java source code
- `./src/main/`
- J2V8 Java test source code
- `./src/test/`
- J2V8 build settings
- `./build_settings.py`

__Artifacts:__
- J2V8 platform-specific packages
- Maven platform-specific packages
- `./build.out/j2v8_{platform}_{abi}-{j2v8_version}.jar`
- e.g. `./build.out/j2v8_linux_x86_64-4.8.0-SNAPSHOT.jar`
- Gradle Android packages
- `./build/outputs/aar/j2v8-release.aar`
---
## JUnit
## Java Tests
CLI name: `j2v8test` / `test`

Runs the Java ([JUnit](http://junit.org/)) unit tests.

Expand All @@ -80,8 +224,18 @@ __Inputs__:
- `./src/test/`

__Artifacts:__
- Maven Surefire test reports
- Maven Surefire test reports (Desktop platforms)
- `./target/surefire-reports/`
- Gradle connected-test reports
- `./build/outputs/androidTest-results/connected/`
- Gradle Spoon test reports (Android only)
- `./build/spoon/debug/`
---

# Cross-Compiling

For cross-compiling J2V8 uses [Docker](https://www.docker.com/) (android, linux, windows) and [Vagrant](https://www.vagrantup.com/) (macos, windows).
The full source-code (of both J2V8 and Node.js) on the build-host are just shared via mounted volumes with the Docker / Vagrant machines, so you can quickly make changes and perform builds fast.

To invoke a cross-compile build, simply invoke the `build.py` script as usual but add the `--docker`, `-dkr` or `--vagrant`, `-vgr` flags.
This will automatically provision and run the necessary virtualization to run the requested build fully independent of your local environment.

<b>Note:</b> using Docker / Vagrant for cross-compilation requires many gigabytes of hard-drive space as well as downloading the required images & tools.
Loading