Get Core GL C++ Into Android Project

[bleege]

In order to support C++ / NDK debugging in Android Studio and streamline CI builds and artifact publication we need to refactor the current make android process to integrate the Core GL C++ project files into the Gradle managed Android project. The current process compiles the Core GL into .so files based on ABI and places them into jniLibs for the Gradle build process to then use as a separate step, while the new process will copy the uncompiled C++ files into jni and make (the Gradle driven) Android NDK responsible for compiling them and linking them via JNI to the Android code along with building the SDK as whole in one step. This process will essentially mirror what is done for iOS and Xcode with make iproj where Xcode is responsible for all the compilation and linking.

Unknowns

• Will custom Android.mk files be needed or will the built in NDK controls in Gradle be enough.
• Will current GL Makefile process work or will refactoring to use something like Bazel be needed?

Resources

• Android Studio, gradle and NDK integration by Xavier Hallade
  • I've personally found this article to be incredibly helpful for configs, architectures, and overall customization.
• Secrets of Android.mk
• Using custom Android.mk with Gradle/Android Studio
• NDK With Android Studio
• Building An Android Project With Djinni (Dropbox)

@mapbox/mobile @mapbox/gl

[bleege]

What Before and After should look like:

[mb12]

This would be really awesome when it is available. Is this already working in your local repo?
In this year's Google IO, they announced that Jetbrains NDK plugin will be available for free for Android development.

https://www.youtube.com/watch?v=f7ihSQ44WO0

[ljbade]

Yeah I look forward to having everything in Studio too. At one point in time (June last year) I did actually have everything built with a custom Android.mk file but a lot has changed since.

Let me know if you need help with this @bleege. First I would try getting NDK to compile all the C++ files. If we can get that to work then the hardest bit will be getting everything linked nicely.

[bleege]

@ljbade Yeah, it'd be fantastic to get your 👀 on this.

First I would try getting NDK to compile all the C++ files.

Agreed. It's a little unclear to me how all the C++ and related dependencies would be compiled via NDK. I assume that Mason has to stay in the mix somewhere? Can you sketch out how you think this would work on a branch?

[ljbade]

Hmm yeah we going to need mason. Can't think off the top of my head the best way to do this.

Perhaps we are going to need to create a series of custom Gradle tasks that somehow add dependencies and etc automatically from mason command calls into the NDK settings in the Gradle state.

[ljbade]

This is my thinking so far:

• gradle has custom task that copies C++ files to jni dir, perhaps by calling a gyp command that does this from the gyp files (it may be better to symlink though so you can edit the files in Android Studio?)
• gradle has another custom task that runs the mason setup - this will install the deps, copy the libs to the jniLibs dir and also insert the required compiler and linker flags into the NDK compile config
• gradle then uses normal NDK process to compile jni dir and link with jniLibs

[ljbade]

@kkaefer How hard would it be to get gyp to copy or symlink the C++ files in the Android target instead of compiling them?

Is it also possible to capture the compile and linker flags set in gyp? Berhaps echoing them to stdout then gradle can grab them

[kkaefer]

Can we get around copying the files into the jni dir by specifying the paths in a (generated) Android.mk file?

[bleege]

@kkaefer Yep. We can point the NDK at what ever path we want to. The only potential issue is if the debugger will work like this, but that's more of an unknown as Google hasn't even released it in Android Studio yet. In other words, lets worry about that later.

My big blocker is just knowing what all needs to be compiled on the C++ side. If I can get pointed at the all the C++ needed I can then get the Gradle / NDK side wired up to start compiling things.

[bleege]

Internal discussion between @kkaefer and @bleege about the pieces involved with C++ compilation for Android.

[bleege]

According to this bug report Android generator is being removed from GYP.

[bleege]

After experimenting with gyp's -f android and being told to run lunch, I'm going to look into using Gradle to set a task to copy the C++ files into jni and compile from there. This seems like a more robust solution based on the GYP bug report ^^ from May 2015 about it being removed anyway. Does this seem reasonable @kkaefer @springmeyer?

Makefile/android-%: CMD = deps/run_gyp android/mapboxgl-app.gyp $(CONFIG_android-$*) $(LIBS_android) --generator-output=./build/android-$* -f android

deps/run_gyp android/mapboxgl-app.gyp -Dhost=android -Iconfig/android-arm-v7.gypi -Dheadless_lib=none -Dplatform_lib=android -Dasset_lib=zip -Dhttp_lib=curl -Dcache_lib=sqlite --depth=. -Goutput_dir=. --generator-output=./build/android-arm-v7 -f android
Traceback (most recent call last):
  File "deps/run_gyp", line 8, in <module>
    sys.exit(gyp.script_main())
  File "deps/gyp/pylib/gyp/__init__.py", line 534, in script_main
    return main(sys.argv[1:])
  File "deps/gyp/pylib/gyp/__init__.py", line 527, in main
    return gyp_main(args)
  File "deps/gyp/pylib/gyp/__init__.py", line 512, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "deps/gyp/pylib/gyp/generator/android.py", line 969, in GenerateOutput
    assert android_top_dir, '$ANDROID_BUILD_TOP not set; you need to run lunch.'
AssertionError: $ANDROID_BUILD_TOP not set; you need to run lunch.
make: *** [Makefile/android-arm-v7] Error 1

[bleege]

Setting up the Copy C++ route. The build.gradle file will now copy in the basic C++ files (Android C++, Core GL, and Platform), while http, asset, and cache remain to be done.

After http, asset, and cache are done the next step will be to integrate the library dependencies as well as configure NDK to compile everything.

[bleege]

Reviewed http, asset and cache .gypi files and determined that they all are already included as part of the platform and android copy process. Future additions or removals in these directories should auto update as the copy processes (as currently) implemented grab default and android specific content (ie, no iOS stuff should ever be in these directories).

Will now look at setting up the basic NDK build process. This will include:

1. Adding the C++ library dependencies
2. Compiler Options like LDFLAGS

[bleege]

Added copying of Certs and Styles to assets directory in SDK file structure to the build file. This should be the last of the files that need to be transferred over AFAICT. The "current" view of what this looks like in the filesystem is the screen shot below.

[bleege]

Versions of Static Library dependencies are documented in configure

[ljbade]

@incanus Hopefully with Bitrise we can avoid the need to wrap make in gradle.

[ljbade]

@bleege I think we can close this now we selected Bitrise?

[bleege]

I think we can close this now we selected Bitrise?

@ljbade No. This is still need to address the need to have inline debugging of Java and C++. Please see the first sentence in this ticket:

In order to support C++ / NDK debugging in Android Studio

[mdakram]

Any update on this ?

[tomlocks]

Bumping...

Would It be possible to get at least working Android.mk and Application.mk files with all flags and dependecies needed to build?

@ljbade You mentioned before that you successfully created Android.mk with all dependecies for older SDK version. Can you share it with us?

[bleege]

@tomlocks We welcome PRs on enabling this functionality. The difficulty has always been trying to integrate the common C++ code that doesn't live directly in the Android project filesystem as it's reused by iOS, OSX, Qt, and others.

[ghost]

@bleege How can I import mapbox-gl-native Android sdk project to Android Studio like you? If I use make android command line, it generate .so files, I can't modify C++ files. Thanks!

[bleege]

How can I import mapbox-gl-native Android sdk project to Android Studio like you?

@zhangzhuowh The oiginal work on this was focused on copying the C++ source code into the Android Studio jni folder but it was never finalized. This is still the current status of this work if you'd like to pick this effort up. Pull Requests are always welcomed.

[jfirebaugh]

⇢ #5855