Strange exported function names in official Maven build for Windows

[theolivenbaum]

I'm trying to switch our C# RockDB binding to use the official JNI builds, and that's fine for macOS and Linux, but weirdly, the function names exported for the Windows .dll files have some weird prefixes on them. When we try to use this library instead of the custom built .dll, it fails because it can't find any of the functions exported.

Expected behavior

Exported function names don't have added JAVA-related prefixes:

Actual behavior

Method names have strange prefixes, like Java_org_rocksdb_AbstractEventListener_...

Steps to reproduce the behavior

Download the latest Windows build from Maven, open it as a zip and use DLL Export or other tools to view exported function names.

[adamretter]

@theolivenbaum You are looking at the names of the JNI interface functions. These exist on all platforms in RocksJava binaries alongside the non-JNI functions. You should also be able to find the non-JNI (i.e. pure C++) API functions too.

[rubo]

Looks like the non-JNI functions are not exported for Windows.

[adamretter]

@rubo if that is the case, then we may need to tweak the CMake build that is used for Windows. @rhubner any ideas on this one?

[rhubner]

@adamretter Nothing in my sleeve at the moment, but at least I can replicate what @theolivenbaum mentioned.

[rubo]

Any chance this will be addressed in the next release?

[adamretter]

@rubo It depends on timing. We have a draft PR that @rhubner has prepared but it is still far from ideal (as it currently means having to list all files to export symbols from). We need to find a better approach in that PR - if you have ideas?

[theolivenbaum]

@adamretter any ideas why this is different than on the Linux/macOS builds?

[rhubner]

@theolivenbaum

It's a limitation of linker on windows. By default it only exports from the library you are compiling, in our case rocksdbjni. To export symbols from librocksdb you need to provide def file where all symbols are explicitly mentioned. Unfortunately there is no simple solution to generate these symbols automatically.

I have two ideas how to fix this, neither is best:

1. First compile rocksdb.dll and then export all symbols and with some PowerShell magic prepare def file.
2. Change build that rocksdb and rocksjni will be compiled together. At the moment we first compile rocksdb to .lib and then we compile rocksdbjni.dll and link together.

Radek

[theolivenbaum]

Wouldn't it be possible to compile just the rocksdb library without the Java methods instead and provide it as part of the package?

[rhubner]

@theolivenbaum
Yes, but then it will double the size of the distribution package(.jar) and code will be there twice. Once as librocksdb.dll and second as librocksdbjni.dll

[theolivenbaum]

Not sure if relevant in this case, but just found this: https://cmake.org/cmake/help/latest/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html

This can be used to export symbols from a .dll that are not in any of its object files 
but are added by the linker from dependencies (e.g. msvcrt.lib).

More info here too.

[theolivenbaum]

Another question about the JNI builds: are they built with or without the jemalloc / tcmalloc flags?

[rhubner]

@theolivenbaum

Another question about the JNI builds: are they built with or without the jemalloc / tcmalloc flags?

Without jemalloc/tcmalloc. We are not at this time planning to support jemalloc/tcmalloc in the Windows builds of RocksJava that are released. If you need that, you will have to build your own binaries. The binaries we release aim to be as portable as possible and have minimal optimisations enabled.

[theolivenbaum]

Perfect, just wanted to confirm that! I've added on our end a separate build for Linux (on top of using the JNI r)eleases) so we can have a librocksd-jemalloc.so, because we very often hit this in production. But that is only affecting Linux so far (and it's on an old RHEL release, so maybe not an issue in newer malloc implementations).

[rhubner]

Hello @theolivenbaum (cc @rubo)

I just updated #12246 Now we should be able to compile and release .dll on windows with rocksdb functions. The solution is not perfect, but it works. Unfortunately WINDOWS_EXPORT_ALL_SYMBOLS didn't work.

It will be great if you can check the PR and test if the output .dll works for you.

Radek

[rubo]

I think we can consider this issue resolved. Thank you for the fix.

[ajkr]

I'm trying to switch our C# RockDB binding to use the official JNI builds, and that's fine for macOS and Linux, but weirdly, the function names exported for the Windows .dll files have some weird prefixes on them. When we try to use this library instead of the custom built .dll, it fails because it can't find any of the functions exported.

Have you considered alternative solutions like updating the build script so it doesn't assume the build host's compression libraries will be present on the target machine? There should be a way to do this using our build system. Using Java artifacts for a non-Java project sounds like a less common solution, though I could be wrong.

[adamretter]

Using Java artifacts for a non-Java project sounds like a less common solution, though I could be wrong.

If you extract the .dll or .so files from the Jar file, then you really just have a binary build of RocksDB as a library. It does also include the JNI C++ API, but that is fairly light-weight, and you can use it if you wish, or just ignore it.