Skip to content
This repository has been archived by the owner on May 11, 2022. It is now read-only.

RocksDB/5.8 #92

Closed
borgoat opened this issue Nov 19, 2017 · 6 comments
Closed

RocksDB/5.8 #92

borgoat opened this issue Nov 19, 2017 · 6 comments

Comments

@borgoat
Copy link

borgoat commented Nov 19, 2017

RocksDB: a library that provides an embeddable, persistent key-value store for fast storage.

I just got to know Conan, I'm checking it out for dependencies in this project of mine: ProfaneDB, and RocksDB is one of the dependencies.
I'll see what I can do and ask for help here if something doesn't work!

I gave a quick look and am quite sure nobody packaged RocksDB already, let me know if that's not the case!

@borgoat
Copy link
Author

borgoat commented Dec 26, 2017

Hello,

I've been having some issues during the test stage. Apparently not all dependencies are being linked.
I use within my test_package/CMakeLists.txt

target_link_libraries(rocks_sample ${CONAN_LIBS})

You may check out the whole code here. The error that happens during the linking follows.
The CI build on Travis also fails, but it may be for different reasons as I find it difficult to test properly when locally, because libraries are already there in my system and there isn't a complete isolation.

RocksDB/5.8@RocksDB/5.8: Package '04f6580e02f3d5f0e28569ff942937d4c741700b' created
RocksDB/5.8@RocksDB/5.8 test package: Generator cmake created conanbuildinfo.cmake
RocksDB/5.8@RocksDB/5.8 test package: Generator txt created conanbuildinfo.txt
RocksDB/5.8@RocksDB/5.8 test package: Generated conaninfo.txt
RocksDB/5.8@RocksDB/5.8 test package: Running build()
-- The CXX compiler identification is GNU 7.2.1
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Conan: called by CMake conan helper
-- Conan: Compiler GCC>=5, checking major version 7.2
-- Conan: Checking correct version: 7.2
-- Conan: Using cmake global configuration
-- Conan: Adjusting default RPATHs Conan policies
-- Conan: C++ stdlib: libstdc++11
-- Configuring done
-- Generating done
-- Build files have been written to: /home/giorgio/Documents/Conan/RocksDB/test_package/build/af2b217d2219eeb85686ff6b94a7f139d8ef9c65
Scanning dependencies of target rocks_sample
[ 50%] Building CXX object CMakeFiles/rocks_sample.dir/rocksdb.cpp.o
[100%] Linking CXX executable bin/rocks_sample
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(env_posix.o): In function `rocksdb::(anonymous namespace)::PosixEnv::WaitForJoin()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/env/env_posix.cc:925: undefined reference to `pthread_join'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(env_posix.o): In function `rocksdb::(anonymous namespace)::PosixEnv::~PosixEnv()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/env/env_posix.cc:130: undefined reference to `pthread_join'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(env_posix.o): In function `~PosixEnv':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/env/env_posix.cc:130: undefined reference to `pthread_join'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(env_posix.o): In function `rocksdb::(anonymous namespace)::PosixEnv::StartThread(void (*)(void*), void*)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/env/env_posix.cc:916: undefined reference to `pthread_create'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::Mutex::Mutex(bool)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:44: undefined reference to `pthread_mutexattr_init'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:45: undefined reference to `pthread_mutexattr_settype'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:49: undefined reference to `pthread_mutexattr_destroy'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::RWMutex::RWMutex()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:126: undefined reference to `pthread_rwlock_init'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::RWMutex::~RWMutex()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:129: undefined reference to `pthread_rwlock_destroy'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::RWMutex::ReadLock()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:131: undefined reference to `pthread_rwlock_rdlock'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::RWMutex::WriteLock()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:133: undefined reference to `pthread_rwlock_wrlock'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::RWMutex::ReadUnlock()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:135: undefined reference to `pthread_rwlock_unlock'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::RWMutex::WriteUnlock()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:137: undefined reference to `pthread_rwlock_unlock'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(port_posix.o): In function `rocksdb::port::InitOnce(int*, void (*)())':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/port/port_posix.cc:163: undefined reference to `pthread_once'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(compaction_job.o): In function `std::thread::thread<void (rocksdb::CompactionJob::*)(rocksdb::CompactionJob::SubcompactionState*), rocksdb::CompactionJob*, rocksdb::CompactionJob::SubcompactionState*>(void (rocksdb::CompactionJob::*&&)(rocksdb::CompactionJob::SubcompactionState*), rocksdb::CompactionJob*&&, rocksdb::CompactionJob::SubcompactionState*&&)':
/usr/include/c++/7.2.1/thread:126: undefined reference to `pthread_create'
/usr/include/c++/7.2.1/thread:126: undefined reference to `pthread_create'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(options_helper.o): In function `rocksdb::ZSTD_Supported()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:84: undefined reference to `ZSTD_versionNumber'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(format.o): In function `rocksdb::ZSTD_Uncompress(char const*, unsigned long, int*, rocksdb::Slice const&)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:775: undefined reference to `ZSTD_createDCtx'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:776: undefined reference to `ZSTD_decompress_usingDict'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:779: undefined reference to `ZSTD_freeDCtx'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(format.o): In function `rocksdb::Snappy_GetUncompressedLength(char const*, unsigned long, unsigned long*)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:170: undefined reference to `snappy::GetUncompressedLength(char const*, unsigned long, unsigned long*)'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(format.o): In function `rocksdb::Snappy_Uncompress(char const*, unsigned long, char*)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:179: undefined reference to `snappy::RawUncompress(char const*, unsigned long, char*)'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(threadpool_imp.o): In function `std::thread::thread<void* (*)(void*), rocksdb::BGThreadMetadata*>(void* (*&&)(void*), rocksdb::BGThreadMetadata*&&)':
/usr/include/c++/7.2.1/thread:126: undefined reference to `pthread_create'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(threadpool_imp.o): In function `rocksdb::ThreadPoolImpl::Impl::StartBGThreads()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/util/threadpool_imp.cc:303: undefined reference to `pthread_setname_np'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(column_family.o): In function `rocksdb::ZSTD_Supported()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:84: undefined reference to `ZSTD_versionNumber'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:84: undefined reference to `ZSTD_versionNumber'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(db_impl.o): In function `rocksdb::ZSTD_Supported()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:84: undefined reference to `ZSTD_versionNumber'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(version_builder.o): In function `std::thread::thread<std::function<void ()>&>(std::function<void ()>&)':
/usr/include/c++/7.2.1/thread:126: undefined reference to `pthread_create'
/usr/include/c++/7.2.1/thread:126: undefined reference to `pthread_create'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(block_based_table_builder.o): In function `rocksdb::ZSTD_Compress(rocksdb::CompressionOptions const&, char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, rocksdb::Slice const&)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:738: undefined reference to `ZSTD_compressBound'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:742: undefined reference to `ZSTD_createCCtx'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:743: undefined reference to `ZSTD_compress_usingDict'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:746: undefined reference to `ZSTD_freeCCtx'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(block_based_table_builder.o): In function `rocksdb::Snappy_Compress(rocksdb::CompressionOptions const&, char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:157: undefined reference to `snappy::MaxCompressedLength(unsigned long)'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/./util/compression.h:159: undefined reference to `snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(delete_scheduler.o): In function `std::thread::thread<void (rocksdb::DeleteScheduler::*)(), rocksdb::DeleteScheduler*>(void (rocksdb::DeleteScheduler::*&&)(), rocksdb::DeleteScheduler*&&)':
/usr/include/c++/7.2.1/thread:126: undefined reference to `pthread_create'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(thread_local.o): In function `rocksdb::ThreadLocalPtr::StaticMeta::StaticMeta()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/util/thread_local.cc:304: undefined reference to `pthread_key_create'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(thread_local.o): In function `rocksdb::ThreadLocalPtr::StaticMeta::GetThreadLocal()':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/util/thread_local.cc:379: undefined reference to `pthread_setspecific'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(thread_local.o): In function `rocksdb::ThreadLocalPtr::StaticMeta::OnThreadExit(void*)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/util/thread_local.cc:283: undefined reference to `pthread_setspecific'
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/package/04f6580e02f3d5f0e28569ff942937d4c741700b/lib/librocksdb.a(malloc_stats.o): In function `rocksdb::DumpMallocStats(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
/home/giorgio/.conan/data/RocksDB/5.8/RocksDB/5.8/build/04f6580e02f3d5f0e28569ff942937d4c741700b/rocksdb-5.8/db/malloc_stats.cc:46: undefined reference to `malloc_stats_print'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/rocks_sample.dir/build.make:95: bin/rocks_sample] Error 1
make[1]: *** [CMakeFiles/Makefile2:68: CMakeFiles/rocks_sample.dir/all] Error 2
make: *** [Makefile:84: all] Error 2
ERROR: PROJECT: Error in build() method, line 12
        cmake.build()
        ConanException: Error 512 while executing cmake --build './' '--' '-j8'

I naively tried to add

env_build.libs.append("pthread")

with little hope for it to succeed: because it is a static library being built it is CMake when building the sample that needs to have this info, not while building RocksDB itself.

The line just above that, trying to set fpic, is because I had issues when attempting a shared library build. Is there a more considerate way to enforce this for shared libraries?

Thanks
Giorgio

@memsharded
Copy link
Member

Hi Giorgio!

If RocksDB needs to link with pthread in linux, I would add to my RocksDB recipe:

def package_info(self):
        self.cpp_info.libs = ["rocksdb"]
        if self.settings.os == "Linux":
             self.cpp_info.libs.append("pthread")

Consumers will automatically get the linking to pthread, so they don't need to remember to add it manually. Is this what you are looking for? The info from package_info() is for consumers, not for the package itself, this might be the reason the package builds correctly, but then the consumer fails to link.

Please tell me if this helps. Thanks for the feedback!

@borgoat
Copy link
Author

borgoat commented Dec 27, 2017

Thanks James! It looks like it fixed that.
It seems now I'll have to pack two other dependencies of RocksDB before I can proceed though... libsnappy and zstandard...

Actually, I think those are optional dependencies, but the Makefile finds them on my workstation and tries to pull them in, it then fails at link time (just like it happened with pthread, because they're not being referenced by Conan). So the question is, how do you make sure that your build environment is clean? I suppose you just let the CI server do the job, or you have some other suggested path for local development?

Another unrelated question: RocksDB requires C++11. Right now I've been setting it like this

conan create RocksDB/5.8 -s compiler.libcxx=libstdc++11

Is there a way to set this in the conanfile.py?

Thanks for the help
Giorgio

@memsharded
Copy link
Member

Hi Giorgio!

So the question is, how do you make sure that your build environment is clean?

That is a very good question. Typically in CI it is common to have a totally clean environment, which can be done at the job level, installing conan each time in a new enviroment, or if usin the same installed conan, you can set a new CONAN_USER_HOME for each execution pointing to a temporary folder, for example.

However, that doesn't remove the problem of dependencies that come at the system level and build systems that might locate them and link them. There might be different approaches:

  • If the target/run system is equal to the develop one, then those dependencies will still be there, so it should still work
  • If the system dependencies are not installed by default, maybe conanfile.py system_requirements() and SystemPackageTool helper might be useful to install them when the package is installed
  • Many users in nix systems use docker exactly to fully control the environment, and enhance reproducibility. We ourselves use docker in the conan-package-tools repo to use different versions of gcc to create OSS packages, so this might be handy.
  • Then, when those system packages are to be replaced with conan packages, it is important to check that they are the ones being linked. Dependending on the technology is different, with cmake, it can be checked in CMakeLists, if using legacy makefiles, maybe the best is to run some tool like ldd over the binaries and check the paths to linked libraries.

What you are setting is libcxx which is the standard library version you are using, which is not exactly the same as setting the C++ standard. This is something that is set typically in the build system directly, as it is quite a constant for a given library and given compiler versions. If you want to create different binaries for different standards, probably the way is to create an option for the package. Look in: conan-io/conan#2042, there is some ongoing work to standardize this, but hasn't been merged because it is a bit complicated and controversial, so it might need further work (to standardize, the idea to use options for your package and you manage it is totally fine). PLease ask if you need further explanation for this pull request.

Cheers!

@gocarlos
Copy link

@Croydon
Copy link
Contributor

Croydon commented Jun 13, 2020

Done via conan-io/conan-center-index#1389

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants