Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LMDB read-only transaction limit fix #1319

Merged
merged 1 commit into from
Oct 19, 2014
Merged

LMDB read-only transaction limit fix #1319

merged 1 commit into from
Oct 19, 2014

Conversation

kmatzen
Copy link
Contributor

@kmatzen kmatzen commented Oct 18, 2014

I incorrectly assumed that you could have many concurrent read-only transactions with LMDB. This preallocates a single read-only transaction that can be reused until commit has been called.

This still leaks cursors. I need to find out whether or not those are automatically cleaned up when a transaction aborts or commits.

New test cases included.

…preallocates one read-only transaction and reuses it. It's very important that iterations are considered invalid after a commit has been performed.
@seanbell
Copy link

This patch seems to fix some failed database unit tests (from #1238). Without this patch, I get an error in the "dynamic" linked tests. It happens in multiple tests. One example error that happens if you use #1238 but not #1319:

[ RUN      ] DataLayerTest/1.TestReadCropTrainSequenceUnseededLMDB
I1015 19:14:43.326297 18597 test_data_layer.cpp:41] Using temporary dataset /tmp/caffe_test.8SJB3R/db
I1015 19:14:43.328063 18597 data_layer.cpp:31] Opening dataset /tmp/caffe_test.8SJB3R/db
*** Aborted at 1413414883 (unix time) try "date -d @1413414883" if you are using GNU date ***
PC: @     0x2b73849ed8c7 __pthread_mutex_lock_full
*** SIGSEGV (@0x2b7390cda058) received by PID 18597 (TID 0x2b73907bdb00) from PID 18446744071843979352; stack trace: ***
    @     0x2b73849f7cb0 (unknown)
    @     0x2b73849ed8c7 __pthread_mutex_lock_full
    @     0x2b73829502a6 mdb_txn_renew0
    @     0x2b73829512ef mdb_txn_begin
    @           0x6dd56d caffe::LmdbDataset<>::open()
    @           0x76e1f2 caffe::DataLayer<>::DataLayerSetUp()
    @           0x77f5b3 caffe::BaseDataLayer<>::LayerSetUp()
    @           0x77f5c9 caffe::BasePrefetchingDataLayer<>::LayerSetUp()
    @           0x5e21d8 caffe::DataLayerTest<>::TestReadCropTrainSequenceUnseeded()
    @           0x69266d testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x68a441 testing::Test::Run()
    @           0x68a526 testing::TestInfo::Run()
    @           0x68a667 testing::TestCase::Run()
    @           0x68a9be testing::internal::UnitTestImpl::RunAllTests()
    @           0x6921ed testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x689a9e testing::UnitTest::Run()
    @           0x46ddd0 main
    @     0x2b7384c2676d (unknown)
    @           0x4738bd (unknown)
Segmentation fault (core dumped)
make: *** [runtest] Error 139

@shelhamer
Copy link
Member

Merging as partial fix while the full fix and interface simplification is done according to the plan in the #1238 post-merge discussion.

shelhamer added a commit that referenced this pull request Oct 19, 2014
LMDB read-only transaction limit fix
@shelhamer shelhamer merged commit a9ac68a into BVLC:dev Oct 19, 2014
RazvanRanca pushed a commit to RazvanRanca/caffe that referenced this pull request Nov 4, 2014
LMDB read-only transaction limit fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants