Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partitioning database for block and transaction #1194

Merged
merged 13 commits into from
Mar 5, 2021

Conversation

riemannulus
Copy link
Member

@riemannulus riemannulus commented Feb 25, 2021

Snapshots are getting bigger. As of Feb 17 and 2021, it maintains about 1.0-1.1GB, which means that 1.0-1.1GB of files are downloaded every time the user plays the game. It is also a problem that even if the DAU becomes 2 million, about 200GB of CF CDN fee will be charged, but receiving 1.1GB of files every time is indeed a burden for users.

For this reason, there is a need to partition the database so that it can incrementally receive snapshots.

This PR is making transactions and blocks stored in a partitioned database.

@riemannulus riemannulus self-assigned this Feb 25, 2021
@riemannulus riemannulus linked an issue Feb 25, 2021 that may be closed by this pull request
@codecov
Copy link

codecov bot commented Feb 25, 2021

Codecov Report

Merging #1194 (f32b31b) into main (95a8409) will decrease coverage by 0.08%.
The diff coverage is 76.15%.

@@            Coverage Diff             @@
##             main    #1194      +/-   ##
==========================================
- Coverage   87.49%   87.41%   -0.09%     
==========================================
  Files         360      360              
  Lines       31607    31724     +117     
==========================================
+ Hits        27656    27730      +74     
- Misses       2175     2207      +32     
- Partials     1776     1787      +11     
Impacted Files Coverage Δ
Libplanet.RocksDBStore/RocksDBStore.cs 88.12% <75.51%> (-5.37%) ⬇️
Libplanet.RocksDBStore/RocksDBUtils.cs 100.00% <100.00%> (ø)
Libplanet/Hashcash.cs 94.44% <0.00%> (-5.56%) ⬇️
Libplanet/Store/DefaultStore.cs 86.78% <0.00%> (-1.50%) ⬇️
Libplanet/Net/Swarm.cs 81.40% <0.00%> (-1.38%) ⬇️
Libplanet/Net/BlockCompletion.cs 94.25% <0.00%> (-0.87%) ⬇️
Libplanet.Tests/Net/SwarmTest.Preload.cs 98.72% <0.00%> (-0.57%) ⬇️
Libplanet/Net/Protocols/KademliaProtocol.cs 78.01% <0.00%> (+0.38%) ⬆️
Libplanet/Net/NetMQTransport.cs 80.29% <0.00%> (+2.20%) ⬆️
Libplanet.RocksDBStore/RocksDBStoreBitConverter.cs 100.00% <0.00%> (+26.08%) ⬆️

Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
CHANGES.md Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
@riemannulus riemannulus changed the title Using distributed database for block and transaction Partitioning database for block and transaction Feb 26, 2021
Lee Suho and others added 5 commits February 26, 2021 18:27
@riemannulus
Copy link
Member Author

PTAL 🙏 @libplanet

longfin
longfin previously approved these changes Mar 3, 2021
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
Libplanet.RocksDBStore/RocksDBStore.cs Outdated Show resolved Hide resolved
@longfin longfin requested a review from area363 March 4, 2021 06:34
limebell
limebell previously approved these changes Mar 4, 2021
Co-authored-by: Swen Mun
<swen@planetariumhq.com>
@riemannulus riemannulus dismissed stale reviews from limebell and longfin via 01976b2 March 5, 2021 04:25
@riemannulus
Copy link
Member Author

PTAL 🙏 @planetarium/libplanet

@riemannulus riemannulus merged commit d1383b4 into planetarium:main Mar 5, 2021
Comment on lines +78 to +81
/// <param name="txEpochUnitSeconds">The number of Transaction to store in DB
/// for each epoch. 86400 by default.</param>
/// <param name="blockEpochUnitSeconds">The number of block to store in DB
/// for each epoch. 86400 by default.</param>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs seem incorrect; aren't they seconds (not # of txs/blocks)?

The number of Transaction

The number of block

dahlia added a commit to dahlia/libplanet that referenced this pull request Mar 5, 2021
dahlia added a commit to dahlia/libplanet that referenced this pull request Apr 14, 2021
dahlia added a commit to dahlia/libplanet that referenced this pull request Apr 14, 2021
dahlia added a commit to dahlia/libplanet that referenced this pull request Apr 15, 2021
dahlia added a commit to dahlia/libplanet that referenced this pull request Apr 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Multi-database for blocks and tx
5 participants