Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MTG-12] merging the asset data into a single column with change to flatbuffer for serialization #298

Open
wants to merge 41 commits into
base: main
Choose a base branch
from
Open

[MTG-12] merging the asset data into a single column with change to flatbuffer for serialization #298

wants to merge 41 commits into from

Commits on Oct 10, 2024

  1. drop the mutex as soon as needed

    @StanChe
    StanChe committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    e79ccd4 View commit details
    Browse the repository at this point in the history

  2. spawn_blocking instead of the regular spawn for blocking code

    @StanChe
    StanChe committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    9e63ab2 View commit details
    Browse the repository at this point in the history

  3. removed parallel processing from synchronizer reading

    @StanChe
    StanChe committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    91bfdee View commit details
    Browse the repository at this point in the history

  4. More metrics for synchronizer (#281) 

    * feat: add more metrics to the synchronizer

* feat: more metrics
    @n00m4d
    n00m4d authored Oct 10, 2024
    Configuration menu
    Copy the full SHA
    b629102 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. MTG-742 Move fungible assets dump to different thread (#282) 

    * feat: move fung dump to different thread

* feat: add more metrics
    @n00m4d
    n00m4d authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    52e511a View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. MTG-747 Save updates for fungible assets during regular sync (#283) 

    * feat: save fungible updates in regular sync

* feat: change migration file and add indexes drop/create during dump load

* fix: fungible tokens

* chore: code style
    @n00m4d
    n00m4d authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    f61d79d View commit details
    Browse the repository at this point in the history

  2. feat: pass optional hashMap to get_asset_indexes

    @n00m4d
    n00m4d committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    06d85a8 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. fix: optimize memory usage

    @n00m4d
    n00m4d committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    fbc242d View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. a complete rework of the rocks storage 

    Now all the assets-related data is persisted in a single column
    @StanChe
    StanChe committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    78b291f View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. renamed the merge function

    @StanChe
    StanChe committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    d1963f0 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. flatbuffer types + merged types + optimized dump client = 30% improve… 

    …ment over the reference, or over 50% over the initial merge
    @StanChe
    StanChe committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    824fc1f View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. dump client optimized further, asset client refactored 

    Current benchmarking results for 1M assets:
Dumping Group/2k batch size
                        time:   [4.8857 s 4.9288 s 4.9785 s]
Dumping Group/get_assets
                        time:   [5.0407 ms 5.0694 ms 5.0968 ms]
Dumping Group/get_assets_individually (taking 1000 assets in a loop one after another)
                        time:   [31.137 ms 33.730 ms 38.050 ms]
    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    d978784 View commit details
    Browse the repository at this point in the history

  2. batch client using flatbuffer

    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    a05855d View commit details
    Browse the repository at this point in the history

  3. insert_gaped_data updated in batch client

    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    d7d9b6b View commit details
    Browse the repository at this point in the history

  4. minor refactor

    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    31aca0f View commit details
    Browse the repository at this point in the history

  5. more benchmarks

    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    edcff2f View commit details
    Browse the repository at this point in the history

  6. get_asset_indexes optimized

    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    936ebda View commit details
    Browse the repository at this point in the history

  7. full refactor to use flatbuffer

    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    ff7e94e View commit details
    Browse the repository at this point in the history

  8. code cleanup

    @StanChe
    StanChe committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    89a6571 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. crazy merge function

    @StanChe
    StanChe committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    73531ff View commit details
    Browse the repository at this point in the history

  2. moved the code around

    @StanChe
    StanChe committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    4221d94 View commit details
    Browse the repository at this point in the history

  3. merge function tests

    @StanChe
    StanChe committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    522cb1a View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. more randomized assets generation in tests

    @StanChe
    StanChe committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    885fbba View commit details
    Browse the repository at this point in the history

  2. another attempt to the merge function

    @StanChe
    StanChe committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    3ea1df2 View commit details
    Browse the repository at this point in the history

  3. using the recent iteration of merge function

    @StanChe
    StanChe committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    de2e42f View commit details
    Browse the repository at this point in the history

  4. Merge commit '88e5d12b86483126a5aa571072dca93b5d4df52f' into feature/… 

    …MTG-12-flatbuffer-merge

# Conflicts:
#	nft_ingester/src/api/dapi/change_logs.rs
#	rocks-db/src/batch_client.rs
#	rocks-db/src/dump_client.rs
#	rocks-db/src/storage_traits.rs
    @StanChe
    StanChe committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    98eb391 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. updated streaming client to use flatbuffers

    @StanChe
    StanChe committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    44df86e View commit details
    Browse the repository at this point in the history

  2. fixed tests

    @StanChe
    StanChe committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    53c4a68 View commit details
    Browse the repository at this point in the history

  3. fixed a test and dropped a debug line from another test 

    Added slot to data generation in tests
added a valid onchain data in test generation
fixed a no longer valid test
    @StanChe
    StanChe committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    ce73e95 View commit details
    Browse the repository at this point in the history

  4. cleanup benches

    @StanChe
    StanChe committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    1e25381 View commit details
    Browse the repository at this point in the history

  5. fixed the imports

    @StanChe
    StanChe committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    5247299 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. fixed migrator, some tests

    @StanChe
    StanChe committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    08062da View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. fixed an issue with mpl core collections getting into response with m… 

    …etadata. fixed another test setup
    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    ab77ec4 View commit details
    Browse the repository at this point in the history

  2. dump client fixed for fungibles

    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    7342535 View commit details
    Browse the repository at this point in the history

  3. fix for tests setup accounting for authority being optional in some t… 

    …ests
    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    4123d05 View commit details
    Browse the repository at this point in the history

  4. Merge commit '962eaf9560de83ae1f917ec168b461666c3d7000' into feature/… 

    …MTG-12-flatbuffer-merge + updated the data read to get dynamic details

# Conflicts:
#	rocks-db/src/bin/leaf_checker/main.rs
    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    fd0ca32 View commit details
    Browse the repository at this point in the history

  5. making fmt and clippy happier

    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    cae65de View commit details
    Browse the repository at this point in the history

  6. less randomness in tests

    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    620d842 View commit details
    Browse the repository at this point in the history

  7. dropped some commented out code

    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    3474b40 View commit details
    Browse the repository at this point in the history

  8. Merge commit 'b410996aaf3584ed572b2dea699d2a66f94359b3' into feature/… 

    …MTG-12-flatbuffer-merge

# Conflicts:
#	rocks-db/Cargo.toml
    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    d8855c4 View commit details
    Browse the repository at this point in the history

  9. Merge commit 'd2f3e0af6b2cb438a77f45e640e2e5731c1f0977' into feature/… 

    …MTG-12-flatbuffer-merge

# Conflicts:
#	nft_ingester/tests/process_accounts.rs
    @StanChe
    StanChe committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    d4baf7d View commit details
    Browse the repository at this point in the history