Skip to content
This repository has been archived by the owner on Dec 8, 2021. It is now read-only.

Support non-uniform batch size #114

Merged
merged 8 commits into from
Jan 14, 2019
Merged

Support non-uniform batch size #114

merged 8 commits into from
Jan 14, 2019

Commits on Jan 9, 2019

  1. mydump: non-uniform batch size

    @lonng
    lonng committed Jan 9, 2019
    Configuration menu
    Copy the full SHA
    451015d View commit details
    Browse the repository at this point in the history

  2. *: make the batch-size-scale configurable

    @lonng
    lonng committed Jan 9, 2019
    Configuration menu
    Copy the full SHA
    9d121ba View commit details
    Browse the repository at this point in the history

  3. *: implemented the optimized non-uniform strategy

    @kennytm
    kennytm committed Jan 9, 2019
    Configuration menu
    Copy the full SHA
    5e68de6 View commit details
    Browse the repository at this point in the history

  4. tests: due to change of strategy, checkpoint_engines count becomes 4 … 

    …again
    @kennytm
    kennytm committed Jan 9, 2019
    Configuration menu
    Copy the full SHA
    38f82b7 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2019

  1. mydump/region: slightly adjust the batch size computation 

    * Use the exact result of 1/Beta(N, R) instead of an approximation
* When the number of engines is small and the total engine size of the
  first (table-concurrency) batches exceed the table size, the last batch
  was truncated, and disrupt the pipeline. Now in these case we will
  reduce the batch size to avoid this disruption.
    @kennytm
    kennytm committed Jan 10, 2019
    Configuration menu
    Copy the full SHA
    7e517bd View commit details
    Browse the repository at this point in the history

  2. restore: log the SQL size and KV size of each engine for debugging

    @kennytm
    kennytm committed Jan 10, 2019
    Configuration menu
    Copy the full SHA
    cd82c4c View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2019

  1. config: change default batch size and ratio given experiment result

    @kennytm
    kennytm committed Jan 12, 2019
    Configuration menu
    Copy the full SHA
    62d8ee3 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2019

  1. config: added more explanation about batch-import-ratio

    @kennytm
    kennytm committed Jan 13, 2019
    Configuration menu
    Copy the full SHA
    cd63d6d View commit details
    Browse the repository at this point in the history