Skip to content
This repository has been archived by the owner on Dec 8, 2021. It is now read-only.

Support multi-engine per table (batching) #113

Merged
merged 13 commits into from
Jan 14, 2019
Merged

Support multi-engine per table (batching) #113

merged 13 commits into from
Jan 14, 2019

Commits on Jan 2, 2019

  1. config,restore: introduced [mydumper] batch-size

    Removed `[tikv-importer] batch-size` to avoid confusion.
    
    Removed `[mydumper] min-region-size` since it is useless now.
    kennytm committed Jan 2, 2019
    Configuration menu
    Copy the full SHA
    44dc8d1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e61bacf View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8e00a50 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c621266 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    142dd9a View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8b2498c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    ed0471a View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    485fc9d View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    cc34c4e View commit details
    Browse the repository at this point in the history
  10. *: fixed code

    kennytm committed Jan 2, 2019
    Configuration menu
    Copy the full SHA
    c7f0a37 View commit details
    Browse the repository at this point in the history
  11. *: addressed comments

    kennytm committed Jan 2, 2019
    Configuration menu
    Copy the full SHA
    294f228 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2019

  1. *: addressed comments

    kennytm committed Jan 3, 2019
    Configuration menu
    Copy the full SHA
    92e8dad View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2019

  1. Support non-uniform batch size (#114)

    * mydump: non-uniform batch size
    
    * *: make the `batch-size-scale` configurable
    
    * *: implemented the optimized non-uniform strategy
    
    * tests: due to change of strategy, checkpoint_engines count becomes 4 again
    
    * mydump/region: slightly adjust the batch size computation
    
    * Use the exact result of 1/Beta(N, R) instead of an approximation
    * When the number of engines is small and the total engine size of the
      first (table-concurrency) batches exceed the table size, the last batch
      was truncated, and disrupt the pipeline. Now in these case we will
      reduce the batch size to avoid this disruption.
    
    * restore: log the SQL size and KV size of each engine for debugging
    
    * config: change default batch size and ratio given experiment result
    
    * config: added more explanation about batch-import-ratio
    lonng authored and kennytm committed Jan 14, 2019
    Configuration menu
    Copy the full SHA
    8ba38ed View commit details
    Browse the repository at this point in the history