Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: introduce the interface of RemoteJobScheduler #4181

Merged
merged 72 commits into from
Jul 2, 2024
Merged

feat: introduce the interface of RemoteJobScheduler #4181

merged 72 commits into from
Jul 2, 2024
+445 −107

Commits on Jun 6, 2024

  1. refactor: add Compactor trait

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    0d5b790 View commit details
    Browse the repository at this point in the history

  2. chore: add compact() in Compactor trait and expose compaction module

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a91c5d6 View commit details
    Browse the repository at this point in the history

  3. refactor: add CompactionRequest and open_compaction_region

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    37b0464 View commit details
    Browse the repository at this point in the history

  4. refactor: export the compaction api

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    014bc22 View commit details
    Browse the repository at this point in the history

  5. refactor: add DefaultCompactor::new_from_request

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    f9d1e66 View commit details
    Browse the repository at this point in the history

  6. refactor: no need to pass mito_config in open_compaction_region()

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    3eb0dab View commit details
    Browse the repository at this point in the history

  7. refactor: CompactionRequest -> &CompactionRequest

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    8cd55a2 View commit details
    Browse the repository at this point in the history

  8. fix: typo

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    891426d View commit details
    Browse the repository at this point in the history

  9. docs: add docs for public apis

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    8ff9c6f View commit details
    Browse the repository at this point in the history

  10. refactor: remove 'Picker' from Compactor

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    acabc07 View commit details
    Browse the repository at this point in the history

  11. chore: add logs

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    de8496c View commit details
    Browse the repository at this point in the history

  12. chore: change pub attribute for Picker

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    6bee93e View commit details
    Browse the repository at this point in the history

  13. refactor: remove do_merge_ssts()

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a63c9f5 View commit details
    Browse the repository at this point in the history

  14. refactor: update comments

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    3a360c0 View commit details
    Browse the repository at this point in the history

  15. refactor: use CompactionRegion argument in Picker

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    35feb02 View commit details
    Browse the repository at this point in the history

  16. chore: make compaction module public and remove unnessary clone

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    55033f9 View commit details
    Browse the repository at this point in the history

  17. refactor: move build_compaction_task() in CompactionScheduler{}

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    15c102c View commit details
    Browse the repository at this point in the history

  18. chore: use in open_compaction_region() and add some comments for publ… 

    …ic structure
    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    8ad6fff View commit details
    Browse the repository at this point in the history

  19. refactor: add 'manifest_dir()' in store-api

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    26ae2a9 View commit details
    Browse the repository at this point in the history

  20. refactor: move the default implementation to DefaultCompactor

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    9a05b6a View commit details
    Browse the repository at this point in the history

  21. refactor: remove Options from MergeOutput

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    130fb90 View commit details
    Browse the repository at this point in the history

  22. chore: minor modification

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    e141e91 View commit details
    Browse the repository at this point in the history

  23. fix: clippy errors

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    e3e6f12 View commit details
    Browse the repository at this point in the history

  24. fix: unit test errors

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    93f5a74 View commit details
    Browse the repository at this point in the history

  25. refactor: remove 'manifest_dir()' from store-api crate(already have o… 

    …ne in opener)
    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    17bf852 View commit details
    Browse the repository at this point in the history

  26. refactor: use 'region_dir' in CompactionRequest

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    48f9398 View commit details
    Browse the repository at this point in the history

  27. refactor: refine naming

    @zyy17
    zyy17 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    2f6d9ef View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. chore: sync main branch

    @zyy17
    zyy17 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    861ea41 View commit details
    Browse the repository at this point in the history

  2. refactor: refine naming

    @zyy17
    zyy17 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    d26c64a View commit details
    Browse the repository at this point in the history

  3. refactor: remove clone()

    @zyy17
    zyy17 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    4968186 View commit details
    Browse the repository at this point in the history

  4. chore: add comments

    @zyy17
    zyy17 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    574fd9f View commit details
    Browse the repository at this point in the history

  5. refactor: add PickerOutput field in CompactorRequest

    @zyy17
    zyy17 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    77c085c View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2024

  1. chore: sync main branch

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    f8d88a2 View commit details
    Browse the repository at this point in the history

  2. feat: introduce RemoteJobScheduler

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    f92c729 View commit details
    Browse the repository at this point in the history

  3. feat: add RemoteJobScheudler in schedule_compaction_request()

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    c0c7b6d View commit details
    Browse the repository at this point in the history

  4. refactor: use Option type for senders field of CompactionFinished

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    84a8ff7 View commit details
    Browse the repository at this point in the history

  5. refactor: modify CompactionJob

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    f421d08 View commit details
    Browse the repository at this point in the history

  6. refactor: schedule remote compaction job by options

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    6f24668 View commit details
    Browse the repository at this point in the history

  7. refactor: remove unused Options

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    cbe2115 View commit details
    Browse the repository at this point in the history

  8. build: remove unused log

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    66ee079 View commit details
    Browse the repository at this point in the history

  9. refactor: fallback to local compaction if the remote compaction failed

    @zyy17
    zyy17 committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    3c183ff View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. fix: clippy errors

    @zyy17
    zyy17 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    90a4794 View commit details
    Browse the repository at this point in the history

  2. chore: sync main branch

    @zyy17
    zyy17 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    c67b9ad View commit details
    Browse the repository at this point in the history

  3. refactor: add plugins in mito2

    @zyy17
    zyy17 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    0f5c4d3 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. refactor: add from_u64() for JobId

    @zyy17
    zyy17 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    6ffdae2 View commit details
    Browse the repository at this point in the history

  2. refactor: make schedule module public

    @zyy17
    zyy17 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    06fc1e4 View commit details
    Browse the repository at this point in the history

  3. refactor: add error for RemoteJobScheduler

    @zyy17
    zyy17 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    01b20b2 View commit details
    Browse the repository at this point in the history

  4. refactor: add Notifier

    @zyy17
    zyy17 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    b740c41 View commit details
    Browse the repository at this point in the history

  5. refactor: use Arc for Notifier

    @zyy17
    zyy17 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    4404973 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. chore: sync main branch

    @zyy17
    zyy17 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    25bfdf9 View commit details
    Browse the repository at this point in the history

  2. refactor: add 'remote_compaction' in compaction options

    @zyy17
    zyy17 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    1d6efa4 View commit details
    Browse the repository at this point in the history

  3. fix: clippy errors

    @zyy17
    zyy17 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    07459d3 View commit details
    Browse the repository at this point in the history

  4. fix: unrecognized table option

    @zyy17
    zyy17 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    d08b56e View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. refactor: add 'start_time' in CompactionJob

    @zyy17
    zyy17 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    840e642 View commit details
    Browse the repository at this point in the history

  2. refactor: modify error type of RemoteJobScheduler

    @zyy17
    zyy17 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    fc0e927 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. Merge branch 'main' into feat/add-experimental-remote-job-scheduler

    @zyy17
    zyy17 authored Jun 24, 2024
    Configuration menu
    Copy the full SHA
    2731e76 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. chore: revert changes for request

    @zyy17
    zyy17 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    1a8a742 View commit details
    Browse the repository at this point in the history

  2. refactor: code refactor by review comment

    @zyy17
    zyy17 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    bf55576 View commit details
    Browse the repository at this point in the history

  3. refactor: use string type for JobId

    @zyy17
    zyy17 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    617f8f4 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feat/add-experimental-remote-job-scheduler

    @zyy17
    zyy17 authored Jun 26, 2024
    Configuration menu
    Copy the full SHA
    f220543 View commit details
    Browse the repository at this point in the history

  5. refactor: add 'waiters' field in DefaultNotifier

    @zyy17
    zyy17 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    2f92206 View commit details
    Browse the repository at this point in the history

  6. fix: build error

    @zyy17
    zyy17 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    fbef517 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. refactor: take coderabbit's review comment

    @zyy17
    zyy17 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    613edc8 View commit details
    Browse the repository at this point in the history

  2. refactor: use uuid::Uuid as JobId

    @zyy17
    zyy17 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    4e3ccdf View commit details
    Browse the repository at this point in the history

  3. refactor: return waiters when schedule failed and add on_failure for … 

    …DefaultNotifier
    @zyy17
    zyy17 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    d299592 View commit details
    Browse the repository at this point in the history

  4. refactor: move waiters from notifier to Job

    @zyy17
    zyy17 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    ca1c8fa View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. refactor: use ObjectStoreManagerRef in open_compaction_region()

    @zyy17
    zyy17 committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    5246a22 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. Merge branch 'main' into feat/add-experimental-remote-job-scheduler

    @zyy17
    zyy17 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    a9b2d5e View commit details
    Browse the repository at this point in the history

  2. refactor: implement for JobId and adds related unit tests

    @zyy17
    zyy17 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    6800383 View commit details
    Browse the repository at this point in the history

  3. fix: run unit tests failed

    @zyy17
    zyy17 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    6502caa View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feat/add-experimental-remote-job-scheduler

    @zyy17
    zyy17 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    305ee27 View commit details
    Browse the repository at this point in the history

  5. refactor: add RemoteJobSchedulerError

    @zyy17
    zyy17 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    0f9335d View commit details
    Browse the repository at this point in the history