rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code #356

sumedhak27 · 2022-07-20T09:11:31Z

For Garbage Collection, cortx-rgw needs a MotrGC thread which
works concurrently to delete all the stale objects.
Added the MotrGC class and the interface to configure and start
GC workers.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

src/rgw/motr/gc/gc.cc

src/rgw/motr/gc/gc.h

sachinpunadikar

work on suggestions.
I hope gc.cc & gc.h are getting compiled / linked, else need to add these files in CMakeLists.txt

For Garbage Collection, cortx-rgw needs a MotrGC thread which works concurrently to delete all the stale objects. Added the MotrGC class and the interface to configure and start GC workers. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

jjxsg · 2022-07-20T11:57:47Z

src/rgw/CMakeLists.txt

@@ -177,7 +177,7 @@ if(WITH_RADOSGW_DBSTORE)
  list(APPEND librgw_common_srcs rgw_sal_dbstore.cc)
 endif()
 if(WITH_RADOSGW_MOTR)
-  list(APPEND librgw_common_srcs rgw_sal_motr.cc)
+  list(APPEND librgw_common_srcs rgw_sal_motr.cc motr/gc/gc.cc)


We will need to add more files in future. Better to add cmakelist in motr directory and add that directory here.
ref. https://github.com/Seagate/cortx-rgw/blob/main/src/rgw/store/dbstore/CMakeLists.txt

Yes, I tried adding a subdirectory, but the compilation failed initially, I did not try to resolve it.
Will try it again in the end.

The below code works but I have to repeat the lines from src/rgw/CMakeLists.txt in the new file,
does anyone know how to avoid that.

// file_name: src/rgw/motr/CMakeLists.txt set(cortx_rgw_srcs gc/gc.cc) add_library(cortx_rgw ${cortx_rgw_srcs}) target_include_directories(cortx_rgw PUBLIC "${CMAKE_SOURCE_DIR}/src/rgw") // *************** repeated section start *************** target_include_directories(cortx_rgw PRIVATE "/usr/include/motr") target_compile_options(cortx_rgw PRIVATE "-Wno-attributes") target_compile_definitions(cortx_rgw PRIVATE "M0_EXTERN=extern" "M0_INTERNAL=") target_link_libraries(cortx_rgw PRIVATE motr motr-helpers) // *************** repeated section end *************** set(link_targets spawn) target_link_libraries(cortx_rgw PUBLIC ${link_targets})

@andriytk @siningwuseagate can you please help here?

For supporting concurrent IOs MotrGC need to have multiple GCWorker threads. Introducing GCWorker class and changing the MotrGC interfaces to manage all the worker threads. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

sachinpunadikar

LGTM

src/rgw/rgw_sal_motr.h

src/rgw/motr/gc/gc.cc

src/rgw/motr/gc/gc.h

Implemented the logic to spawn multiple GCWorker threads on startup, and gracefully exit on shutdown. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

We are bound to create MotrMetaCache objects as we are directly using class methods during CRUD operations without checking if metadata caching is enabled or not. Earlier I was trying to not create meta cache objects which was breaking the CRUD, so now reverting it to create those objects again. NOTE: Ideally we should create MetaCache objects only if caching is enabled. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

add code to fetch the max gc indices to create. create and append the index name in gc index list. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

…ra code (#356) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code Behaviour - With Garbage Collector enabled, MotrGC will have GC indexes & GC worker threads. - GC worker threads will run for the configured max processing time and then will wait for the configured time between two consecutive runs. Additions/Changes - Add the Garbage Collector infrastructure to support the start & stop of worker threads. - MotrGC class with the interfaces to initialize(), start(), stop() and finalize(). - GCWorker class with entry() and stop() methods. - Refactor Initialization of MotrStore - add setter methods to configure MotrStore - add initialize() method to call initialization of all the other components eg. MetadataCache, GC, (in future LC, QuotaHandler), etc. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

…ra code (#356) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code Behaviour - With Garbage Collector enabled, MotrGC will have GC indexes & GC worker threads. - GC worker threads will run for the configured max processing time and then will wait for the configured time between two consecutive runs. Additions/Changes - Add the Garbage Collector infrastructure to support the start & stop of worker threads. - MotrGC class with the interfaces to initialize(), start(), stop() and finalize(). - GCWorker class with entry() and stop() methods. - Refactor Initialization of MotrStore - add setter methods to configure MotrStore - add initialize() method to call initialization of all the other components eg. MetadataCache, GC, (in future LC, QuotaHandler), etc. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com>

…ra code (Seagate#356) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code Behaviour - With Garbage Collector enabled, MotrGC will have GC indexes & GC worker threads. - GC worker threads will run for the configured max processing time and then will wait for the configured time between two consecutive runs. Additions/Changes - Add the Garbage Collector infrastructure to support the start & stop of worker threads. - MotrGC class with the interfaces to initialize(), start(), stop() and finalize(). - GCWorker class with entry() and stop() methods. - Refactor Initialization of MotrStore - add setter methods to configure MotrStore - add initialize() method to call initialization of all the other components eg. MetadataCache, GC, (in future LC, QuotaHandler), etc. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com>

…on. (Seagate#385) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code (Seagate#356) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code Behaviour - With Garbage Collector enabled, MotrGC will have GC indexes & GC worker threads. - GC worker threads will run for the configured max processing time and then will wait for the configured time between two consecutive runs. Additions/Changes - Add the Garbage Collector infrastructure to support the start & stop of worker threads. - MotrGC class with the interfaces to initialize(), start(), stop() and finalize(). - GCWorker class with entry() and stop() methods. - Refactor Initialization of MotrStore - add setter methods to configure MotrStore - add initialize() method to call initialization of all the other components eg. MetadataCache, GC, (in future LC, QuotaHandler), etc. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> * rgw: [CORTX-33179] GC Metadata Implementation (Seagate#363) Implement GC metadata structure & corresponding functions Added motr_gc_obj_info structure to hold gc metadata for an object marked for deletion. Added logic for creating GC indices based on config parameters. Signed-off-by: Jeet Jain <jeet.jain@seagate.com> * GC thread processing logic (Seagate#371) The GC thread will aquire GC index and process the object entries for deletion either upto the count governed by "rgw_gc_max_trim_chunk" or time allowed by "rgw_gc_processor_max_time". Signed-off-by: Sachin Punadikar <sachin.punadikar@seagate.com> * motr_sync: [CORTX-33691] Define and implement rgw_motr synchronization (Seagate#373) * rgw_sal_motr: [CORTX-33691] Defines abstract synchronization primitives and implements it for Motr GC across multiple RGW instances. Signed-off-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> * motr_sync_impl: [CORTX-33691] update lock() & remove_lock() methods Update and refactor the MotrLock::lock() and MotrKVLockProvider::remove_lock() methods. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> Co-authored-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> * rgw_sal_motr, motr_gc: [CORTX-32689] Implement MotrGC::enqueue() (Seagate#379) * rgw_sal_motr, motr_gc: [CORTX-32689] Implement MotrGC::enqueue() * Push {0_ObjTag: motr_gc_obj_info} and {1_ExpiryTime: motr_gc_obj_info} entry to the gc queues, i.e. motr dix. * Call gc->enqueue() method on simple object delete request without actually deleting the obj. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> * motr_gc: [CORTX-32689] Implement MotrGC::list() function MotrGC::list() goes through every gc queue and lists the adds the pending delete requests in resultant vector. Interface to list objs using admin tool or rest api's is yet to be developed. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> * motr_sync: [CORTX-33853] Support race coditions during lock acquisition. Changed lock() implementation to avoid multiple KV operations, thereby reducing the effect of race during lock() call by caller. Signed-off-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> * motr_sync: [CORTX-33853] Support race coditions during lock acquisition. - Changed lock() implementation to avoid multiple KV operations, thereby reducing the effect of race during lock() call by caller. - Added check_lock() to reconfirm the lock by caller. Signed-off-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> Co-authored-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> Co-authored-by: Jeet Jain <jeet.jain@seagate.com> Co-authored-by: Sachin Punadikar <sachin.punadikar@seagate.com>

…ra code (Seagate#356) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code Behaviour - With Garbage Collector enabled, MotrGC will have GC indexes & GC worker threads. - GC worker threads will run for the configured max processing time and then will wait for the configured time between two consecutive runs. Additions/Changes - Add the Garbage Collector infrastructure to support the start & stop of worker threads. - MotrGC class with the interfaces to initialize(), start(), stop() and finalize(). - GCWorker class with entry() and stop() methods. - Refactor Initialization of MotrStore - add setter methods to configure MotrStore - add initialize() method to call initialization of all the other components eg. MetadataCache, GC, (in future LC, QuotaHandler), etc. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com>

…on. (Seagate#385) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code (Seagate#356) * rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code Behaviour - With Garbage Collector enabled, MotrGC will have GC indexes & GC worker threads. - GC worker threads will run for the configured max processing time and then will wait for the configured time between two consecutive runs. Additions/Changes - Add the Garbage Collector infrastructure to support the start & stop of worker threads. - MotrGC class with the interfaces to initialize(), start(), stop() and finalize(). - GCWorker class with entry() and stop() methods. - Refactor Initialization of MotrStore - add setter methods to configure MotrStore - add initialize() method to call initialization of all the other components eg. MetadataCache, GC, (in future LC, QuotaHandler), etc. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> * rgw: [CORTX-33179] GC Metadata Implementation (Seagate#363) Implement GC metadata structure & corresponding functions Added motr_gc_obj_info structure to hold gc metadata for an object marked for deletion. Added logic for creating GC indices based on config parameters. Signed-off-by: Jeet Jain <jeet.jain@seagate.com> * GC thread processing logic (Seagate#371) The GC thread will aquire GC index and process the object entries for deletion either upto the count governed by "rgw_gc_max_trim_chunk" or time allowed by "rgw_gc_processor_max_time". Signed-off-by: Sachin Punadikar <sachin.punadikar@seagate.com> * motr_sync: [CORTX-33691] Define and implement rgw_motr synchronization (Seagate#373) * rgw_sal_motr: [CORTX-33691] Defines abstract synchronization primitives and implements it for Motr GC across multiple RGW instances. Signed-off-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> * motr_sync_impl: [CORTX-33691] update lock() & remove_lock() methods Update and refactor the MotrLock::lock() and MotrKVLockProvider::remove_lock() methods. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> Co-authored-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> * rgw_sal_motr, motr_gc: [CORTX-32689] Implement MotrGC::enqueue() (Seagate#379) * rgw_sal_motr, motr_gc: [CORTX-32689] Implement MotrGC::enqueue() * Push {0_ObjTag: motr_gc_obj_info} and {1_ExpiryTime: motr_gc_obj_info} entry to the gc queues, i.e. motr dix. * Call gc->enqueue() method on simple object delete request without actually deleting the obj. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> * motr_gc: [CORTX-32689] Implement MotrGC::list() function MotrGC::list() goes through every gc queue and lists the adds the pending delete requests in resultant vector. Interface to list objs using admin tool or rest api's is yet to be developed. Signed-off-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> * motr_sync: [CORTX-33853] Support race coditions during lock acquisition. Changed lock() implementation to avoid multiple KV operations, thereby reducing the effect of race during lock() call by caller. Signed-off-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> * motr_sync: [CORTX-33853] Support race coditions during lock acquisition. - Changed lock() implementation to avoid multiple KV operations, thereby reducing the effect of race during lock() call by caller. - Added check_lock() to reconfirm the lock by caller. Signed-off-by: Dattaprasad Govekar <dattaprasad.govekar@seagate.com> Co-authored-by: Sumedh Anantrao Kulkarni <sumedh.a.kulkarni@seagate.com> Co-authored-by: Jeet Jain <jeet.jain@seagate.com> Co-authored-by: Sachin Punadikar <sachin.punadikar@seagate.com>

github-actions bot added the rgw label Jul 20, 2022

sumedhak27 marked this pull request as draft July 20, 2022 09:12

sumedhak27 requested review from sachinpunadikar, bkirunge7, DPG17 and jjxsg July 20, 2022 09:12

sachinpunadikar reviewed Jul 20, 2022

View reviewed changes

src/rgw/motr/gc/gc.cc Outdated Show resolved Hide resolved

sachinpunadikar reviewed Jul 20, 2022

View reviewed changes

src/rgw/motr/gc/gc.cc Outdated Show resolved Hide resolved

sachinpunadikar reviewed Jul 20, 2022

View reviewed changes

src/rgw/motr/gc/gc.h Show resolved Hide resolved

sachinpunadikar suggested changes Jul 20, 2022

View reviewed changes

sumedhak27 force-pushed the motr/gc/add_gc_class branch from e347362 to 0a50508 Compare July 20, 2022 11:45

github-actions bot added the build/ops label Jul 20, 2022

jjxsg reviewed Jul 20, 2022

View reviewed changes

sumedhak27 force-pushed the motr/gc/add_gc_class branch from 3e93cce to 8d3be94 Compare July 21, 2022 09:02

sumedhak27 force-pushed the motr/gc/add_gc_class branch from 8d3be94 to 543a44d Compare July 21, 2022 09:09

sachinpunadikar approved these changes Jul 21, 2022

View reviewed changes

DPG17 reviewed Jul 21, 2022

View reviewed changes

src/rgw/rgw_sal_motr.h Show resolved Hide resolved

jjxsg reviewed Jul 21, 2022

View reviewed changes

src/rgw/motr/gc/gc.cc Outdated Show resolved Hide resolved

jjxsg reviewed Jul 22, 2022

View reviewed changes

src/rgw/motr/gc/gc.h Show resolved Hide resolved

rgw_sal_motr, motr_gc: [CORTX-33148] implement MotrGC::start_processor()

cbc95ae

Implemented the logic to spawn multiple GCWorker threads on startup, and gracefully exit on shutdown. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

jjxsg marked this pull request as ready for review July 25, 2022 06:10

sumedhak27 changed the title ~~[WIP] rgw_sal_motr, motr_gc: [CORTX-33148] Add MotrGC class~~ rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code Jul 25, 2022

sumedhak27 force-pushed the motr/gc/add_gc_class branch from 384ba60 to 76ae806 Compare July 25, 2022 08:43

jjxsg approved these changes Jul 25, 2022

View reviewed changes

motr_gc: [CORTX-33148] partially implement MotrGC::initialize()

bc6b0e4

add code to fetch the max gc indices to create. create and append the index name in gc index list. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>

sumedhak27 merged commit efe655d into Seagate:Motr_GC Jul 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code #356

rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code #356

sumedhak27 commented Jul 20, 2022 •

edited

Loading

sachinpunadikar left a comment

jjxsg Jul 20, 2022

sumedhak27 Jul 21, 2022

sumedhak27 Jul 25, 2022 •

edited

Loading

jjxsg Jul 25, 2022

sachinpunadikar left a comment

rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code #356

rgw_sal_motr, motr_gc: [CORTX-33148] add MotrGC, MotrGC::GCWorker infra code #356

Conversation

sumedhak27 commented Jul 20, 2022 • edited Loading

Checklist

sachinpunadikar left a comment

Choose a reason for hiding this comment

jjxsg Jul 20, 2022

Choose a reason for hiding this comment

sumedhak27 Jul 21, 2022

Choose a reason for hiding this comment

sumedhak27 Jul 25, 2022 • edited Loading

Choose a reason for hiding this comment

jjxsg Jul 25, 2022

Choose a reason for hiding this comment

sachinpunadikar left a comment

Choose a reason for hiding this comment

sumedhak27 commented Jul 20, 2022 •

edited

Loading

sumedhak27 Jul 25, 2022 •

edited

Loading