Add EclipseStore to memory storage for persistence #1484

emerkle826 · 2024-03-20T16:39:01Z

adejanovski

Issue: It looks like someone removed the unit test runs in CI, which is hiding a handful of errors I'm seeing when building the project:

[ERROR] testDontFailRepairAfterTopologyChange(io.cassandrareaper.service.RepairRunnerTest)  Time elapsed: 0.117 s  <<< ERROR!
org.eclipse.serializer.persistence.exceptions.PersistenceExceptionTypeNotPersistable: Type not persistable: "class com.google.common.collect.ImmutableMapEntry".
	at io.cassandrareaper.service.RepairRunnerTest.addNewRepairRun(RepairRunnerTest.java:1000)
	at io.cassandrareaper.service.RepairRunnerTest.testDontFailRepairAfterTopologyChange(RepairRunnerTest.java:878)

[ERROR] getNodeMetricsInLocalDcAvailabilityForLocalDcNodeTest(io.cassandrareaper.service.RepairRunnerTest)  Time elapsed: 0.101 s  <<< ERROR!
org.eclipse.serializer.persistence.exceptions.PersistenceExceptionTypeNotPersistable: Type not persistable: "class com.google.common.collect.ImmutableMapEntry".
	at io.cassandrareaper.service.RepairRunnerTest.addNewRepairRun(RepairRunnerTest.java:1000)
	at io.cassandrareaper.service.RepairRunnerTest.getNodeMetricsInLocalDcAvailabilityForLocalDcNodeTest(RepairRunnerTest.java:1299)

[ERROR] testDontFailRepairAfterTopologyChangeIncrementalRepair(io.cassandrareaper.service.RepairRunnerTest)  Time elapsed: 0.114 s  <<< ERROR!
org.eclipse.serializer.persistence.exceptions.PersistenceExceptionTypeNotPersistable: Type not persistable: "class com.google.common.collect.ImmutableMapEntry".
	at io.cassandrareaper.service.RepairRunnerTest.addNewRepairRun(RepairRunnerTest.java:1000)
	at io.cassandrareaper.service.RepairRunnerTest.testDontFailRepairAfterTopologyChangeIncrementalRepair(RepairRunnerTest.java:1076)

[INFO] Running io.cassandrareaper.service.HeartTest
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.989 s - in io.cassandrareaper.service.HeartTest
[INFO] Running io.cassandrareaper.AssertionTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in io.cassandrareaper.AssertionTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   HttpCassandraManagementProxyTest.testSSLHotReload:818 » NoSuchFile /home/runne...
[ERROR]   RepairRunResourceTest.displaysExplicitBlacklistedCompactionStrategies:365->addRepairRun:230 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunResourceTest.doesNotDisplayBlacklistedCompactionStrategies:325->addRepairRun:230 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunResourceTest.testAddRepairRun:253->addDefaultRepairRun:204->addRepairRun:230 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunResourceTest.testModifyIntensity:512->addDefaultRepairRun:204->addRepairRun:230 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunResourceTest.testPauseNotRunningRun:480->addDefaultRepairRun:204->addRepairRun:230 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunResourceTest.testTriggerAlreadyRunningRun:403->addDefaultRepairRun:204->addRepairRun:230 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunResourceTest.testTriggerNewRunAlreadyRunningRun:420->addDefaultRepairRun:204->addRepairRun:230 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunnerTest.getNodeMetricsInLocalDcAvailabilityForLocalDcNodeTest:1299->addNewRepairRun:1000 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunnerTest.testDontFailRepairAfterTopologyChange:878->addNewRepairRun:1000 » PersistenceExceptionTypeNotPersistable
[ERROR]   RepairRunnerTest.testDontFailRepairAfterTopologyChangeIncrementalRepair:1076->addNewRepairRun:1000 » PersistenceExceptionTypeNotPersistable

We'd need to reintroduce running the unit tests in the "Build Reaper" job and investigate why we're getting these errors.

adejanovski · 2024-03-27T17:12:15Z

src/server/src/main/java/io/cassandrareaper/storage/MemoryStorageFacade.java

+      this.memoryStorageRoot.getClusters().entrySet().stream().forEach(entry -> {
+        Cluster cluster = entry.getValue();
+        LOG.info("Loaded cluster: {} / seeds: {}", cluster.getName(), cluster.getSeedHosts());
+      });


Todo: these lines can be removed, they were added for debugging purposes.

Will do. I was thinking of leaving it in as it is a quick way to tell if the EclipseStore stuff is broken, but it is unnecessary noise in the logs at this point.

Todo: This still needs to be moved to debug level at least.

I got caught up trying to resolve the serialization issues and forgot to address your earlier comments. I'll address them all in another commit

src/server/src/main/java/io/cassandrareaper/storage/MemoryStorageFacade.java

adejanovski · 2024-03-28T07:07:03Z

I'm seeing the error at runtime as well:

INFO  [2024-03-28 07:04:37,834] io.cassandrareaper.storage.MemoryStorageFacade: Adding RepairSegment Token Ranges: [(6083251659656910587,6107424999151749596], (-9104419452840277688,-9102509490550099867], (-2270133911460013913,-2256937252766634776], (893860044235803218,901583255025913501], (92738551409269697,107741857405993256], (6340960424175699889,6353113292708633402], (-6138200817756007410,-6110383572737697036], (6671466593238674942,6679895794982887345], (-6701797388067458407,-6635828587888775463], (-4592428009307264289,-4584216478015188842], (-7552924057449383653,-7550445701062542606], (-734766440320207266,-705456293043214132], (-4800886656279118057,-4766897477514830888], (-4032014401275665485,-4018251167609629111], (-4351265931315251400,-4345014210196939237], (844989585998104097,857822122554933312], (-5796666629985782443,-5778484105896892022], (6899765605735392604,6901152226373324748], (2387384085044470415,2398773519464099897]]
ERROR [2024-03-28 07:04:37,867] io.cassandrareaper.service.SchedulingManager: Failed managing repair schedule with id '6c091a80-ecd1-11ee-a082-797c3ce00e30'
! org.eclipse.serializer.persistence.exceptions.PersistenceExceptionTypeNotPersistable: Type not persistable: "class com.google.common.collect.ImmutableMapEntry".
! at org.eclipse.serializer.persistence.binary.types.BinaryHandlerUnpersistable.guaranteeSpecificInstanceViablity(BinaryHandlerUnpersistable.java:79)
! at org.eclipse.serializer.persistence.types.PersistenceTypeHandlerManager$Default.ensureTypeHandler(PersistenceTypeHandlerManager.java:356)
! at org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.registerGuaranteed(BinaryStorer.java:603)
! at org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.registerLazyOptional(BinaryStorer.java:618)
! at org.eclipse.serializer.persistence.types.PersistenceObjectManager$Default.ensureObjectId(PersistenceObjectManager.java:196)
! at org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.register(BinaryStorer.java:637)
! at org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.apply(BinaryStorer.java:313)
! at org.eclipse.serializer.persistence.binary.types.Binary.storeReferencesAsList(Binary.java:1833)
! at org.eclipse.serializer.persistence.binary.types.Binary.storeReferences(Binary.java:1196)
! at org.eclipse.serializer.persistence.binary.java.lang.BinaryHandlerNativeArrayObject.store(BinaryHandlerNativeArrayObject.java:101)
! at org.eclipse.serializer.persistence.binary.java.lang.BinaryHandlerNativeArrayObject.store(BinaryHandlerNativeArrayObject.java:30)
! at org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.storeItem(BinaryStorer.java:444)
! at org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.storeGraph(BinaryStorer.java:427)
! at org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.storeAll(BinaryStorer.java:460)
! at org.eclipse.serializer.persistence.types.PersistenceManager$Default.storeAll(PersistenceManager.java:313)
! at org.eclipse.store.storage.types.StorageConnection.storeAll(StorageConnection.java:410)
! at io.cassandrareaper.storage.MemoryStorageFacade.persist(MemoryStorageFacade.java:183)
! at io.cassandrareaper.storage.MemoryStorageFacade.addRepairSegment(MemoryStorageFacade.java:277)
! at io.cassandrareaper.storage.repairsegment.MemoryRepairSegmentDao.addRepairSegments(MemoryRepairSegmentDao.java:54)
! at io.cassandrareaper.storage.repairrun.MemoryRepairRunDao.addRepairRun(MemoryRepairRunDao.java:103)
! at io.cassandrareaper.service.RepairRunService.registerRepairRun(RepairRunService.java:279)
! at io.cassandrareaper.service.SchedulingManager.createNewRunForUnit(SchedulingManager.java:261)
! at io.cassandrareaper.service.SchedulingManager.manageSchedule(SchedulingManager.java:220)
! at io.cassandrareaper.service.SchedulingManager.run(SchedulingManager.java:132)
! at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
! at java.base/java.util.TimerThread.run(Timer.java:506)

adejanovski

Everything is working great now!

just a few small things to fix and we're good to go.

adejanovski · 2024-04-10T09:43:40Z

src/server/src/main/java/io/cassandrareaper/storage/MemoryStorageFacade.java

+        this.embeddedStorage.storeAll(objects);
+        this.embeddedStorage.storeRoot();
+      } catch (Throwable ex) {


Issue: Can we catch a RuntimeException instead of a Throwable which is a bit too wide?

This was put in to debug where/what was causing the not-persistable issue I was facing. I can probably just remove the try-catch all together, unless you think it would be helpful to catch it?

adejanovski · 2024-04-10T09:45:18Z

src/server/src/main/java/io/cassandrareaper/storage/MemoryStorageFacade.java

+        this.embeddedStorage.storeRoot();
+      } catch (Throwable ex) {
+        ex.printStackTrace();


Todo: This should be replaced by a call to the logger

Suggested change

ex.printStackTrace();

LOG.error("Failed persisting Reaper state on disk", ex);

Same comment here, the stack trace was just making it easier to track down what was failing to persist. Now that I've fixed it, we probably don't need the try-catch at all.

Catching RuntimeExceptions is probably a good thing to do so we can have an error message that's easy to read.
wdyt?

Ok, I can leave it in. Will change it from Throwable to RuntimeException.

adejanovski · 2024-04-10T09:45:51Z

src/server/src/main/java/io/cassandrareaper/storage/MemoryStorageFacade.java

+      this.memoryStorageRoot.getClusters().entrySet().stream().forEach(entry -> {
+        Cluster cluster = entry.getValue();
+        LOG.info("Loaded cluster: {} / seeds: {}", cluster.getName(), cluster.getSeedHosts());
+      });


Todo: This still needs to be moved to debug level at least.

adejanovski

Awesome work, thanks!

adejanovski · 2024-04-11T05:32:14Z

...erver/src/test/java/io/cassandrareaper/management/http/HttpCassandraManagementProxyTest.java

+    // use relative paths as we are likely running locally
+    Path projectRoot = FileSystems.getDefault().getPath("..", "..", "..");
+    Path ks = projectRoot.resolve(".github/files/keystore.jks");
+    Path ts = projectRoot.resolve(".github/files/truststore.jks");


Praise: nice fix, thanks!

adejanovski and others added 6 commits March 11, 2024 17:39

Add a K/V store for the memory backend

0db0cf5

Fix Memory storage persistence so objects are reinstantiated correctly

57a03f1

Configure EclipseStore to persist transient fields

19bdc29

update Management API client version

4e7e5c9

Fix segments by repair run for memory storage

b378cac

[fixup] cleanup unnecessary changes

91b031a

emerkle826 marked this pull request as ready for review March 26, 2024 21:18

adejanovski reviewed Mar 28, 2024

View reviewed changes

Remove Guava ImmutableMap as it doesn't serialize in EclipseStore

a0399a5

adejanovski requested changes Apr 10, 2024

View reviewed changes

[fixup] Addressing review comments, fix Http test

ca540d9

adejanovski approved these changes Apr 11, 2024

View reviewed changes

adejanovski merged commit bb99e2d into master Apr 11, 2024
20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add EclipseStore to memory storage for persistence #1484

Add EclipseStore to memory storage for persistence #1484

emerkle826 commented Mar 20, 2024 •

edited by adejanovski

Loading

adejanovski left a comment

adejanovski Mar 27, 2024

emerkle826 Mar 28, 2024

adejanovski Apr 10, 2024

emerkle826 Apr 10, 2024

adejanovski commented Mar 28, 2024

adejanovski left a comment

adejanovski Apr 10, 2024

emerkle826 Apr 10, 2024

adejanovski Apr 10, 2024

emerkle826 Apr 10, 2024

adejanovski Apr 10, 2024

emerkle826 Apr 10, 2024

adejanovski Apr 10, 2024

adejanovski left a comment

adejanovski Apr 11, 2024

	ex.printStackTrace();
	LOG.error("Failed persisting Reaper state on disk", ex);

Add EclipseStore to memory storage for persistence #1484

Add EclipseStore to memory storage for persistence #1484

Conversation

emerkle826 commented Mar 20, 2024 • edited by adejanovski Loading

adejanovski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adejanovski commented Mar 28, 2024

adejanovski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adejanovski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emerkle826 commented Mar 20, 2024 •

edited by adejanovski

Loading