Horizontal scaling issue with EclipseStore in the cloud

[Flo9520]

Hello,

I am currently facing issues with horizontal scaling in the cloud. I have a pod A that has the storage injected and also the application logic. This pod reads and writes data, which works fine.

Now I start a second pod B (same service and same storage but only in read mode). Upon statup it loads the current data from the actual storage files. Now I have 2 pods running with the same storage. When the podA creates a new object I want to notify pod B about it, so it can update it's object graph. I send a message to pod B to reload the data from the storage with the Reloader so it also holds the newly created object.

I use the following code to reload the new object in pod B with:

Reloader.New(storageManager.persistenceManager()).reloadFlat(storageManager.getObject(objectId));

where the objectId is the Id given by storageManager.persistenceManager().lookupObjectId(object); from pod A.

This is where the problems begin. The pod B has no information about the new object and is also not able to load it, since it's ID is not present in the object graph, which ends with an Exception. There is no actual lookup in the storage file performed, only the RAM is checked.

This fact basically means I can not scale my pod horizontally since they will never be able to load the objects the other pod created even though they use the same file share.
Is there even a way to achieve scaling like this or do I need to take the storage logic to a separate service and only scale my application logic services?

Kind regards and thank you for any help in advance!

[hg-ms]

Hello,

It is NOT supported to access the storage data from more than one EmbeddedStorageManager instance simultaneously. As the instances don’t know about each other they will not recognize changes done to the storage by another instance. Starting a second instance may even cause corrupted data files.
Any scaling has to be done by the application that exclusively owns the storage.

[Flo9520]

Hallo hg-ms,

so I would need to split my application into a business logic part and a storage part. The storage part is a one-of and the business application part I can scale horizontally. Is this correct?
Do you have any best practices on how to achieve this kind of vertical scaling with EclipseStore?

[grubeninspekteur]

That's a pretty important point to solve, and with Eclipse Store beging advertised as cloud-native, it puzzles me that access by multiple instances is not supported by default. It seems that MicroStream offers a commercial solution with MicroStream Cluster. It's generous that EclipseStore was donated as open source, however it seems incomplete without the commercial features offered by the donor.

The linked page reads:

We started developing EclipseStore in 2013. At this time Java enterprise applications were usually big monoliths. That’s why the EclipseStore was built to run embedded within an application in a single JVM on a single server.
[...] During the last 10 years, cloud computing, horizontally scaling systems, microservices and serverless computing came up. Meanwhile, the Java community has recognizes that microservices is not the perfect approach for every case, because microservices are linked to higher complexity, new challenges, additional effort and costs of development.
[...] Thus, for most Java users, monolithic single JVM applications will be the standard architecture also in the future. That’s why the EclipseStore open-source project delivers enormous added value to the Java community.

Even Monoliths should always run at least two instances for fault-tolerance and zero-downtime deployment.