Horizontally scaling with multiple instances - synchronization and other features between service instances.

[c-ankur]

Hi,

For adoption check purposes, I would like to ask about horizontal scalability which has been mentioned as a feature.
Looks like these features are not supported automatically by Flowret (the application has to do it), but would like some confirmation / feedback:-

a. what features are supported as part of horizontal scalability?
b. how is synchronization managed between different service instances / multiple nodes running Flowret? (seems like application has to manage it on its own)
c. For example: will Flowret automatically ensure no conflicts between workflows running on different service instances?
d. If a service instance goes down, will other service instances be able to pick up pending workflows automatically through Flowret? ( seems like a batch job would have to scan and pick orphaned / pending cases via custom application code)
e. Assuming that the incrCounter method would have to be implemented in a way that avoids conflicts with multiple service instances. For example, after a crash, if a different service instance needs to resume the workflow? (probably counters/ process info has to be stored in a database or a common file system for this to work)

[deepakarora3]

a. what features are supported as part of horizontal scalability?
Flowret is an embedded orchestrator which means that it runs in the same JVM as the application. All the processing triggered with Flowret also happens on the same JVM. So if the application uses Flowret and the application is horizontally scalable i.e. can run on multiple pods, then so will Flowret. Of course it is up to the application to take care of issues like distributed locking at an application level i.e. avoiding situations where the same case gets picked up on more than one pod etc.

b. how is synchronization managed between different service instances / multiple nodes running Flowret? (seems like application has to manage it on its own)
Yes, application needs to manage the same.

c. For example: will Flowret automatically ensure no conflicts between workflows running on different service instances?
Yes, as long as it is not the same case that is being processed. Here in our internal use in Amex, we use a distributed lock to ensure this.

d. If a service instance goes down, will other service instances be able to pick up pending workflows automatically through Flowret? ( seems like a batch job would have to scan and pick orphaned / pending cases via custom application code)
Yes - a batch job will need to be written to scan and pickup orphaned cases. Once the orphaned cases are identified, they can be resumed. Flowret has the capability to pick up from what was recorded last.

e. Assuming that the incrCounter method would have to be implemented in a way that avoids conflicts with multiple service instances. For example, after a crash, if a different service instance needs to resume the workflow? (probably counters/ process info has to be stored in a database or a common file system for this to work)
It is expected that the implementation of the Dao interfaces is persistent. Hence when a counter is incremented, the next value of the same is also persisted in some database.

[c-ankur]

Thanks for the confirmation!