[ Documentation request ] Explain Java threads usage

[artemik]

Brief description

Worker performance docs mention things like maxConcurrentWorkflowTaskExecutionSize, maxConcurrentActivityExecutionSize, workflowPollThreadCount, activityPollThreadCount, workflowCacheSize, maxWorkflowThreadCount and also mention some corresponding invariants.

But these settings and their relation to each other and to threads is still very confusing.

Example questions are:

• what is "workflow cache", does it include the whole user's workflow object state?
• "It's recommended that maxWorkflowThreadCount be at least 2x of maxConcurrentWorkflowTaskExecutionSize." - why exactly 2x? What happens if maxConcurrentWorkflowTaskExecutionSize == maxWorkflowThreadCount (which on paper sounds like an absolutely normal thing)? How are these settings related to each other exactly? How exactly "resource contention/stealing" happens? And why wouldn't a user want to set these settings equal (i.e. concurrent threads == max threads for maximum performance you'd think)?
• Why maxConcurrentWorkflowTaskExecutionSize don't default to CPU_CORES? Temporal Java SDK requires a thread per workflow (sticky) which is (afaik) mostly just for keeping workflow executions in memory and at, kind of, replayed-stacktrace-position, so that's understandable why high maxWorkflowThreadCount is needed. But considering workflow code is never blocking, what sense does it make to have concurrency higher than CPU_CORES?
• "workflowCacheSize should be ≤ maxWorkflowThreadCount. " - Why cant the cache be bigger than threads? Is it because otherwise when a thread yielded and wanted to continue some other cached workflow, it wouldn't really be able to "continue" it because it would have to replay all events, because workflow yielding works via throwing DestroyWorkflowThreadError exception from workflow code and we can't reliably know/recover the state of the cached workflow object (in memory) after that?
• if a workflow starts 1000 activities in parallel (and then waits on the list of their promises), how does it translate to threads usage? Can a single thread handle it?

So overall would be great to provide clarity regarding threading, including some theoretical examples.

Your recommended content

[fairlydurable]

Thank you for submitting this issue and highlight improvements.