Calculate workflow history size and count and expose that to client #1270

timl3136 · 2023-09-01T23:13:21Z

What changed?
Add fields exposing workflow history size and count to client in form of api calls. Client side now will calculate the history size of a workflow while processing events using size of the immutable fields and a constant buffer to account for smaller fields that might change to avoid non-determinism.
Combined with server change that emits history data to client #5392 that will be used to compare against. now customers can access their workflow history data and decide whether creating a ContinueAsNew workflow base on that information.

Why?
Due to Cadence history limit, many customers find their workflow being terminated and there is currently no mechanisms for customer to know their workflow's history usage.
Resolves issues #917

How did you test it?
The code is tested pairing with server code implementation on development environment

Potential risks
In the worst case, the server might not have the latest update with transmitting the history data to client. In that case, the sanity check (not yet implemented) might failed.

Add logger to get whole history sync js change in idl and remove logger add history size and logger into task handlers Update reference to cadence-idl, sync field name with idl repo update idl ref

internal/internal_task_handlers.go

…dd more logs

…y size after processing every history event

…ratio for monitoring

internal/internal_event_handlers.go

internal/internal_task_pollers.go

taylanisikdemir · 2023-10-27T20:49:30Z

internal/internal_event_handlers.go

@@ -1379,3 +1384,139 @@ func (weh *workflowExecutionEventHandlerImpl) handleSignalExternalWorkflowExecut

 	return nil
 }
+
+func (weh *workflowExecutionEventHandlerImpl) estimateHistorySize(event *m.HistoryEvent) int {


This function doesn't need to be attached to *workflowExecutionEventHandlerImpl. it only uses weh.logger which can be passed as argument. So please make this a standalone function and move to internal_utils.go as well as other helper functions you added below. This file is already huge (like many others in internal package unfortunately) and we will slowly refactor each type into its own file.

Moved in latest commit

internal/internal_event_handlers.go

workflow/workflow.go

…ber-go#1270) Enable client side estimated history size exposure via API

petergardfjall · 2023-12-04T12:13:52Z

Hi, after upgrading to go.uber.org/cadence v1.2.6 we are seeing some log warnings that seem to stem from this PR.

2023-12-04T12:21:28.005+0100	WARN	internal/internal_utils.go:473	unknown event type { ... "Event Type": "TimerFired"}
go.uber.org/cadence/internal.estimateHistorySize
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_utils.go:473
go.uber.org/cadence/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_event_handlers.go:945
go.uber.org/cadence/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:917
go.uber.org/cadence/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:805
go.uber.org/cadence/internal.(*workflowTaskPoller).processWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:344
go.uber.org/cadence/internal.(*workflowTaskPoller).ProcessTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:317
go.uber.org/cadence/internal.(*baseWorker).processTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_worker_base.go:380

Similarly for ChildWorkflowExecutionStarted:

2023-12-04T12:25:44.870+0100	WARN	internal/internal_utils.go:473	unknown event type { ... "Event Type": "ChildWorkflowExecutionStarted"}
go.uber.org/cadence/internal.estimateHistorySize
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_utils.go:473
go.uber.org/cadence/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_event_handlers.go:945
go.uber.org/cadence/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:917
go.uber.org/cadence/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:805
go.uber.org/cadence/internal.(*workflowTaskPoller).processWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:344
go.uber.org/cadence/internal.(*workflowTaskPoller).ProcessTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:317
go.uber.org/cadence/internal.(*baseWorker).processTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_worker_base.go:380

Is this just a case of the added estimateHistorySize not covering all possible events or is it something for us to be concerned about?

taylanisikdemir · 2023-12-04T16:46:37Z

Hi, after upgrading to go.uber.org/cadence v1.2.6 we are seeing some log warnings that seem to stem from this PR.

2023-12-04T12:21:28.005+0100	WARN	internal/internal_utils.go:473	unknown event type { ... "Event Type": "TimerFired"}
go.uber.org/cadence/internal.estimateHistorySize
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_utils.go:473
go.uber.org/cadence/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_event_handlers.go:945
go.uber.org/cadence/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:917
go.uber.org/cadence/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:805
go.uber.org/cadence/internal.(*workflowTaskPoller).processWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:344
go.uber.org/cadence/internal.(*workflowTaskPoller).ProcessTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:317
go.uber.org/cadence/internal.(*baseWorker).processTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_worker_base.go:380

Similarly for ChildWorkflowExecutionStarted:

2023-12-04T12:25:44.870+0100	WARN	internal/internal_utils.go:473	unknown event type { ... "Event Type": "ChildWorkflowExecutionStarted"}
go.uber.org/cadence/internal.estimateHistorySize
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_utils.go:473
go.uber.org/cadence/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_event_handlers.go:945
go.uber.org/cadence/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:917
go.uber.org/cadence/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_handlers.go:805
go.uber.org/cadence/internal.(*workflowTaskPoller).processWorkflowTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:344
go.uber.org/cadence/internal.(*workflowTaskPoller).ProcessTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_task_pollers.go:317
go.uber.org/cadence/internal.(*baseWorker).processTask
	/home/peterg/go/pkg/mod/go.uber.org/cadence@v1.2.6/internal/internal_worker_base.go:380

Is this just a case of the added estimateHistorySize not covering all possible events or is it something for us to be concerned about?

That's right. It's being addressed in #1300

timl3136 changed the title ~~Expose history~~ Expose workflow history size and count to client Sep 2, 2023

timl3136 force-pushed the expose-history branch 3 times, most recently from d838f8d to c2a5c26 Compare September 6, 2023 23:41

timl3136 added 3 commits September 6, 2023 16:42

add new field mapping to historySize from pollFromDecisionTask

d720b1e

add new field and logger for testing

4b5fea8

Add logger to get whole history sync js change in idl and remove logger add history size and logger into task handlers Update reference to cadence-idl, sync field name with idl repo update idl ref

update idl version

a6034c9

timl3136 force-pushed the expose-history branch from c2a5c26 to a6034c9 Compare September 6, 2023 23:43

Merge branch 'master' into expose-history

04e9d27

Groxx reviewed Sep 7, 2023

View reviewed changes

internal/internal_task_handlers.go Show resolved Hide resolved

timl3136 and others added 18 commits October 4, 2023 12:40

Merge branch 'uber-go:master' into expose-history

b908498

update history size calculation

9e6216f

update logic

fb247eb

add logger

aea7ba9

some change in logic

99460b2

move calculation to per event base, not active

e59bf0a

add logger and metric

aac2f9b

Change buffer size to 1024 and add logger tag for workflowtype

5ff41f1

Change buffer size to 100 and add logger tag for workflowtype, also a…

ba6be7f

…dd more logs

test another logic

e83cb2b

Add buffer and replay logic

222f725

increase buffer value, add ratio tag

91c76d4

Merge branch 'uber-go:master' into expose-history

8dc580d

remove estimate history size in task handlers, now we estimate histor…

c3f8aea

…y size after processing every history event

add backlog hint logger

2aae765

Add more fields to calculation, remove logger, keep a metric for the …

9e37aeb

…ratio for monitoring

cleanup

610051b

Merge branch 'master' into expose-history

d0d39c4

timl3136 changed the title ~~Expose workflow history size and count to client~~ Calculate workflow history size and count and expose that to client Oct 24, 2023

Merge branch 'master' into expose-history

6bcf0a1

separate metrics for better tracking and info

98ced27

taylanisikdemir reviewed Oct 24, 2023

View reviewed changes

timl3136 added 2 commits October 24, 2023 13:30

respond to comments

dbf41d3

Add npe checker for string pointer

dc6dac5

timl3136 force-pushed the expose-history branch from e91c244 to dc6dac5 Compare October 24, 2023 21:12

Merge branch 'master' into expose-history

bdd300d

taylanisikdemir approved these changes Oct 27, 2023

View reviewed changes

timl3136 and others added 3 commits October 27, 2023 14:06

Move estimateHistorySize function to internal_utils.go

a16cf09

Merge branch 'master' into expose-history

8f644b7

Merge branch 'master' into expose-history

e68da25

timl3136 merged commit c20b993 into uber-go:master Nov 15, 2023
9 checks passed

timl3136 added a commit to timl3136/cadence-client that referenced this pull request Nov 16, 2023

Calculate workflow history size and count and expose that to client (u…

ffd2d75

…ber-go#1270) Enable client side estimated history size exposure via API

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Calculate workflow history size and count and expose that to client #1270

Calculate workflow history size and count and expose that to client #1270

timl3136 commented Sep 1, 2023 •

edited

Loading

taylanisikdemir Oct 27, 2023

timl3136 Oct 27, 2023

petergardfjall commented Dec 4, 2023

taylanisikdemir commented Dec 4, 2023

Calculate workflow history size and count and expose that to client #1270

Calculate workflow history size and count and expose that to client #1270

Conversation

timl3136 commented Sep 1, 2023 • edited Loading

taylanisikdemir Oct 27, 2023

Choose a reason for hiding this comment

timl3136 Oct 27, 2023

Choose a reason for hiding this comment

petergardfjall commented Dec 4, 2023

taylanisikdemir commented Dec 4, 2023

timl3136 commented Sep 1, 2023 •

edited

Loading