Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix #3721] Optimize event grouping #3739

Merged
merged 3 commits into from
Oct 25, 2024
Merged

Conversation

fjtirado
Copy link
Contributor

@fjtirado fjtirado commented Oct 18, 2024

Fix #3721

Merge with apache/incubator-kie-kogito-apps#2122

When property kogito.events.grouping.binary is set, process instance events collection is sent in binary format, which takes less space than sending the collection in json.
The binary format is propietary, but it does not include any java stuff (besides the fact that is using Java byte stucture approach), so it is theoretically possible to interpret it from a different language.

The binary format reduce space by taking advantage of the following facts:

  1. The information in cloud event extension is always the same for all events related with a particular process instance (except for variable name and event date) Nonetheless, for event date, we just store the delta from a time reference (at most as 5 bytes, rather than 8, if serializing dates as timestamps or a good number of characted, if serializing the date as UTF)
  2. A substantial portion of cloud event extension information is duplicated in the event body (for example, processId, processInstanceId, processVersion, eventDate..) so we can send it just once
  3. Property names does not need to be sent when the data structure is known by the sender and the receiver
  4. Numbers can be sent as numbers, not as char sequences (saving a lot of space for number with more than one digit)

If after passing to binary, we still need to reduce the event size further, kogito.events.grouping.compress property can be set to true. This will gzip the resulting byte[] at the cost of some performance.

@kie-ci3
Copy link
Contributor

kie-ci3 commented Oct 21, 2024

PR job #18 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer

build-chain build full_downstream -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-runtimes -u #3739 --skipParallelCheckout

NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest_jobs/job/kogito-runtimes-pr/job/PR-3739/18/display/redirect

Test results:

  • PASSED: 3396
  • FAILED: 1

Those are the test failures:

org.jbpm.bpmn2.IntermediateEventTest.testIntermediateCatchEventTimerCycle3
expected: 3
but was: 2

@fjtirado fjtirado force-pushed the Fix_#3721 branch 3 times, most recently from 4f828a8 to c15c0fd Compare October 22, 2024 11:19
@kie-ci3
Copy link
Contributor

kie-ci3 commented Oct 22, 2024

PR job #21 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer

build-chain build full_downstream -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-runtimes -u #3739 --skipParallelCheckout

NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest_jobs/job/kogito-runtimes-pr/job/PR-3739/21/display/redirect

Test results:

  • PASSED: 3378
  • FAILED: 2

Those are the test failures:

PR check / Build projects / org.kie.kogito.it.JDBCOptimisticLockingIT.testPersistence java.util.concurrent.CompletionException: java.lang.RuntimeException: Unable to start Quarkus test resource class org.kie.kogito.testcontainers.quarkus.PostgreSqlQuarkusTestResource
PR check / Build projects / org.kie.kogito.it.JDBCPersistenceIT.testPersistence java.util.concurrent.CompletionException: java.lang.RuntimeException: Unable to start Quarkus test resource class org.kie.kogito.testcontainers.quarkus.PostgreSqlQuarkusTestResource

@fjtirado fjtirado marked this pull request as ready for review October 22, 2024 15:20
Copy link
Contributor

@gmunozfe gmunozfe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, outstanding work @fjtirado !

@fjtirado fjtirado force-pushed the Fix_#3721 branch 2 times, most recently from 7211e81 to a863046 Compare October 24, 2024 13:17
LOGGER.debug("Deserialize user task instance data event: {}", node);
String type = node.get("type").asText();

switch (type) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit; maybe in an upcoming change, but in ProcessInstanceDataEvent, you created enum types for it.

@kie-ci3
Copy link
Contributor

kie-ci3 commented Oct 24, 2024

PR job #27 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer

build-chain build full_downstream -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-runtimes -u #3739 --skipParallelCheckout

NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest_jobs/job/kogito-runtimes-pr/job/PR-3739/27/display/redirect

Test results:

  • PASSED: 3396
  • FAILED: 1

Those are the test failures:

org.jbpm.bpmn2.IntermediateEventTest.testIntermediateCatchEventTimerCycle3
expected: 3
but was: 2

@fjtirado fjtirado merged commit ce37bdc into apache:main Oct 25, 2024
5 of 6 checks passed
rgdoliveira pushed a commit to rgdoliveira/kogito-runtimes that referenced this pull request Nov 7, 2024
* [Fix apache#3721] Optimize event grouping

* [Fix apache#3721] Adding addons

* [Fix apache#3721] Refactoring KogitoIndexConverter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce event group size
4 participants