Investigate high latency ingestion

[raminqaf]

After upgrading Kafka to 3.1, the ingestion time increased noticeably. I had to add sleeps in the e2e tests after each ingest. We should investigate what the main reason is.

[DawidNiezgodka]

Results

[DawidNiezgodka]

The second part of the test involves testing the consumption time, that is, we want to measure how much time a consumer needs to read all records send by the ingest service to a specific kafka topic.

[DawidNiezgodka]

Results with the consumer time

[DawidNiezgodka]

Hypothesis: Kafka 3 needs more time for the stuff around producer and consumer (de)activation. When they are already active, then the processing is faster than Kafka 2. One of the solution to the problem could be to add additional resources to speed up the start-up.