feat: add kafka importer as option

[gasymovrv]

Connected to the question:
#571

[CLAassistant]

All committers have signed the CLA.

[nitram509]

@gasymovrv thank you for this PR - overall it looks quite good.
I have commented on a few things and would appreciate hearing your view on it.
Also, may I ask to extend the README, with some example(s) and description, please?

[nitram509]

@gasymovrv
Ruslan, I did a local test and everything worked fine.
I made two comments. If you agree and would adjust these, then I will merge.
Again, great work 👍

[nitram509]

Great, you though on updating this one.
Just being nitty gritty here, would you please adjust the arrows, following a consistent meaning (right now its mixed) - I propose 'technical dependency'.
Also, please adjust the text per each box being on the same semantical level, which makes reading easier.

See my proposed sketch, expressing what I have in mind...

[gasymovrv]

Сould you take a look to see if I understood you correctly (I didn't fully understand the DB dependency, you probably meant the arrow from ZSM to DB)?

[nitram509]

good catch, you're right.
My proposal was wrong at this point 🙈

[nitram509]

When I naively tried docker compose up it crashed on me, because multiple backends overlap/clash.
Also, firing up too many services will stress people, who don't have enough hardware resources.
While I like your idea of these prepared services defined, may I propose to introduce profiles.
E.g. there could be: hazelcast, kafka, postgres, mysql, and inmemory.

Which means, we could also add a small note in the CONTRIBUTION.md file,
saying something like: "For testing purpose, use docker profiles ... COMPOSE_PROFILES=inmemory,hazelcast docker compose up"

See also https://docs.docker.com/compose/profiles/

[gasymovrv]

Martin, it's a good idea, but I think there some troubles with multiple profiles. For example:
COMPOSE_PROFILES=in_memory,hazelcast docker compose up
will try to start all services that have "in_memory" OR "hazelcast" profile, because profiles works as "has any".
So each of these services matches:

  simple-monitor-in-memory:
    ...
    profiles: ["in_memory", "hazelcast"]

  simple-monitor-in-memory-kafka:
    ...
    profiles: ["in_memory", "kafka"]

  simple-monitor-postgres:
    ...
    profiles: ["postgres", "hazelcast"]

  simple-monitor-mysql:
    ...
    profiles: ["mysql", "hazelcast"]

The workaround is to use compound names for profiles and run docker-compose as follows: COMPOSE_PROFILES=hazelcast,hazelcast_postgres,postgres

[nitram509]

Right, I see.
Thanks for pointing this out.

[nitram509]

Great job, Ruslan 🚀
Thank you very much for your contribution. 👍

[gasymovrv]

Thank you for the good comments! I hope this feature will be useful to people