Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batcher and Loader config restructure is required #203

Open
alok87 opened this issue Apr 22, 2021 · 0 comments
Open

Batcher and Loader config restructure is required #203

alok87 opened this issue Apr 22, 2021 · 0 comments
Labels
enhancement New feature or request p3 not urgent, intermittent issue

Comments

@alok87
Copy link
Contributor

alok87 commented Apr 22, 2021

Now when we have started to run every topic into its separate consumer group. The current structure of batcher and loader config is leading to a huge config file based on the total number of topics.

We should restructure it to reduce the duplicacy and compromise on the flexiblity for all use cases.

Since the config map size is limited to etcd restriction on size, we are making a pod not have more than 100 topics and we are dividing into chunk size of 100. #202 (Workaround is live)

@alok87 alok87 added enhancement New feature or request p3 not urgent, intermittent issue labels Apr 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request p3 not urgent, intermittent issue
Projects
None yet
Development

No branches or pull requests

1 participant