Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not possible to run two monitoring stacks side-by-side #2164

Closed
avikivity opened this issue Jan 28, 2024 · 4 comments · Fixed by #2172
Closed

Not possible to run two monitoring stacks side-by-side #2164

avikivity opened this issue Jan 28, 2024 · 4 comments · Fixed by #2172
Labels
bug Something isn't working right operation

Comments

@avikivity
Copy link
Member

I have two runs I want to compare. But I can't start two stacks, because the container names are hardcoded everywhere. I can't even start one stack with the two runs, because one can import only one prometheus dump.

@avikivity avikivity added the bug Something isn't working right label Jan 28, 2024
@amnonh
Copy link
Collaborator

amnonh commented Jan 29, 2024

you can, to do that you need to specify the different ports, the ports numbers will be part of the names.
This is the recommended way of doing a safe upgrade:
https://monitoring.docs.scylladb.com/stable/upgrade/upgrade-guide-from-monitoring-4.x-to-monitoring-4.y.html#validate-the-version-installed-correctly

Let me know tf that does not answer the question, if it does, please close

@tzach
Copy link
Contributor

tzach commented Jan 30, 2024

Another idea, is to collect both clusters metrics to the same monitoring stack, but to two different "clusters" ot "DCs"

# An example with a single cluster and two DC
- targets:
       - 172.17.0.1:9180
       - 172.17.0.2:9180
       - 172.17.0.3:9180
  labels:
       cluster: cluster1
       dc: dc1
- targets:
       - 172.16.0.1:9180
       - 172.16.0.2:9180
       - 172.16.0.3:9180
  labels:
       cluster: cluster2
       dc: dc1

@amnonh
Copy link
Collaborator

amnonh commented Feb 8, 2024

@avikivity can we close this?

@avikivity
Copy link
Member Author

One needs to disable random other containers for this to work, because not all the code follow the naming convention.

It really needs to be done in a better way.

amnonh added a commit to amnonh/scylla-grafana-monitoring that referenced this issue Feb 11, 2024
It is sometimes usefull to run a secondary stack on the same machine.
This involved having different set of ports.
To make this task easier, there is now an additional command line option
for start-all.sh and kill-all.sh, the --stack id.

id could be 1, 2, 3, or 4.

This will run a minimize secondary stack on a secondary set of port
numbers.

Fixes scylladb#2164

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
@amnonh amnonh added this to the Monitoring 4.7 milestone Feb 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working right operation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants