Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hanging issue with get dominfo during VM shutdown #58

Closed
donghunkim-zacky opened this issue Dec 19, 2024 · 1 comment
Closed

Hanging issue with get dominfo during VM shutdown #58

donghunkim-zacky opened this issue Dec 19, 2024 · 1 comment

Comments

@donghunkim-zacky
Copy link

donghunkim-zacky commented Dec 19, 2024

We're experiencing an issue with the inovex libvirt exporter hanging during VM shutdown, particularly for resource-intensive VMs (e.g., multiple GPUs, high memory).

Issue:

  • VM shutdown for large instances can take 1-2 minutes.
  • The exporter, collecting data every minute, hangs on get dominfo during shutdown.
  • This causes gaps in our monitoring data.

Question:

  • Can we implement a non-blocking or timeout mechanism for read-only operations like get dominfo? This would allow continued monitoring even during VM shutdowns.
  • The lock is specific to a particular VM, but it's causing the entire host monitoring information to hang. Is there a way to implement a "nowait" option or similar mechanism to prevent this global impact?

Any insights or suggestions are appreciated.

@frittentheke
Copy link
Collaborator

frittentheke commented Dec 19, 2024

This is somewhat of a duplicate of #53, which reports the same issue for migrations and also some ideas on improving or fixing this. So the issue is known and there attempt to fix this or make the exporter more tolerant in this regard will me made. Ideas and PRs are always welcome though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants