-
Notifications
You must be signed in to change notification settings - Fork 665
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add 1-5 minutes of delay after maintenance mode was enabled #5872
Comments
PR for maintenance mode detection: #5755 |
@guruz I think this would be easy, we just need a random "coming back up" timer. I think it should apply to maintinance mode and bare 503. |
This only applies when the server was explicitly in maintenance mode or when it was 503-unavailable.
PR #5874 |
This only applies when the server was explicitly in maintenance mode or when it was 503-unavailable.
@ckamm Perhaps use an environment variable to disable this for debugging purpose. Otherwise debugging can become quite time consuming… (theming could be interesting too) |
@michaelstingl Sorry, I think that's overkill and would pollute the code. Not worth it for this kind of feature. |
@guruz okay, then it's fine for me |
After quite a few tests, it seems pretty much behaving the way expected now - I'm with @guruz in this one:
If switching off maintenance in the server was properly handled, there should be no need for this, which might affect smaller instances as collateral. Also I'm worried, if the client could still be affected by a Byzantine 503 from the server (e.g. with some ext.sto. backends that do not behave so well like in #5088) - being fooled into a 5'-wait when maintenance mode was not activated. Closing here as behavior is the expected, we could keep the discussion on a core ticket. |
@SamuAlfageme A stay 503 won't make the client think maintenance mode is active. The server has to actively have |
@ckamm I expressed myself wrong:
Should say "even when maintenance mode was not activated" which was the original reason to introduce the delay. And reading: Lines 153 to 157 in 63741d9
I was assuming both sources resulted on the application of the delay; is it wrong? |
@SamuAlfageme You're right, I forgot about that. Yes, a 503 reply to the auth check in connectionvalidator would also trigger the delay. A random 503 during sync would not though. |
otherwise an a large instance all sync clients will come back at once, hitting the system ... hard
The text was updated successfully, but these errors were encountered: