-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes synchronicity between transactions/updates #254
Conversation
/assign @dave-tucker |
testing in upstream ovnkube: ovn-kubernetes/ovn-kubernetes#2599 |
Pull Request Test Coverage Report for Build 1381829607
💛 - Coveralls |
@trozet there appear to still be panics in |
from that log, it's clear that we got an |
6db1815
to
33bbf40
Compare
The changes to use a buffered channel to handle updates broken the synchronization ovsdb depends on with txns. This is because OVSDB protocol specifies an update will be sent before a txn reply. We depend on this to ensure our cache is valid when we complete a txn. This patch reverts the changes from cda509c and while the monitor is being setup, defers updates to a buffer. After the monitor setup is complete, the defered updates will be processed in order. Afterwards, the db cacheMutex is unlocked so that subsequent updates maybe processed normally. Signed-off-by: Tim Rozet <trozet@redhat.com>
33bbf40
to
d11896c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another thought.
We might need to deferUpdates
anytime we are setting up a monitor and not only on reconnect. If we monitor a new table well after connect then we might start receiving updates for that table before its initial state?
Previously failures to populate the cache would result in panic. This should be avoided where possible and error handling should take place to reconnect and rebuild the cache if the client and server become out of sync. This change fixes the remaining panics to return errors, and errors are sent to signal a reconnect and cache flush. Additionally, errors are propagated up to the RPC handler and error responses may be sent. Currently these responses may race with client disconnect, but at least its a step in the right direction. Signed-off-by: Tim Rozet <trozet@redhat.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @trozet this is looking really good 👌
The changes to use a buffered channel to handle updates broken the
synchronization ovsdb depends on with txns. This is because OVSDB
protocol specifies an update will be sent before a txn reply. We depend
on this to ensure our cache is valid when we complete a txn.
This patch reverts the changes from
cda509c and while the monitor is being
setup, defers updates to a buffer. After the monitor setup is complete,
the defered updates will be processed in order. Afterwards, the db
cacheMutex is unlocked so that subsequent updates maybe processed
normally.
Signed-off-by: Tim Rozet trozet@redhat.com