You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix possible purge of requests after discarding is in-progress after error handling.
Why ?
It fixes a new type of "iov data corruption" issue found by MTT/IO-Demo testing:
client and server exchange data
some network issues happen on some links, but some other links are still health
client detects an issue while some RNDV operations are in-progress and RTS for them was sent
ucp_ep_set_failed discards all UCT lanes (they operations are in-progress)
ucp_ep_set_failed invokes a user's error callback which closes the failed endpoint with mode=FORCE
ucp_ep_reqs_purge is called from ucp_ep_discard_lanes, since no lanes to discard
send operations are canceled and their invalidate data
server doesn't detect an issue and continues reading data using health links, but the data is invalid
How ?
Save a pointer to ucp_ep_discard_lanes_t in the control extension of UCP endpoint.
When starting discarding of UCT lanes, check whether discard_lanes was already allocated: if so - use it, otherwise - allocate a new one. So all discarding operations will use the same discard_lanes argument and UCP EP protected from being purged before all UCT lanes are closed.
Write a test to test detecting UCP EP failure and FORCE_CLOSE during in-progress RNDV operation.
Maybe just don't call purge during ep_close if the ep is already failed?
do you mean to not do invoke ucp_ep_discard_lanes() if UCP_EP_FLAG_FAILED set?
if you mean to not invoke ucp_ep_reqs_purge() during ucp_ep_close_nbx()? but we don't do it directly. we invoke ucp_ep_reqs_purge() only as a part of ucp_ep_discard_lanes()'s completion flow.
if you mean to not invoke ucp_ep_reqs_purge() during ucp_ep_close_nbx()? but we don't do it directly. we invoke ucp_ep_reqs_purge() only as a part of ucp_ep_discard_lanes()'s completion flow.
if you mean to not invoke ucp_ep_reqs_purge() during ucp_ep_close_nbx()? but we don't do it directly. we invoke ucp_ep_reqs_purge() only as a part of ucp_ep_discard_lanes()'s completion flow.
Maybe discard with a different callback?
maybe just not do discard in case of UCP_EP_FLAG_FAILED flag set? since, if this flag is set, it means that all lanes are already marked as failed ones - so, nothing to discard.
I've implemented this way - just to not rely on UCP_EP_FLAG_FAILED flag
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dmitrygx
force-pushed
the
topic/ucp/purge_reqs_after_fail_and_close
branch
from
What
Fix possible purge of requests after discarding is in-progress after error handling.
Why ?
It fixes a new type of "iov data corruption" issue found by MTT/IO-Demo testing:
ucp_ep_set_faileddiscards all UCT lanes (they operations are in-progress)ucp_ep_set_failedinvokes a user's error callback which closes the failed endpoint with mode=FORCEucp_ep_reqs_purgeis called fromucp_ep_discard_lanes, since no lanes to discardHow ?
ucp_ep_discard_lanes_tin the control extension of UCP endpoint.discard_laneswas already allocated: if so - use it, otherwise - allocate a new one. So all discarding operations will use the samediscard_lanesargument and UCP EP protected from being purged before all UCT lanes are closed.