Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout connection requests when in a deadlock. #67

Merged
merged 6 commits into from
Jun 30, 2020

Conversation

MrLotU
Copy link
Member

@MrLotU MrLotU commented May 23, 2020

Fixes a bug when requesting n+1 connections from a connection pool with limit n where each new connection waited on the previous one.

Previously, this would create a deadlock. Now, a timeout can be added to EventLoopConnectionPool, that will return a failed future if the timeout is reached. Timeout defaults to 10 seconds but can be configured.
Fixes #63

@MrLotU MrLotU requested a review from tanner0101 May 23, 2020 09:33
@codecov-commenter
Copy link

codecov-commenter commented May 23, 2020

Codecov Report

Merging #67 into master will increase coverage by 0.10%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
+ Coverage   94.66%   94.76%   +0.10%     
==========================================
  Files          23       23              
  Lines        1180     1204      +24     
==========================================
+ Hits         1117     1141      +24     
  Misses         63       63              
Flag Coverage Δ
#unittests 94.76% <100.00%> (+0.10%) ⬆️
Impacted Files Coverage Δ
...ncKit/ConnectionPool/EventLoopConnectionPool.swift 94.82% <100.00%> (+0.28%) ⬆️
.../ConnectionPool/EventLoopGroupConnectionPool.swift 93.06% <100.00%> (+0.06%) ⬆️
Tests/AsyncKitTests/ConnectionPoolTests.swift 90.23% <100.00%> (+0.69%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e2cad8...cb9a97f. Read the comment docs.


let task = eventLoop.scheduleTask(in: self.creationTimeout) { [weak self, logger, promise] in
logger.error("Connection creation timed out. This might indicate a connection deadlock in your application.")
if let idx = self?.waiters.firstIndex(where: { $0.2 == uuid }) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both the search and the remove would have linear runtime each. That is not ideal, but hopefully this occurs rarely enough to not be a bottleneck.

Unless in those 10 seconds before timing out so many waiters accumulate that it actually does become a bottleneck... When stuff fails, it often all fails at once, after all.

Copy link
Member

@tanner0101 tanner0101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to this feature. Thanks @MrLotU!

I agree with all @MrMage's comments.

@tanner0101 tanner0101 added the enhancement New feature or request label May 28, 2020
@MrLotU MrLotU requested a review from tanner0101 June 5, 2020 05:56
Copy link
Member

@tanner0101 tanner0101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just minor comments.

promise.fail(ConnectionPoolTimeoutError.connectionRequestTimeout)
}

return promise.futureResult.always { _ in task.cancel() }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cancelling a task that has already completed fine?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a quick test where the task would always finish before cancelling and didn't see anything going wrong, so guessing it's fine. Not 100% sure though.

cc @Lukasa

@MrLotU MrLotU added the semver-minor Contains new APIs label Jun 30, 2020
@MrLotU MrLotU merged commit 7457413 into master Jun 30, 2020
@MrLotU MrLotU deleted the LotU-Connection-Deadlock branch June 30, 2020 07:56
@tanner0101
Copy link
Member

These changes are now available in 1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request semver-minor Contains new APIs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Detect connection pool deadlocks
5 participants