-
Notifications
You must be signed in to change notification settings - Fork 2.9k
"My benchmark doesn't show a difference."
"My benchmark doesn't show a difference", is a phrase we have heard before from users looking to switch from an existing pool to HikariCP.
While HikariCP is probably best known for being fast, we have actually spent more effort on achieving that speed within the constraints of providing the highest reliability possible. We are confident that HikariCP is not slightly more reliable, but substantially more reliable than currently available pools.
One of the primary reasons users can't measure a difference between HikariCP and other pools is that most available pools default to a mode of operation where performance is prioritized over reliability.
In contrast, HikariCP has no "unsafe" operational modes -- no way to disable "correct" behavior. As soon as you start turning on the reliability options in other pools to match that of HikariCP, the performance of those pools starts to drop dramatically.
The benchmarks cited on our page are probably overly generous to other pools. In order to measure only the overhead imposed by each pool, the benchmarks are run against a JDBC stub-driver in which every operation is an empty method. When a real driver and DB are put into the loop instead, the difference in results beg believability. But believe them.
I'm going to talk about HikariCP, Apache DBCP2, and Tomcat DBCP here; talking some about speed, and bringing in the reliability pieces. Here we have run HikariCP-benchmark with the three pools against a real database (MySQL) instead of a stub.
Keep in mind no query is being run here, only getConnection() followed by close().
First, all three pools in default configuration (+ autocommit=false):
Benchmark (pool) Mode Score
ConnectionBench.cycleCnnection hikari thrpt 45289.116 ops/ms
ConnectionBench.cycleCnnection tomcat thrpt 2329.692 ops/ms
ConnectionBench.cycleCnnection dbcp2 thrpt 21.750 ops/ms
DBCP2 does get bonus points for one default on the side of safety; rollbackOnReturn defaults to true.
👉 What is not visible is that DBCP2 is generating ~3MB/sec of traffic to the DB, because it is unconditionally rolling back.
Unfortunately, it is not validating connections on borrow.
HikariCP is generating zero traffic to the DB.
HikariCP also defaults to "rollback on return" (it can't be turned off because that is the correct behavior for a pool), but it additionally tracks transaction state and does not rollback if the SQL has already been committed or no SQL was run.
HikariCP also defaults to "test on borrow" (it can't be turned off...). HikariCP does employ an optimization that says, "If a connection had activity within the past 500ms, it must be alive, so bypass connection validation."
Tomcat DBCP is also generating zero traffic to the DB, but for a different reason.
👉 Tomcat simply is not validating connections at all, nor is it rolling back on return.
Now, let's try to level the playing field as a little. For Tomcat and DBCP, we need to enable connection validation.
Benchmark (pool) Mode Score
ConnectionBench.cycleCnnection tomcat thrpt 2133.992 ops/ms
ConnectionBench.cycleCnnection dbcp2 thrpt 5.296 ops/ms
DBCP2 took a big hit here. And, of course, it is still generating ~3MB/sec of traffic to the DB.
At ~5 ops/ms, TPS is capped at a maximum of only 5000 TPS. And 3MB/sec of overhead is generated at that level.
Tomcat DBCP does support a similar optimization to HikariCP, the config goes something like this:
setTestOnBorrow(true);
setValidationInterval(500);
setValidator( new Validator() {
public boolean validate(Connection connection, int mode) {
return connection.isValid(0);
}
} );
The result is the above performance of 2133.992 ops/ms. Still very good, and likely difficult to measure in most benchmarks.
👉 But we forgot "rollback on return" for Tomcat. Turning that on, performance drops dramatically:
ConnectionBench.cycleCnnection tomcat thrpt 20.706 ops/ms
ConnectionBench.cycleCnnection dbcp2 thrpt 5.296 ops/ms
Tomcat too is now generating ~3MB/sec of traffic to the DB. It does not track transaction state and therefore must unconditionally rollback. We thought maybe enabling "ConnectionState tracking" might help, but it does not.
There is a lot more that HikariCP is doing by default, including guarding against network partitions.
Correctness, in the context of SQL connections and JDBC, means that an ideal pool implementation presents a Connection instance to the application that is indistinguishable from a "fresh" Connection obtained directly from the driver.
In other words, the pool should be transparent to the application, and its presence or absence merely improves or degrades performance, and should not alter the behavior of the application in any way.
✅ Default, ⭕ Supported, ❌ Not Supported
| HikariCP | Tomcat DBCP | DBCP2 | |
|---|---|---|---|
| Response time Guarantee | ✅ | ❌ | ❌ |
| Reset Catalog | ✅ | ⭕ | ❌ |
| Reset Read-only | ✅ | ⭕ | ❌ |
| Reset Auto-commit | ✅ | ⭕ | ❌ |
| Reset Transaction Isolation | ✅ | ⭕ | ❌ |
| Reset Network Timeout | ✅ | ❌ | ❌ |
| Rollback-on-return | ✅ | ⭕ | ✅ |
| Disposable Proxies | ✅ | ⭕ | ❌ |
| Track/Close Open Statements | ✅ | ⭕ | ⭕ |
| SQLException State Scanning | ✅ | ❌ | ⭕ |
| Clear SQL warnings | ✅ | ❌ | ❌ |
This could also be called "accurate timeouts". HikariCP runs connection acquisition asynchronously to calls to getConnection() and therefore can provide extremely accurate timeouts to the application.
Other pools run connection acquisition on the user's thread, so if no connection is available in the pool when getConnection() is called, it is possible for the application to block for an indefinite amount of time.
Resetting connection state is essential to providing reliable connections to the application. DBCP2 does not reset state, which means the actions of one thread can adversely affect another.
Tomcat does not reset state by default, but supports it as an option. Unfortunately, because Tomcat does not track state, it unconditionally resets state, which can generate megabytes of unnecessary traffic to the database every second.
Only HikariCP tracks and properly resets connection.setNetworkTimeout().
Without rollback-on-return capability, a pool with allow transactions to "leak" across threads, potentially causing inconsistent DB state and nearly impossible to locate bugs.
DBCP2 enables it by default. Tomcat can optionally enable it. Unfortunately, neither tracks transaction state, and therefore generate megabytes of performance killing unnecessary traffic to the database when enabled.
"Disposable Proxies" prevent code from erroneously using a Connection after returning to the pool. Without them threads can corrupt unrelated transactions.
The JDBC specification requires that Statement amd ResultSet instances be closed automatically when a Connection is closed.
DBCP2 and Tomcat can track Statement objects, do not do this by default. Neither tracks ResultSet instances. When Statement tracking is enabled, the implementation based on WeakReferences can loose track of and/or generate OutOfMemoryErrors under heavy load.
Check SQLExceptions for SQL-92 and vendor-specific disconnection codes.
Tomcat does not support it. DBCP2 does, but comes with no defaults. So, it is up to the user to track down and specify all the various esoteric SQLState codes.
Only HikariCP clears SQL warnings because connection usage.