Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add benchmark result for batch write #1025

Merged
merged 3 commits into from
Aug 16, 2019
Merged

add benchmark result for batch write #1025

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2c9e568
add benchmark
marsishandsome Aug 15, 2019
26ea09a
address code review
marsishandsome Aug 15, 2019
350e831
Merge branch 'master' into feature/batch-write-benchmark
marsishandsome Aug 15, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/datasource_api_userguide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,3 +341,48 @@ The full conversion metrics is as follows.
| LONGBLOB | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
| ENUM | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: |
| SET | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |

## Write Benchmark
Tested on 4 machines as follows:

```
Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz * 2 = 40Vu
12 * 16G = 188G
```

`FIO` test result:

```
WRITE: bw=705MiB/s (740MB/s), 705MiB/s-705MiB/s (740MB/s-740MB/s), io=20.0GiB (21.5GB), run=29034-29034msec
```

The table schema is:
```
CREATE TABLE ORDERS (O_ORDERKEY INTEGER NOT NULL,
O_CUSTKEY INTEGER NOT NULL,
O_ORDERSTATUS CHAR(1) NOT NULL,
O_TOTALPRICE DECIMAL(15,2) NOT NULL,
O_ORDERDATE DATE NOT NULL,
O_ORDERPRIORITY CHAR(15) NOT NULL,
O_CLERK CHAR(15) NOT NULL,
O_SHIPPRIORITY INTEGER NOT NULL,
O_COMMENT VARCHAR(79) NOT NULL);

```

### TiSpark Write Benchmark

| count(*) | data size | parallel number | prepare(s) | prewrite (s) | commit (s) | total (s) |
| ----------- | --------- | --------------- | ---------- | ------------ | ---------- | --------- |
| 1,500,000 | 165M | 2 | 17 | 68 | 62 | 148 |
| 15,000,000 | 1.7G | 24 | 49 | 157 | 119 | 326 |
| 150,000,000 | 17G | 120 | 630 | 1236 | 1098 | 2964 |


## Spark with JDBC Benchmark

| count(*) | data size | parallel number | spark jdbc write (s) | comments |
| ----------- | --------- | --------------- | -------------------- | ----------------------------------- |
| 1,500,000 | 165M | 24 | 22 | |
| 15,000,000 | 1.7G | 24 | 411 | use 120 parallel will cause KV Busy |
| 150,000,000 | 17G | 24 | 2936 | use 120 parallel will cause KV Busy |