import into failed when inject pdleader io delay 1s last for 1mins

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb_enable_dist_task='on'
1、import into
2、inject pdleader io delay 1s last for 1mins

case config
tag: "ha-test-import-into"
cases:
- name: ha_pd(leader)_io_delay(1s)
faultType: iochaos_io_delay(volumePath:/var/lib/pd,path:/var/lib/pd/data/**/*,delay:1s,percent:100)
warmUpTime: 1m
selector: pd(leader)
period: "@every 5m"
faultDuration: 1m
faultTotalRunTime: 20m
importInto:
- db: "importtest"
schema: "CREATE TABLE user_data1 (
uid bigint(20) unsigned NOT NULL AUTO_INCREMENT,
data mediumblob NOT NULL,
update_data blob NOT NULL,
create_time datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
update_time datetime DEFAULT NULL,
PRIMARY KEY (uid) /*T![clustered_index] CLUSTERED /,
KEY k_1 (update_time)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin AUTO_INCREMENT=5850196;"
cmd: "import into user_data1 from 's3://brie/lightning-mhy-100G-csv/sysbench.user_data1.000001.csv?access-key=xxx\u0026secret-access-key=xxx\u0026endpoint=http://minio.pingcap.net:9001\u0026force-path-style=true' WITH DETACHED,skip_rows=1"
timeout: 120m
crc32Checksum: 3582177512

2. What did you expect to see? (Required)

impor into can success

3. What did you see instead (Required)

import into failed when inject pdleader io delay 1s last for 1mins

the status of import job is not finished or running (now: 2023-11-05 05:48:04, jobId: 30011, step: importing, status: failed)
operatorLogs:
[2023-11-05 05:33:33] ###### start import into
import into user_data1 from 's3://brie/lightning-mhy-100G-csv/sysbench.user_data1.000001*.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://minio.pingcap.net:9001&force-path-style=true' WITH DETACHED,skip_rows=1
[2023-11-05 05:33:33] ###### wait for import job to finish
[2023-11-05 05:48:04] ###### wait for import job to finish failed
select id, step, status from mysql.tidb_import_jobs where start_time >= '2023-11-05 05:33:33'
jobId: 30011, step: importing, status: failed

4. What is your TiDB version? (Required)

git hash： aa93091

[Lily2025]

/assign D3Hunter

[D3Hunter]

the body cannot be reused

[2023/11/05 05:47:34.890 +08:00] [ERROR] [backend.go:371] ["import failed"] [engineTag=`importtest`.`user_data1`:0] [engineUUID=9a09d9d7-e635-547b-86cb-e8564a7e4a95] [retryCnt=0] [takeTime=49.215683303s] [error="[400] {\n  \"code\": \"input\",\n  \"msg\": \"unexpected end of JSON input\",\n  \"data\": {\n    \"Offset\": 0\n  }\n}\n http://tc-pd:2379/pd/api/v1/config/region-label/rule: [BR:PD:ErrPDInvalidResponse]PD invalid response"]

tidb/br/pkg/pdutil/pd.go

Line 160 in b93b2f5

req, err = http.NewRequestWithContext(ctx, method, reqURL, body)