lightning: sample once parquet file #56205

zeminzhou · 2024-09-22T11:57:59Z

What problem does this PR solve?

Issue Number: close #56104

Problem Summary:

What changed and how does it work?

From test, before this pr, lightning importing ten files in parquet format needs to takes 4m40s in the preparation stage. After this pr, lightning importing ten files in parquet format needs to takes 1m10s.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: zeminzhou <zhouzemin@pingcap.cn>

tiprow · 2024-09-22T11:58:16Z

Hi @zeminzhou. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-09-22T12:12:18Z

Codecov Report

Attention: Patch coverage is 55.55556% with 12 lines in your changes missing coverage. Please review.

Project coverage is 74.1398%. Comparing base (a97aa45) to head (f4fa1ac).
Report is 571 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #56205        +/-   ##
================================================
+ Coverage   72.9898%   74.1398%   +1.1499%     
================================================
  Files          1611       1713       +102     
  Lines        447632     512905     +65273     
================================================
+ Hits         326726     380267     +53541     
- Misses       100915     110772      +9857     
- Partials      19991      21866      +1875

Flag	Coverage Δ
integration	`47.1132% <55.5555%> (?)`
unit	`73.0296% <15.3846%> (+0.9460%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (-0.1895%)`	⬇️
parser	`∅ <ø> (∅)`
br	`42.2034% <ø> (-3.6727%)`	⬇️

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

zeminzhou · 2024-09-24T06:32:53Z

/cc @D3Hunter @lance6716

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

lance6716

rest lgtm

pkg/lightning/mydump/loader.go

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

zeminzhou · 2024-10-12T02:27:00Z

/test pull-lightning-integration-test

tiprow · 2024-10-12T02:27:21Z

@zeminzhou: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pkg/lightning/mydump/loader.go

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

pkg/lightning/mydump/loader.go

lightning/pkg/importer/import.go

lance6716 · 2024-11-13T08:40:04Z

ping @zeminzhou

ti-chi-bot · 2024-11-13T13:34:10Z

@zeminzhou: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-e2e-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_lightning_integration_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test pull-integration-ddl-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zeminzhou · 2024-11-13T13:34:13Z

/test unit-test

tiprow · 2024-11-13T13:34:28Z

@zeminzhou: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-integration-ddl-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2024-11-13T13:34:34Z

@zeminzhou: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zeminzhou · 2024-11-14T02:08:58Z

/test pull-integration-ddl-test

tiprow · 2024-11-14T02:09:21Z

@zeminzhou: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-integration-ddl-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter

still too slow to read all parquet files to get row count in serial, we can optimize it in later pr

ti-chi-bot · 2024-11-18T02:10:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~lightning/OWNERS~~ [D3Hunter,lance6716]
~~pkg/lightning/OWNERS~~ [D3Hunter,lance6716]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-11-18T02:10:46Z

[LGTM Timeline notifier]

Timeline:

2024-10-11 12:00:35.931573563 +0000 UTC m=+13233.080483390: ☑️ agreed by lance6716.
2024-11-18 02:10:45.665205993 +0000 UTC m=+840607.856074986: ☑️ agreed by D3Hunter.

tiprow · 2024-11-18T02:12:44Z

@zeminzhou: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`f4fa1ac`	link	true	`/test fast_test_tiprow`
tidb_parser_test	`f4fa1ac`	link	true	`/test tidb_parser_test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

zeminzhou · 2024-11-18T02:55:38Z

/test fast_test_tiprow

ti-chi-bot · 2024-11-18T02:55:41Z

@zeminzhou: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-e2e-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_lightning_integration_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zeminzhou · 2024-11-18T02:55:44Z

/test tidb_parser_test

ti-chi-bot · 2024-11-18T02:55:47Z

@zeminzhou: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-e2e-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_lightning_integration_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test tidb_parser_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiprow · 2024-11-18T02:56:00Z

@zeminzhou: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2024-11-18T02:56:05Z

@zeminzhou: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test tidb_parser_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zeminzhou · 2024-11-18T02:56:44Z

/ok-to-test

fzzf678 · 2024-11-18T03:03:27Z

/retest

tiprow · 2024-11-18T03:03:47Z

@fzzf678: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-11-18T03:25:41Z

In response to a cherrypick label: new pull request created to branch release-8.5: #57446.

close #56104

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2024-12-03T08:07:21Z

In response to a cherrypick label: new pull request created to branch release-7.5: #57921.

close #56104

sample once

ba81619

Signed-off-by: zeminzhou <zhouzemin@pingcap.cn>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed do-not-merge/needs-tests-checked size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed do-not-merge/needs-tests-checked labels Sep 22, 2024

zeminzhou added 2 commits September 24, 2024 14:06

fix bazel_build

7587cbf

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

don't return error

8fcabe2

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

ti-chi-bot bot requested review from D3Hunter and lance6716 September 24, 2024 06:32

remove unused changes

c55ca99

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Sep 24, 2024

zeminzhou mentioned this pull request Sep 25, 2024

lightning might stuck for hours when importing parquet files from cloud storage #56104

Closed

sample every table

bc15c74

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

lance6716 reviewed Oct 10, 2024

View reviewed changes

pkg/lightning/mydump/loader.go Show resolved Hide resolved

pkg/lightning/mydump/loader.go Outdated Show resolved Hide resolved

fix comment

c992438

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

lance6716 approved these changes Oct 11, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Oct 11, 2024

D3Hunter reviewed Oct 21, 2024

View reviewed changes

pkg/lightning/mydump/loader.go Outdated Show resolved Hide resolved

call ReadParquetFileRowCountByFile once

ec6d507

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

D3Hunter reviewed Oct 21, 2024

View reviewed changes

pkg/lightning/mydump/loader.go Show resolved Hide resolved

D3Hunter reviewed Oct 25, 2024

View reviewed changes

lightning/pkg/importer/import.go Show resolved Hide resolved

ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Nov 13, 2024

D3Hunter approved these changes Nov 18, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 18, 2024

ti-chi-bot bot merged commit 0a9a231 into pingcap:master Nov 18, 2024
22 of 24 checks passed

ti-chi-bot mentioned this pull request Nov 18, 2024

lightning: sample once parquet file (#56205) #57446

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Nov 19, 2024

lightning: sample once parquet file (#56205) (#57446)

213d42b

close #56104

ti-chi-bot bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Dec 3, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 3, 2024

This is an automated cherry-pick of pingcap#56205

af1effd

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Dec 3, 2024

lightning: sample once parquet file (#56205) #57921

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Dec 6, 2024

lightning: sample once parquet file (#56205) (#57921)

e5fd514

close #56104

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lightning: sample once parquet file #56205

lightning: sample once parquet file #56205

zeminzhou commented Sep 22, 2024 •

edited

Loading

tiprow bot commented Sep 22, 2024

codecov bot commented Sep 22, 2024 •

edited

Loading

zeminzhou commented Sep 24, 2024

lance6716 left a comment

zeminzhou commented Oct 12, 2024

tiprow bot commented Oct 12, 2024

lance6716 commented Nov 13, 2024

ti-chi-bot bot commented Nov 13, 2024

zeminzhou commented Nov 13, 2024

tiprow bot commented Nov 13, 2024

tiprow bot commented Nov 13, 2024

zeminzhou commented Nov 14, 2024

tiprow bot commented Nov 14, 2024

D3Hunter left a comment

ti-chi-bot bot commented Nov 18, 2024

ti-chi-bot bot commented Nov 18, 2024

tiprow bot commented Nov 18, 2024

zeminzhou commented Nov 18, 2024

ti-chi-bot bot commented Nov 18, 2024

zeminzhou commented Nov 18, 2024

ti-chi-bot bot commented Nov 18, 2024

tiprow bot commented Nov 18, 2024

tiprow bot commented Nov 18, 2024

zeminzhou commented Nov 18, 2024

fzzf678 commented Nov 18, 2024

tiprow bot commented Nov 18, 2024

ti-chi-bot commented Nov 18, 2024

ti-chi-bot commented Dec 3, 2024

lightning: sample once parquet file #56205

lightning: sample once parquet file #56205

Conversation

zeminzhou commented Sep 22, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Sep 22, 2024

codecov bot commented Sep 22, 2024 • edited Loading

Codecov Report

zeminzhou commented Sep 24, 2024

lance6716 left a comment

Choose a reason for hiding this comment

zeminzhou commented Oct 12, 2024

tiprow bot commented Oct 12, 2024

lance6716 commented Nov 13, 2024

ti-chi-bot bot commented Nov 13, 2024

zeminzhou commented Nov 13, 2024

tiprow bot commented Nov 13, 2024

tiprow bot commented Nov 13, 2024

zeminzhou commented Nov 14, 2024

tiprow bot commented Nov 14, 2024

D3Hunter left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 18, 2024

ti-chi-bot bot commented Nov 18, 2024

[LGTM Timeline notifier]

tiprow bot commented Nov 18, 2024

zeminzhou commented Nov 18, 2024

ti-chi-bot bot commented Nov 18, 2024

zeminzhou commented Nov 18, 2024

ti-chi-bot bot commented Nov 18, 2024

tiprow bot commented Nov 18, 2024

tiprow bot commented Nov 18, 2024

zeminzhou commented Nov 18, 2024

fzzf678 commented Nov 18, 2024

tiprow bot commented Nov 18, 2024

ti-chi-bot commented Nov 18, 2024

ti-chi-bot commented Dec 3, 2024

zeminzhou commented Sep 22, 2024 •

edited

Loading

codecov bot commented Sep 22, 2024 •

edited

Loading