[FOLLOW-UP] Support for building DMatrix from Apache Arrow data format #7512

xwu99 · 2021-12-16T03:51:24Z

Continue work for #7283 #5667

xwu99 · 2021-12-16T04:00:12Z

@trivialfis This is a follow-up work for #7283, Could you help to kick off CI? It looks it is still using Jenkins there but my fork is using github actions? There is a type check error in github actions but not relavent to this PR.

trivialfis · 2021-12-16T04:39:50Z

Could you rebase to master?

xwu99 · 2021-12-16T04:49:35Z

rebased. I didn't change xgboost/dask.py. Could you approve ci again? @trivialfis

trivialfis · 2021-12-16T05:26:04Z

Submitted a fix #7513 .

trivialfis · 2021-12-17T05:20:41Z

I saw some compiler warnings with gcc 9.3 and nvcc 11.4, some of them like pointless comparison are quite useful. Please review and address them.

log.txt

xwu99 · 2021-12-20T02:31:26Z

@trivialfis I have enabled test_with_arrow for CI.
Github Actions: "macos-10.15" passed. But "windows-2016" just skipped the test.
Jenkins: "Win64 Test" failed.
It seems they have very different packages installed for different OSes. I need more time to debug this.

trivialfis · 2021-12-20T06:04:16Z

Thanks for the update. Will review the PR later. I don't mind disabling Windows support for now.

xwu99 · 2022-01-04T06:21:37Z

Thanks for the update. Will review the PR later. I don't mind disabling Windows support for now.

I have fixed for win64_cpu_test.yml (added cffi) but looks there is some other packages missing in win64_test.yml in order to run with arrow. The test case error silently without useful messages.

trivialfis · 2022-01-05T02:49:51Z

It seems to be hanging on both Linux and Windows.

https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-7512/18/pipeline/180

xwu99 · 2022-01-05T08:21:33Z

It seems to be hanging on both Linux and Windows.

https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-7512/18/pipeline/180

Thanks, you are right, I will look into details.

xwu99 · 2022-01-05T14:25:36Z

It seems to be hanging on both Linux and Windows.

https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-7512/18/pipeline/180

@trivialfis Hi, my Jenkinsfile-win64 has changed for debugging, I have added -DUSE_ARROW=ON to cmake and also changed pytest line.

But the change is not effective, Jenkins still execute the old Jenkinsfile-win64 file. I am not familiar with Jenkins, anything missing?

https://xgboost-ci.net/blue/organizations/jenkins/xgboost-win64/detail/PR-7512/22/pipeline/39

trivialfis · 2022-01-05T14:27:06Z

@hcho3 Could you please help take a look if time allows?

hcho3 · 2022-01-05T14:44:03Z

The test server should now use the updated Jenkinsfile.

xwu99 · 2022-01-06T03:05:08Z

The test server should now use the updated Jenkinsfile.

Thanks a lot, I will rebase to master next time. Seems Jenkins only pull in rebased one.

trivialfis · 2022-01-06T09:27:06Z

I have interrupted the win64 CI as it was approaching 3 hours.

xwu99 · 2022-01-06T09:34:17Z

It seems to be hanging on both Linux and Windows.
https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-7512/18/pipeline/180

Thanks, you are right, I will look into details.

The Linux one is fixed by adding -DUSE_ARROW=ON, for Win64 it passed yesterday by adding -DUSE_ARROW=ON https://xgboost-ci.net/blue/organizations/jenkins/xgboost-win64/detail/PR-7512/24/pipeline. but the same setting hang today.

trivialfis · 2022-01-06T09:54:07Z

Thanks a lot, I will rebase to master next time. Seems Jenkins only pull in rebased one.

Please let me know when you have rebased the branch. I will make some more changes based on your current PR. I couldn't open clean PR to your repository due to the upstream merge.

xwu99 · 2022-01-06T11:52:54Z

@trivialfis I have rebased now. Right now when I only test tests\python\test_with_arrow.py, it passed. I am not sure why it doesn't proceed as a whole (tests\python).

trivialfis · 2022-01-06T14:37:40Z

I am not sure why it doesn't proceed as a whole (tests\python).

I don't know either. Could you reproduce it locally?

I opened a PR for cmake script: xwu99#1 will open some other PRs later.

xwu99 · 2022-01-07T02:28:27Z

I am not sure why it doesn't proceed as a whole (tests\python).

I don't know either. Could you reproduce it locally?

I opened a PR for cmake script: xwu99#1 will open some other PRs later.

I will check Windows locally. Could you cancel the checks? it seems to hang.

xwu99 · 2022-01-09T05:50:29Z

I am not sure why it doesn't proceed as a whole (tests\python).

I don't know either. Could you reproduce it locally?
I opened a PR for cmake script: xwu99#1 will open some other PRs later.

I will check Windows locally. Could you cancel the checks? it seems to hang.
0
It's a hard to find bug. Digging out code locally on Win10 finally I found improrting modin.pandas mess up with pyarrow iterator on Windows. Attach the reproducer. hang on next(rb_iter) if comment out import modin line.

Environment:

OS: Windows 10
modin 0.11.3 py38haa95532_0
pyarrow 6.0.1 py38h0b4a547_7_cpu conda-forge

test_modin_arrow_hang.zip

For XGboost , the previous running of test_dmatrix.py: test_unknown_data (imported modin.pandas) impact the following test_with_arrow.

I have not found a workaround for modin and pyarrow to co-exist on Windows yet. If user import modin.pandas first, and then use arrow to load, it will hang.

[UPDATE] It also hangs on Linux. If I removed modin, all tests passed.
[UPDATE] only works with arrow 4.0 rather than arrow 6.0 (latest modin)

xwu99 · 2022-01-15T04:07:43Z

@trivialfis I have specified arrow to use 4.0 version in the CI instead of latest version due to importing modin resulting in hang for arrow 6.0. May need to file an issue for modin.

trivialfis · 2022-01-15T12:12:16Z

@xwu99 Thanks for the hard work on this. Do you know of any plan on supporting latest arrow and modin?

xwu99 · 2022-01-15T13:00:56Z

I will contact Modin to see if there is a fix.

[UPDATE] modin-project/modin#3982
[UPDATE] It's a pyarrow issue. workaround in https://issues.apache.org/jira/browse/ARROW-15362

trivialfis · 2022-01-17T19:14:50Z

@xwu99 Do you mind if I rebase your commits?

xwu99 · 2022-01-17T23:24:24Z

@xwu99 Do you mind if I rebase your commits?

Pls go ahead to do whatever you need. Btw. Modin team is looking into the hang issue.

xwu99 · 2022-01-19T03:53:27Z

@trivialfis I think I have fixed arrow hang issue from workaround https://issues.apache.org/jira/browse/ARROW-15362. the CI error seems irrelavant, you can rebase as you wish.

trivialfis · 2022-01-20T09:19:59Z

Now the arrow is built by default, clang tidy will check the code.

xwu99 · 2022-01-20T12:34:54Z

Now the arrow is built by default, clang tidy will check the code.

ok, I will run clang tidy locally to check.

trivialfis · 2022-01-20T12:57:01Z

I will follow up on this PR. Apologies for being slow here.

xwu99 · 2022-01-20T14:15:11Z

I will follow up on this PR. Apologies for being slow here.

Thanks. for clang-tidy some renames will do. pls let me know when you need me.

trivialfis · 2022-02-07T09:11:19Z

I made some cleanups to the PR by removing unused code. From my perspective, it's best to have everything in an adapter instead of DMatrix, but I don't think I will try to implement that in this release.

@hcho3 @RAMitchell Could you please take a look?

RAMitchell

It's a shame we can't use any existing data iterator structure to do this. After all the attempted refactoring of xgboost's data import mechanisms and DMatrix it seems to get more and more confusing and we are just adding API functions for each new format.

trivialfis · 2022-02-14T20:37:32Z

The arrow format can chunk the data arbitrarily for both rows and columns, which makes iterating it really difficult. We can move some of the code into adapter at the cost of some inefficiency. Will follow up tomorrow and rebase.

* Integrate with Arrow C data API. * Support Arrow dataset. * Support Arrow table. Co-authored-by: Xiaochang Wu <xiaochang.wu@intel.com> Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>

trivialfis · 2022-03-15T05:25:49Z

Thank you for the great work and apologies for the slow response.

pseudotensor · 2022-05-08T05:53:20Z

It's a shame we can't use any existing data iterator structure to do this. After all the attempted refactoring of xgboost's data import mechanisms and DMatrix it seems to get more and more confusing and we are just adding API functions for each new format.

And the datatable ingest became horribly slow after the refactor.

hcho3 · 2022-05-08T15:07:43Z

@pseudotensor Thanks for letting us know. Please file a new GitHub issue so that we can track it. Or is it related to #7642?

xwu99 force-pushed the arrow-support branch from 8fc7042 to 0ee3c61 Compare December 16, 2021 04:45

xwu99 force-pushed the arrow-support branch from 0ee3c61 to 3873074 Compare December 17, 2021 02:00

xwu99 force-pushed the arrow-support branch from 26fead0 to 4659857 Compare December 18, 2021 02:29

xwu99 force-pushed the arrow-support branch from 8f1a478 to 784f8b3 Compare January 6, 2022 09:38

xwu99 force-pushed the arrow-support branch from 629fc5b to 7b503ca Compare January 19, 2022 03:18

hcho3 mentioned this pull request Feb 2, 2022

Support for building DMatrix from Apache Arrow data format #7283

Closed

RAMitchell reviewed Feb 14, 2022

View reviewed changes

Support building SimpleDMatrix from Arrow data format

2fd751e

* Integrate with Arrow C data API. * Support Arrow dataset. * Support Arrow table. Co-authored-by: Xiaochang Wu <xiaochang.wu@intel.com> Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>

trivialfis force-pushed the arrow-support branch from 618717d to 2fd751e Compare March 14, 2022 09:53

trivialfis merged commit 613ec36 into dmlc:master Mar 15, 2022

Yard1 mentioned this pull request Jul 12, 2022

Add zero-copy DMatrix creation with Arrow ray-project/xgboost_ray#224

Open

thomasaarholt mentioned this pull request Jul 22, 2022

Documentation for creating DMatrix from Apache Arrow #8111

Closed

braaannigan mentioned this pull request Dec 7, 2022

Apache arrow support for I/O catboost/catboost#1984

Open

trivialfis mentioned this pull request Mar 15, 2023

Feature names not captured for Arrow table inputs #8910

Closed

jameslamb mentioned this pull request May 3, 2023

Create Dataset from Arrow format microsoft/LightGBM#3369

Closed

borchero mentioned this pull request Aug 12, 2023

[python-package] Allow to pass Arrow table as training data microsoft/LightGBM#6034

Merged

[FOLLOW-UP] Support for building DMatrix from Apache Arrow data format #7512

[FOLLOW-UP] Support for building DMatrix from Apache Arrow data format #7512

Conversation

xwu99 commented Dec 16, 2021

xwu99 commented Dec 16, 2021

trivialfis commented Dec 16, 2021

xwu99 commented Dec 16, 2021 • edited Loading

trivialfis commented Dec 16, 2021

trivialfis commented Dec 17, 2021

xwu99 commented Dec 20, 2021

trivialfis commented Dec 20, 2021

xwu99 commented Jan 4, 2022 • edited Loading

trivialfis commented Jan 5, 2022

xwu99 commented Jan 5, 2022

xwu99 commented Jan 5, 2022

trivialfis commented Jan 5, 2022

hcho3 commented Jan 5, 2022

xwu99 commented Jan 6, 2022

trivialfis commented Jan 6, 2022

xwu99 commented Jan 6, 2022

trivialfis commented Jan 6, 2022 • edited Loading

xwu99 commented Jan 6, 2022 • edited Loading

trivialfis commented Jan 6, 2022

xwu99 commented Jan 7, 2022 • edited Loading

xwu99 commented Jan 9, 2022 • edited Loading

xwu99 commented Jan 15, 2022

trivialfis commented Jan 15, 2022

xwu99 commented Jan 15, 2022 • edited Loading

trivialfis commented Jan 17, 2022

xwu99 commented Jan 17, 2022

xwu99 commented Jan 19, 2022

trivialfis commented Jan 20, 2022

xwu99 commented Jan 20, 2022

trivialfis commented Jan 20, 2022

xwu99 commented Jan 20, 2022

trivialfis commented Feb 7, 2022

RAMitchell left a comment

Choose a reason for hiding this comment

trivialfis commented Feb 14, 2022

trivialfis commented Mar 15, 2022

pseudotensor commented May 8, 2022 • edited Loading

hcho3 commented May 8, 2022

xwu99 commented Dec 16, 2021 •

edited

Loading

xwu99 commented Jan 4, 2022 •

edited

Loading

trivialfis commented Jan 6, 2022 •

edited

Loading

xwu99 commented Jan 6, 2022 •

edited

Loading

xwu99 commented Jan 7, 2022 •

edited

Loading

xwu99 commented Jan 9, 2022 •

edited

Loading

xwu99 commented Jan 15, 2022 •

edited

Loading

pseudotensor commented May 8, 2022 •

edited

Loading