Features/1550 extension of load npy from path to csv-files with same functionality #1551

Reisii · 2024-07-02T12:33:05Z

Due Diligence

General:
- title of the PR is suitable to appear in the Release Notes
Implementation:
- unit tests: tested with and without input function
- unit tests: multiple dtypes tested
- documentation updated where needed

Description

Adds the same feature of PR#1388 now for .csv-files. Loads multiple .csv-files at once and concatenates them along an axis given as an input by the user. A function can be passed on as a parameter. The function would edit the pandas.Dataframe in way that it can be converted into a NumPy-Array without any problems

Issue/s resolved: #1550

Changes proposed:

Type of change

Memory requirements

Performance

Does this change modify the behaviour of other functions? If so, which?

no

…h_to_csv-fIles_with_same_functionality

for more information, see https://pre-commit.ci

github-actions · 2024-07-04T15:11:50Z

Thank you for the PR!

codecov · 2024-07-04T16:16:43Z

Codecov Report

Attention: Patch coverage is 91.66667% with 3 lines in your changes missing coverage. Please review.

Project coverage is 92.06%. Comparing base (9d3af11) to head (956e3d0).
Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
heat/core/io.py	91.66%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1551      +/-   ##
==========================================
- Coverage   92.07%   92.06%   -0.01%     
==========================================
  Files          83       83              
  Lines       12144    12180      +36     
==========================================
+ Hits        11181    11214      +33     
- Misses        963      966       +3

Flag	Coverage Δ
unit	`92.06% <91.66%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

heat/core/io.py

heat/core/tests/test_io.py

mrfh92 · 2024-07-08T07:58:36Z

discussed in the PR talk: we will make pandas an optional dependency, as already the case for h5py etc.

…les_with_same_functionality' of github.com:helmholtz-analytics/heat into features/1550-Extension_of_load_npy_from_path_to_csv-fIles_with_same_functionality

…h_to_csv-fIles_with_same_functionality

github-actions · 2024-07-09T09:08:58Z

Thank you for the PR!

github-actions · 2024-07-09T14:43:33Z

Thank you for the PR!

github-actions · 2024-07-10T10:50:27Z

Thank you for the PR!

Reisii · 2024-07-15T13:59:25Z

@Reisii something failed on the CUDA-runner. Can you reproduce this on our workstation?

So it works perfectly fine with the workstation, I'll just try to test out using ht.MPI_WORLD.Barrier() on different places

github-actions · 2024-07-15T14:02:50Z

Thank you for the PR!

just done to let CI run

…h_to_csv-fIles_with_same_functionality

…les_with_same_functionality' of github.com:helmholtz-analytics/heat into features/1550-Extension_of_load_npy_from_path_to_csv-fIles_with_same_functionality

github-actions · 2024-07-16T13:06:35Z

Thank you for the PR!

…h_to_csv-fIles_with_same_functionality

github-actions · 2024-07-19T14:06:54Z

Thank you for the PR!

…h_to_csv-fIles_with_same_functionality

github-actions · 2024-07-22T07:29:14Z

Thank you for the PR!

…h_to_csv-fIles_with_same_functionality

github-actions · 2024-08-12T12:50:56Z

Thank you for the PR!

…h_to_csv-fIles_with_same_functionality

github-actions · 2024-08-13T09:10:33Z

Thank you for the PR!

mrfh92 · 2024-08-13T11:14:28Z

lines 1187/88, 1268/87 are not covered by tests so far. It looks like that there is no remainder in the devision (?)
That lines 1203-1209 are not covered can't be resolved as this would require a different setup without h5py, which is currently out of scope

…les_with_same_functionality' of github.com:helmholtz-analytics/heat into features/1550-Extension_of_load_npy_from_path_to_csv-fIles_with_same_functionality

github-actions · 2024-08-13T12:27:04Z

Thank you for the PR!

…h_to_csv-fIles_with_same_functionality

github-actions · 2024-08-21T12:13:08Z

Thank you for the PR!

…h_to_csv-fIles_with_same_functionality

github-actions · 2024-08-26T08:18:11Z

Thank you for the PR!

…h_to_csv-fIles_with_same_functionality

mrfh92

Changes look fine :)
Thanks for your work 👍

github-actions · 2024-08-26T13:46:07Z

Thank you for the PR!

Now with docstrings

683cf22

Reisii changed the title ~~Features/1550 extension of load npy from path to csv f iles with same functionality~~ Features/1550 extension of load npy from path to csv-files with same functionality Jul 2, 2024

mrfh92 assigned mrfh92 and Reisii Jul 2, 2024

Reisii and others added 6 commits July 3, 2024 12:37

Removed blank line after docstring

4c1a242

Testing for callable function

98052b3

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

3f5e349

…h_to_csv-fIles_with_same_functionality

[pre-commit.ci] auto fixes from pre-commit.com hooks

4c6f255

for more information, see https://pre-commit.ci

Update setup.py

c6fbdf0

[pre-commit.ci] auto fixes from pre-commit.com hooks

82d45d8

for more information, see https://pre-commit.ci

mrfh92 self-requested a review July 4, 2024 20:19

mrfh92 reviewed Jul 5, 2024

View reviewed changes

heat/core/io.py Outdated Show resolved Hide resolved

mrfh92 reviewed Jul 5, 2024

View reviewed changes

heat/core/io.py Outdated Show resolved Hide resolved

mrfh92 reviewed Jul 5, 2024

View reviewed changes

heat/core/io.py Show resolved Hide resolved

mrfh92 reviewed Jul 5, 2024

View reviewed changes

heat/core/io.py Outdated Show resolved Hide resolved

mrfh92 reviewed Jul 5, 2024

View reviewed changes

heat/core/tests/test_io.py Show resolved Hide resolved

mrfh92 added the PR talk label Jul 8, 2024

Reisii added 4 commits July 9, 2024 10:11

Added test for other dtype and list comprehension commented

0b61f8f

Merge branch 'features/1550-Extension_of_load_npy_from_path_to_csv-fI…

521f928

…les_with_same_functionality' of github.com:helmholtz-analytics/heat into features/1550-Extension_of_load_npy_from_path_to_csv-fIles_with_same_functionality

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

dafc6c9

…h_to_csv-fIles_with_same_functionality

List compr, TypError when non-callable, pandas optional

fd2a007

Reisii added 2 commits July 9, 2024 14:48

pandas optional requirement

aa54ed6

Fixed issue with pandas

c9cfe24

Importing pandas when available

30745e4

mrfh92 self-requested a review July 15, 2024 14:03

Reisii added 3 commits July 16, 2024 14:57

Added a ht.MPI_WORLD.Barrier() where needed

a79c3cd

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

bbd5496

…h_to_csv-fIles_with_same_functionality

Merge branch 'features/1550-Extension_of_load_npy_from_path_to_csv-fI…

9735db6

…les_with_same_functionality' of github.com:helmholtz-analytics/heat into features/1550-Extension_of_load_npy_from_path_to_csv-fIles_with_same_functionality

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

527c6af

…h_to_csv-fIles_with_same_functionality

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

1ee06a8

…h_to_csv-fIles_with_same_functionality

Reisii added 2 commits August 12, 2024 09:29

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

aed50e8

…h_to_csv-fIles_with_same_functionality

Fixed unbalanced Loading for csv. files

5ac77b8

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

70dd21e

…h_to_csv-fIles_with_same_functionality

Reisii added 2 commits August 13, 2024 14:21

Tests with uneven amount of files

a0b037a

Merge branch 'features/1550-Extension_of_load_npy_from_path_to_csv-fI…

176bf42

…les_with_same_functionality' of github.com:helmholtz-analytics/heat into features/1550-Extension_of_load_npy_from_path_to_csv-fIles_with_same_functionality

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

58d5e62

…h_to_csv-fIles_with_same_functionality

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

4608847

…h_to_csv-fIles_with_same_functionality

Merge branch 'main' into features/1550-Extension_of_load_npy_from_pat…

956e3d0

…h_to_csv-fIles_with_same_functionality

mrfh92 approved these changes Aug 26, 2024

View reviewed changes

mrfh92 merged commit cbdf49a into main Aug 29, 2024
42 checks passed

mrfh92 deleted the features/1550-Extension_of_load_npy_from_path_to_csv-fIles_with_same_functionality branch August 29, 2024 11:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Features/1550 extension of load npy from path to csv-files with same functionality #1551

Features/1550 extension of load npy from path to csv-files with same functionality #1551

Reisii commented Jul 2, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024

codecov bot commented Jul 4, 2024 •

edited

Loading

mrfh92 commented Jul 8, 2024

github-actions bot commented Jul 9, 2024

github-actions bot commented Jul 9, 2024

github-actions bot commented Jul 10, 2024

Reisii commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 22, 2024

github-actions bot commented Aug 12, 2024

github-actions bot commented Aug 13, 2024

mrfh92 commented Aug 13, 2024

github-actions bot commented Aug 13, 2024

github-actions bot commented Aug 21, 2024

github-actions bot commented Aug 26, 2024

mrfh92 left a comment

github-actions bot commented Aug 26, 2024

Features/1550 extension of load npy from path to csv-files with same functionality #1551

Features/1550 extension of load npy from path to csv-files with same functionality #1551

Conversation

Reisii commented Jul 2, 2024 • edited Loading

Due Diligence

Description

Changes proposed:

Type of change

Memory requirements

Performance

Does this change modify the behaviour of other functions? If so, which?

github-actions bot commented Jul 4, 2024

codecov bot commented Jul 4, 2024 • edited Loading

Codecov Report

mrfh92 commented Jul 8, 2024

github-actions bot commented Jul 9, 2024

github-actions bot commented Jul 9, 2024

github-actions bot commented Jul 10, 2024

Reisii commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 22, 2024

github-actions bot commented Aug 12, 2024

github-actions bot commented Aug 13, 2024

mrfh92 commented Aug 13, 2024

github-actions bot commented Aug 13, 2024

github-actions bot commented Aug 21, 2024

github-actions bot commented Aug 26, 2024

mrfh92 left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 26, 2024

Reisii commented Jul 2, 2024 •

edited

Loading

codecov bot commented Jul 4, 2024 •

edited

Loading