Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return the total number of rows written in metadata. #306

Merged
merged 2 commits into from
Jul 19, 2024

Conversation

delucchi-cmu
Copy link
Contributor

Change Description

Relevant for astronomy-commons/hats-import#344

Return the total number of rows across all parquet files in the new parquet dataset.

Solution Description

Aggregates a count of number of rows inside the row group fragments, as they are aggregated in the parquet metadata.

Code Quality

  • I have read the Contribution Guide
  • My code follows the code style of this project
  • My code builds (or compiles) cleanly without any errors or warnings
  • My code contains relevant comments and necessary documentation

Copy link

codecov bot commented Jul 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.93%. Comparing base (b02ac54) to head (09bdeb9).
Report is 33 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #306   +/-   ##
=======================================
  Coverage   93.92%   93.93%           
=======================================
  Files          58       58           
  Lines        2057     2060    +3     
=======================================
+ Hits         1932     1935    +3     
  Misses        125      125           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

Before [b02ac54] After [ba3e56b] Ratio Benchmark (Parameter)
13.3±0.2ms 13.7±0.8ms 1.03 benchmarks.Suite.time_inner_pixel_alignment
119±0.8ms 120±1ms 1.01 benchmarks.time_test_alignment_even_sky
380±3ms 379±3ms 1 benchmarks.Suite.time_outer_pixel_alignment
979±2μs 984±4μs 1 benchmarks.time_test_cone_filter_multiple_order
20.6±0.8ms 20.4±0.4ms 0.99 benchmarks.MetadataSuite.time_load_partition_info_order6
77.0±0.2ms 76.6±0.7ms 0.99 benchmarks.MetadataSuite.time_load_partition_info_order7
77.2±0.2ms 76.7±1ms 0.99 benchmarks.MetadataSuite.time_load_partition_join_info
41.8±0.3ms 41.2±0.3ms 0.99 benchmarks.Suite.time_pixel_tree_creation
92.6±3ms 86.1±2ms 0.93 benchmarks.Suite.time_paths_creation

Click here to view all benchmarks.

@delucchi-cmu delucchi-cmu requested review from a team, aimalz, wenneman and smcguire-cmu and removed request for a team, aimalz and wenneman July 19, 2024 14:48
@delucchi-cmu delucchi-cmu merged commit 96fa40b into main Jul 19, 2024
12 checks passed
@delucchi-cmu delucchi-cmu deleted the hipscat-import/issue/344 branch July 19, 2024 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants