Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix calculating the report_date in demand_hourly_pa_ferc714 #2999

Merged
merged 2 commits into from
Nov 1, 2023

Conversation

rousik
Copy link
Collaborator

@rousik rousik commented Nov 1, 2023

For some reason report_date could be missing for some number of records (0.7%). This addresses that problem by changing the way we align the reports to year boundaries.

@rousik rousik requested a review from zaneselvans November 1, 2023 16:53
Copy link

codecov bot commented Nov 1, 2023

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (548b958) 88.6% compared to head (bf77d1a) 88.6%.
Report is 62 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff           @@
##             dev   #2999    +/-   ##
======================================
  Coverage   88.6%   88.6%            
======================================
  Files         91      91            
  Lines      10854   10991   +137     
======================================
+ Hits        9618    9749   +131     
- Misses      1236    1242     +6     
Files Coverage Δ
src/pudl/metadata/fields.py 100.0% <ø> (ø)
src/pudl/transform/ferc714.py 100.0% <100.0%> (ø)
src/pudl/transform/params/ferc1.py 100.0% <ø> (ø)
src/pudl/output/ferc1.py 88.2% <69.2%> (-0.5%) ⬇️
src/pudl/transform/ferc1.py 96.7% <94.8%> (+<0.1%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@zaneselvans zaneselvans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you happen to run the full FERC-714 ETL locally for all years with these changes?

It looks like there were NaT values included in the 2020 data which got processed by the integration tests in CI, and it passed, so it seems like it will probably work in general. 🤞🏼

@rousik
Copy link
Collaborator Author

rousik commented Nov 1, 2023

Did you happen to run the full FERC-714 ETL locally for all years with these changes?

It looks like there were NaT values included in the 2020 data which got processed by the integration tests in CI, and it passed, so it seems like it will probably work in general. 🤞🏼

I have ran the affected table locally with dagster with etl_fast.yml. I'm expecting this to be low risk WRT possibility of this breaking on additional data.

@rousik rousik merged commit e848d64 into dev Nov 1, 2023
9 checks passed
@zaneselvans zaneselvans deleted the fix-hourly-demand-report-date branch November 2, 2023 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants