Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure continuous data availability for CEMS and 860M archives #218

Closed
2 tasks
Tracked by #2902
jdangerx opened this issue Dec 4, 2023 · 2 comments · Fixed by #232
Closed
2 tasks
Tracked by #2902

Ensure continuous data availability for CEMS and 860M archives #218

jdangerx opened this issue Dec 4, 2023 · 2 comments · Fixed by #232
Assignees
@aesharpe

Comments

@jdangerx
Copy link
Member

jdangerx commented Dec 4, 2023 •
edited by cmgosnell
Loading

CEMS data now includes 1 file for each quarter (~110 files). One useful additional dataset specific validation to add to DatasetSpecificValidation would be to check that that there are no quarters of missing data (that each quarter of CEMS data from the first partition to the last is present). This check could be extended to 860M data as well, to ensure that there are no missing months of data.

Tasks
Preview Give feedback
@jdangerx jdangerx mentioned this issue Dec 4, 2023
Closed
2 tasks
@jdangerx
Copy link
Member Author

jdangerx commented Dec 4, 2023

@e-belfer it would be helpful if you could dump some context for the CEMS check you had in mind!

@e-belfer e-belfer changed the title Identify and implement any dataset-specific validation criteria required (e.g. all quarters present for CEMS). Ensure continuous data availability for CEMS and 860M archives Dec 6, 2023
@e-belfer e-belfer transferred this issue from catalyst-cooperative/pudl Dec 6, 2023
@aesharpe aesharpe moved this from Backlog to In progress in Catalyst Megaproject Dec 13, 2023
@aesharpe
Copy link
Member

See #214 for success conditions

@aesharpe aesharpe linked a pull request Dec 15, 2023 that will close this issue
Validate cems archive #232
Merged
@e-belfer e-belfer moved this from In progress to In review in Catalyst Megaproject Dec 18, 2023
@e-belfer e-belfer moved this from In review to In progress in Catalyst Megaproject Dec 18, 2023
@github-project-automation github-project-automation bot moved this from In progress to Done in Catalyst Megaproject Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aesharpe aesharpe

Labels
None yet
Projects
Catalyst Megaproject
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Validate cems archive catalyst-cooperative/pudl-archiver
2 participants
@jdangerx @aesharpe