Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix secret config load and add clean-up for re-run #1

Merged
merged 3 commits into from
Apr 5, 2024

Conversation

stevenwinship
Copy link
Collaborator

@stevenwinship stevenwinship commented Apr 5, 2024

Fix loading of secrets config file

Add code to allow Dataverse to re-trigger processing of archived log files. Alleviate the need to manually edit the state file and delete the year-month database

Needed for IQSS/dataverse.harvard.edu#3

@stevenwinship stevenwinship self-assigned this Apr 5, 2024
@stevenwinship stevenwinship requested review from pdurbin and qqmyers April 5, 2024 14:03
@qqmyers
Copy link
Member

qqmyers commented Apr 5, 2024

@stevenwinship, given that you're adding a new flag, can you document it in the README and update the instructions about how to manage state, e.g. in

counter-processor/README.md

Lines 126 to 131 in 4c662cd

If you wish to completely reprocess and submit a month's data from log files you can:
1. Manually send a DELETE request to the hub for an id to remove a report.
2. Remove the state data from the json file for a particular year-month.
3. Remove the appropriate month's sqlite database from the file system
4. Reprocess the month. If it's after the month, use *year_month* for the months report you'd like.
.

@stevenwinship
Copy link
Collaborator Author

Updated the README and also added the DELETE from DataCite to the re-run automation

Copy link
Member

@qqmyers qqmyers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. An upgrade to remove the deprecated/now removed yaml.load() call and a new flag to be able to clear state to rerun a month's processing.

@stevenwinship stevenwinship merged commit c11e810 into main Apr 5, 2024
2 checks passed
@stevenwinship stevenwinship deleted the mdc-fixes-and-automated-features branch April 5, 2024 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants