Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HYC-1699 - Analytics cache script #985

Merged
merged 20 commits into from
Jul 11, 2023
Merged

HYC-1699 - Analytics cache script #985

merged 20 commits into from
Jul 11, 2023

Conversation

bbpennel
Copy link
Member

@bbpennel bbpennel commented Jul 5, 2023 •
edited
Loading

https://jira.lib.unc.edu/browse/HYC-1699

  • Adds a script that pulls data from GA3 (UA) for every work and fileset in the repository, populating the local database cache
  • Performs this work in parallel
  • Adds analytics.yml config back in for GA3, so that the application can be used with GA3 or 4

@bbpennel
Add task for updating the analytics cache for all objects in the repo
9ad43e5
@bbpennel
Fix names
50725dd
@bbpennel
Eager load legato model classes, so that we can retrieve pageviews an…
73efde8 
…d downloads. Add average time tracking
@bbpennel
Update the script to process ids in parallel. Add locks around shared…
df83931 
… resources, including writing to the progress log.
@bbpennel
Merge branch 'main' into hyc-1699-cache
9b2cf2b
@bbpennel
Use a queue subclass for getting ids
1bb0208
@bbpennel
Add retries for oauth errors
a073228
@bbpennel
Fix case
fd6ba23
@bbpennel
Progress to next id when skipping completed ids. Eager load more classes
0c89124
@bbpennel
Handle deleted items. Load more classes
bb3a058
@bbpennel
Add in config for regular google
8fbd8e1
@bbpennel
Add blank value for privkey_value so it'll start up
1181b38
@bbpennel
Add test for resumption case. Rubocop fixes
a3b11e7
@bbpennel
Allow timeout errors when getting stats to be raised or suppressed, s…
ab333ee 
…o that the caching script can retry these errors. Add tests
@bbpennel bbpennel changed the title HYC-1699 - Analytics cache script WIP: HYC-1699 - Analytics cache script Jul 7, 2023
@bbpennel
Use variable delays after errors, depending on if we've reached quota…
a1ff6b2 
…s with google. Adjust timer reporting to show time for the last batch rather than for all time
@bbpennel bbpennel changed the title WIP: HYC-1699 - Analytics cache script HYC-1699 - Analytics cache script Jul 10, 2023
sharonluong
sharonluong approved these changes Jul 11, 2023
View reviewed changes
Copy link
Contributor

@sharonluong sharonluong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@sharonluong sharonluong merged commit b1f1d14 into main Jul 11, 2023
@bbpennel bbpennel deleted the hyc-1699-cache branch October 10, 2023 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sharonluong sharonluong sharonluong approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bbpennel @sharonluong