Followup to #7478: Move Cover Tars -> Zips

[mekarpeles]

Problem

A clear and concise description of what you want to happen

Open Library relies on Archive.org as a storage layer for millions of book covers and has a resolver at covers.openlibrary.org which looks in the Open Library cover DB for a cover and determines where it lives (either on OL disk or in an archive.org item).

Right now w're seeing what appears to be high volume, programatic access of our covers... Instead of downloading the entire tar, they are downloading a cover at a time rapidly. This is causing performance issues on Archive.org nodes because accessing tars is much slower than zip.

We should block this high volume access first of all as a stop gap, but either way we should prioritize moving these tars to zips.

Prerequisites

• https://webarchive.jira.com/browse/PBOX-3879

Phase I

The strategy is:

• keep all tars in their archive.org cover items
• produces corresponding zips for each batch
• update the OL cover database to point to the zip files
• test that the covers are indeed serving the zips, not the tars
• remove the tars

Phase II

There is one exceptional case, which is that there is a batch of tars on disk that we'd like to be zips. One solution (while not ideal) is to perform this same process, to:

• upload the tars to an archive.org item but DON'T serve from them
• run Hank's derive scripts to convert those tar to zips
• delete the tars from archive.org
• update the OL db to use these zips
• confirm they're working
• remove these final remaining tars from OL disk.

Proposal & Constraints

What is the proposed solution / implementation?

Is there a precedent of this approach succeeding elsewhere?

Which suggestions or requirements should be considered for how feature needs to appear or be implemented?

Leads

Related files

Stakeholders

---

Instructions for Contributors

• Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

[hbromley]

See jira issue PBOX-3879 for creation of the fixer op that would make the zips.

[cdrini]

Marking as Blocked since waiting for PBOX-3879.

[hbromley]

Jira issue PBOX-3879 is now closed, and the new fixer op is currently being deployed to the PRI servers where it would run.

Let me know if you need any help running it on all your items that contain tars.