Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move AMZ retry code from Edition.from_isbn() to affiliate-server #8541

Closed
scottbarnes opened this issue Nov 18, 2023 · 0 comments · Fixed by #8690
Closed

Move AMZ retry code from Edition.from_isbn() to affiliate-server #8541

scottbarnes opened this issue Nov 18, 2023 · 0 comments · Fixed by #8690
Assignees
@scottbarnes
Labels
Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] Module: Import Issues related to the configuration or use of importbot and other bulk import systems. [managed] Priority: 2 Important, as time permits. [managed] Type: Feature Request Issue describes a feature or enhancement we'd like to implement. [managed]

Comments

@scottbarnes
Copy link
Collaborator

scottbarnes commented Nov 18, 2023 •
edited
Loading

When importing via /isbn, Edition.from_isbn() is used, and that code:

  1. checks Open Library to see if the ISBN is already in the database;
  2. if there's no result from the Open Library database, the import_item table is checked for staged items; then
  3. finally, if no result is found in import_item, an import is attempted from Amazon (note: this code path is currently disabled).

Step 3 above should be moved to the affiliate-server, which should be modified to take a new GET param for /submit (say, ?import=1), then:

  1. loop/wait until the ISBN in in cache, waiting 1 sec, up to 5 retries -- look at core/model.py Edition.from_isbn for an example of this code;
  2. If it gets data back from AMZ, queue the record in import_item as staged and then we immediately call do_import (or call load(), or make it an ImportItem object and call a class method to import) on that record and wait for the result;
  3. Make sure we're not bulk importing amazon records. e.g. Update

    openlibrary/scripts/affiliate_server.py

    Line 210 in 7bf8200

    [{'ia_id': b['source_records'][0], 'data': b} for b in pending_books]
    to status="staged" if we re-enable AMZ.

@mekarpeles
@scottbarnes scottbarnes added Type: Feature Request Issue describes a feature or enhancement we'd like to implement. [managed] Module: Import Issues related to the configuration or use of importbot and other bulk import systems. [managed] Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] labels Nov 18, 2023
@scottbarnes scottbarnes self-assigned this Nov 18, 2023
@scottbarnes scottbarnes added Needs: Triage This issue needs triage. The team needs to decide who should own it, what to do, by when. [managed] Needs: Lead and removed Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] labels Nov 18, 2023
@jimchamp jimchamp added Priority: 2 Important, as time permits. [managed] Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] Lead: @scottbarnes Issues overseen by Scott (Community Imports) and removed Needs: Triage This issue needs triage. The team needs to decide who should own it, what to do, by when. [managed] Needs: Lead labels Nov 20, 2023
@scottbarnes scottbarnes removed the Lead: @scottbarnes Issues overseen by Scott (Community Imports) label Nov 20, 2023
scottbarnes added a commit to scottbarnes/openlibrary that referenced this issue Jan 5, 2024
@scottbarnes
Stop bulk imports from AMZ records
6c789ef 
Import AMZ records as `staged`.
See internetarchive#8541
@scottbarnes scottbarnes mentioned this issue Jan 5, 2024
Merged
scottbarnes added a commit to scottbarnes/openlibrary that referenced this issue Jan 5, 2024
@scottbarnes
Stop bulk imports from AMZ records
3e0f4db 
Import AMZ records as `staged`.
See internetarchive#8541
scottbarnes added a commit to scottbarnes/openlibrary that referenced this issue Jan 5, 2024
@scottbarnes
Stop bulk imports from AMZ records
af040fd 
Import AMZ records as `staged`.
See internetarchive#8541
scottbarnes added a commit to scottbarnes/openlibrary that referenced this issue Jan 5, 2024
@scottbarnes
Stop bulk imports from AMZ records
f789ed7 
Import AMZ records as `staged`.
See internetarchive#8541
scottbarnes added a commit to scottbarnes/openlibrary that referenced this issue Jan 22, 2024
@scottbarnes
Stop bulk imports from AMZ records
36971ea 
Import AMZ records as `staged`.
See internetarchive#8541
scottbarnes added a commit that referenced this issue Feb 13, 2024
@scottbarnes @mekarpeles
Re-enable Amazon imports from /isbn (#8690)
2f59017 
* Renable `/isbn` and AMZ imports

This PR reenables AMZ imports from `/isbn` and `/api/books.json`.

See this comment examples of how to use the endpoints and what
to expect:
#8690 (comment)

The logic upon visiting `/isbn/[some isbn]` is now:
- attempt to fetch the book from the OL database;
- attempt to fetch the book from the `import_item` table (likely ISBNdb);
- attempt to fetch the metadata from the Amazon Products API, clean that
  metadata for import, add the as a `staged` import in the `import_item`
  table, and then immediately import it `load()`, by way of
  `ImportItem.import_first_staged()`.

If any of these find or create an edition, the ention is returned.

* Stop bulk imports from AMZ records

Import AMZ records as `staged`.
See #8541

* Modify the affiliate server to accept a GET parameter, `high_priority`, at
`/isbn`. E.g., `http://localhost:31337/isbn/059035342X?high_priority=true`.

`high_priority=true` will put the ISBN straight to the front of the queue
for an AMZ Products API look up, and attempt for three seconds to fetch
the cached AMZ data (if it becomes available), returning that data,
marshalled into a form suitable for creating an Edition, if possible.

* Use `high_priority=false` (the default) on the affiliate server to fetch AM
data if available,then queue for import and immediately import the item,
returning the resulting `Edition`, or `None`.

* Feature: `/api/books` will attempt to import from ISBN
- adds an `high_priority` parameter to `/api/books` to make the API try
  to import books from ISBN.
- relies on changes to `Edition.from_isbn()` which attempt to import
  editions first from `staged` items in the `import_item` table, and
  second from Amazon via the affiliate server.

---------

Co-authored-by: Mek <michael.karpeles@gmail.com>
Achorn pushed a commit to Achorn/openlibrary that referenced this issue Mar 14, 2024
@scottbarnes @mekarpeles @Achorn
Re-enable Amazon imports from /isbn (internetarchive#8690)
1d0aa3d 
* Renable `/isbn` and AMZ imports

This PR reenables AMZ imports from `/isbn` and `/api/books.json`.

See this comment examples of how to use the endpoints and what
to expect:
internetarchive#8690 (comment)

The logic upon visiting `/isbn/[some isbn]` is now:
- attempt to fetch the book from the OL database;
- attempt to fetch the book from the `import_item` table (likely ISBNdb);
- attempt to fetch the metadata from the Amazon Products API, clean that
  metadata for import, add the as a `staged` import in the `import_item`
  table, and then immediately import it `load()`, by way of
  `ImportItem.import_first_staged()`.

If any of these find or create an edition, the ention is returned.

* Stop bulk imports from AMZ records

Import AMZ records as `staged`.
See internetarchive#8541

* Modify the affiliate server to accept a GET parameter, `high_priority`, at
`/isbn`. E.g., `http://localhost:31337/isbn/059035342X?high_priority=true`.

`high_priority=true` will put the ISBN straight to the front of the queue
for an AMZ Products API look up, and attempt for three seconds to fetch
the cached AMZ data (if it becomes available), returning that data,
marshalled into a form suitable for creating an Edition, if possible.

* Use `high_priority=false` (the default) on the affiliate server to fetch AM
data if available,then queue for import and immediately import the item,
returning the resulting `Edition`, or `None`.

* Feature: `/api/books` will attempt to import from ISBN
- adds an `high_priority` parameter to `/api/books` to make the API try
  to import books from ISBN.
- relies on changes to `Edition.from_isbn()` which attempt to import
  editions first from `staged` items in the `import_item` table, and
  second from Amazon via the affiliate server.

---------

Co-authored-by: Mek <michael.karpeles@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@scottbarnes scottbarnes

Labels
Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] Module: Import Issues related to the configuration or use of importbot and other bulk import systems. [managed] Priority: 2 Important, as time permits. [managed] Type: Feature Request Issue describes a feature or enhancement we'd like to implement. [managed]
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Re-enable Amazon imports from /isbn scottbarnes/openlibrary
2 participants
@scottbarnes @jimchamp