Internet Archive import often not setting language

[cdrini]

Problem

Eg:

• https://openlibrary.org/books/OL43438350M/500_G%C3%A2teaux?_compare=Compare&b=4&a=3&m=diff
• https://openlibrary.org/books/OL43193978M/Acres_de_Diamantes?b=3&a=2&_compare=Compare&m=diff

Reproducing the bug

1. Trigger an import of an IA book

• Expected behavior: Language on IA is copied over to Open Library
• Actual behavior: Language is just missing

Context

• Browser (Chrome, Safari, Firefox, etc):
• OS (Windows, Mac, etc):
• Logged in (Y/N):
• Environment (prod, dev, local): prod

Breakdown

Requirements Checklist

• [ ]

Related files

Stakeholders

---

Instructions for Contributors

• Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

[NickDevi]

hello @cdrini I would like to work on this issue. Thank you!

[mekarpeles]

@NickDevi thank you for offering! I think first we'll have @scottbarnes investigate and provide next steps if necessary

[scottbarnes]

@NickDevi, sorry for the inexcusable delay. If you are still interested in this, I think this is what's going on.

1. Languages work for NEW imports via the /api/import/ia endpoint (see Direct Import by OCAID.
2. Languages ARE NOT added to EXISTING items when using that endpoint to supplement an existing record.
3. This is because 'languages' is not included in fields which are added to an existing record if the field is empty/absent in the existing record. See

   openlibrary/openlibrary/catalog/add_book/__init__.py

   Lines 808 to 814 in 1e7fce7

   	edition_list_fields = [
   	'local_id',
   	'lccn',
   	'lc_classifications',
   	'oclc_numbers',
   	'source_records',
   	]

4. HOWEVER, simply adding 'languages' there will cause an import error. Note: we here enter the realm of belief, where I may be mistaken. This seems to be because the language field is in the wrong format in rec at this point. We need it to look like [{"key": "/languages/eng"}], for example, but it seems to look like "languages": ["eng"]. That won't work.
5. The reason for this seems to be related to why this would work as an initial import (when there is no matching record) but not when matched to an existing record. That is I think because build_query() is not called when re-importing, but is called upon an initial import, and I think that's the only place the language field gets formatted. See

   openlibrary/openlibrary/catalog/add_book/load_book.py

   Lines 286 to 291 in 1e7fce7

   	if k in ('languages', 'translated_from'):
   	for language in v:
   	if web.ctx.site.get('/languages/' + language.lower()) is None:
   	raise InvalidLanguage(language.lower())
   	book[k] = [{'key': '/languages/' + language.lower()} for language in v]
   	continue

What this probably means is that:

1. 'languages' needs to be added to add_book/__init__.py as mentioned in (3) above, and;
2. the language formatting code in build_query() mentioned in (4) above needs to be broken out into an a helper function and called from both build_query() and update_edition_with_rec_data() (found in add_book/__init__.py).

I don't think any of this is necessarily hard in and of itself, but imports are a bit of a quagmire.

If you are interested in this, and I'm more than happy to do this if you think this is will involve too much wandering in the wilderness, you'd want to read:

1. https://github.com/internetarchive/openlibrary/wiki/Developer's-Guide-to-Data-Importing#api-authentication-for-imports (for learning how to set a cookie when using imports locally) and
2. https://github.com/internetarchive/openlibrary/wiki/Developer's-Guide-to-Data-Importing#direct-import-by-ocaid (for importing by OCAID/identifier, such as isbn_9781849353946.

And then test this all locally. This would get you started testing your code using curl:

$ curl -c cookies.txt -X POST "http://localhost:8080/account/login" -d "username=openlibrary@example.com&password=admin123"
$ curl -L -v -X POST "http://localhost:8080/api/import/ia" \
     -b ./cookies.txt \
     -d "identifier=isbn_9781849353946&require_marc=false&force_import=true"
...
{"authors": [{"key": "/authors/OL5A", "name": "Leslie Kaplan", "status": "created"}], "success": true, "edition": {"key": "/books/OL11M", "status": "created"}, "work": {"key": "/works/OL1W", "status": "created"}}%

That should import. If you remove the language (possibly using http://localhost:8080/books/OL11M.yml?m=edit, where OL11M is the edition identifier you got when importing), and then attempt to re-import the same work, it should match, but not add English back as a language, which is the problem and the reason this issue exists.

From there, the fun begins editing the Python code.

Again, please let me know if you're still interested. I would say because of the various parts here, this isn't a great first issue.

Edit: fix file names.

[qotkdrn]

Hello, my name is Alex Bae. I was wondering if I could contribute to this issue?

[scottbarnes]

@NickDevi, is this still of interest to you?