Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export incomplete and import throws error #121

Closed
b3nj5m1n opened this issue Mar 24, 2021 · 3 comments · Fixed by #136
Closed

Export incomplete and import throws error #121

b3nj5m1n opened this issue Mar 24, 2021 · 3 comments · Fixed by #136

Comments

@b3nj5m1n
Copy link

I'm trying to export my deck containing card templates, the json is incomplete, there's only 3 of the 11 note types.

When trying to import from this json, I get this error:

Error 
An error occurred. Please start Anki while holding down the shift key, which will temporarily disable the add-ons you have installed. 
If the issue only occurs when add-ons are enabled, please use the Tools>Add-ons menu item to disable some add-ons and restart Anki, repeat until you discover the add-on that is causing the problem. 
When you've discovered the add-on that is causing the problem, please report the issue on the add-ons section of our support site. 
Debug info:
Anki 2.1.35 (84dcaa86) Python 3.8.1 Qt 5.15.1 PyQt 5.15.1
Platform: Linux
Flags: frz=True ao=True sv=1
Add-ons, last update check: 2021-03-24 13:27:47
Add-ons possibly involved: ⁨CrowdAnki JSON exportimport Edit history Collaborate on deck creation⁩

Caught exception:
Traceback (most recent call last):
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/anki/ui/action_vendor.py", line 31, in <lambda>
    lambda: AnkiJsonImporter.import_deck(self.window.col, self.directory_vendor))
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/importer/anki_importer.py", line 112, in import_deck
    AnkiJsonImporter.import_deck_from_path(collection, Path(directory_path))
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/importer/anki_importer.py", line 101, in import_deck_from_path
    if importer.load_deck(directory_path):
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/importer/anki_importer.py", line 39, in load_deck
    deck.save_to_collection(self.collection, import_config=import_config)
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/representation/deck.py", line 131, in save_to_collection
    self.save_decks_and_notes(collection=collection,
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/representation/deck.py", line 154, in save_decks_and_notes
    note.save_to_collection(collection, self, model_map_cache, import_config=import_config)
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/representation/note.py", line 115, in save_to_collection
    self.handle_import_config_changes(import_config, note_model)
  File "/home/b3nj4m1n/.local/share/Anki2/addons21/1788670778/representation/note.py", line 131, in handle_import_config_changes
    if import_config.is_personal_field(note_model.anki_dict['name'], note_model.anki_dict['flds'][num]['name']):
IndexError: list index out of range

To try to reproduce, here is the deck on AnkiWeb.

@aplaice
Copy link
Collaborator

aplaice commented Mar 24, 2021

Thanks very much for reporting and sorry for the bug! I can indeed reproduce the issue. I believe I have a "palliative" fix for the issue (see below, section "Localised fix").


Causes

I think I've found the source of the bug. The ultimate reason is that when Anki clones a note model (as when you go to Tools > Manage note types > Add > Clone: xxx), it also copies the crowdanki_uuid of that note model (if the note model already has a crowdanki_uuid). (This is #118 (for note models — see the edit at the very end).) CrowdAnki should remove or change the crowdanki_uuid of these cloned note models, but currently does not.

The end result is that when you created the many nice note models, by cloning each other (as is the sensible route!), you ended up with many of the note models sharing the same crowdanki_uuid. Hence, when CrowdAnki exported the deck it erroneously believed that there were only four different note models and only exported them. As a result, when it tried to re-import the notes (it had exported all the notes), it crashed since the number of actual fields in some of the notes didn't match the number of fields in the not-actually-matching-but-having-a-"lying-"uuid note models.

Palliative solutions

Visual diagnosis

If you're interested you can see the UUIDs using Anki's debug console, which allows you to execute python code within Anki. If you press Ctrl+Shift+;, enter the following into the console (at the top) and press Ctrl+Enter:

for m in map(lambda model: (model['crowdanki_uuid'], model['name']),
             filter(lambda model: 'crowdanki_uuid' in model,
                    self.col.models.all())):
    print(m)
you'll see something like this
('3d263d1c-b7ae-11ea-b976-40b076dccfad', 'prettyBasic')
('7012404a-fb4c-11ea-b291-40b076dccfad', 'prettyBook')
('3d2653d8-b7ae-11ea-b976-40b076dccfad', 'prettyCloze')
('3d263d1c-b7ae-11ea-b976-40b076dccfad', 'prettyConjugation')
('7012404a-fb4c-11ea-b291-40b076dccfad', 'prettyEvent')
('3d263d1c-b7ae-11ea-b976-40b076dccfad', 'prettyList')
('7012404a-fb4c-11ea-b291-40b076dccfad', 'prettyPerson')
('3d263d1c-b7ae-11ea-b976-40b076dccfad', 'prettyPoem')
('3d268c54-b7ae-11ea-b976-40b076dccfad', 'prettySentence')
('3d268c54-b7ae-11ea-b976-40b076dccfad', 'prettyWord')

i.e. there are only four different UUIDs used by all the note types.

Localised fix

You can also fix (i.e. disambiguate) the UUIDs using the console (again pasting into it), with the following code:

fix_uuids_dict = {
# They're all being changed, though in principle one could keep four the way they were
'prettyBasic': '6cb207c7-22d9-4c24-a7ce-f06652ccb288',
'prettyCloze': '673ddbaf-13f2-46ef-a669-9a3637a52f26',
'prettyPerson': '97555342-72c0-4718-a14e-9ea53d01da84',
'prettyWord': 'd1049052-fc70-4c04-927b-5d65142cf7d8',
'prettyBook': '425f9772-6a2c-4912-a65b-f1e373fec4d0',
'prettyConjugation': '8f62895b-c508-4e8b-95d6-17fa64f7df03',
'prettyEvent': 'a3b68cea-479a-4538-9336-21288105bf43',
'prettyList': '9adc7a52-1c3a-473a-848d-b04509be9cbc',
'prettyPoem': '90a49382-4c6e-4ad9-854d-f2402978230e',
'prettySentence': '78f5753d-5bf5-4a67-978d-b40e649c7263'
}

for model in filter(lambda model: 'crowdanki_uuid' in model,
                    self.col.models.all()):
    if model["name"] in fix_uuids_dict:
        model["crowdanki_uuid"] = fix_uuids_dict[model["name"]]
        self.col.models.save(model)

This should work for you, for this particular deck, but obviously won't work in the general case. It also won't prevent new note models having duplicate UUIDs in the future (this needs to be fixed in CrowdAnki — see below).

General diagnosis

For anybody else coming across this in the future, one can check whether one's Anki collection has note models with duplicated/colliding/repeated UUIDs with the following (again pasted into the console):

repeated = {}
for model in filter(lambda model: 'crowdanki_uuid' in model,
                    self.col.models.all()):
    cu = model["crowdanki_uuid"]
    if cu in repeated:
        repeated[cu] = True
    else:
        repeated[cu] = False

if any(repeated.values()):
    print("WARNING! There are repeated Note Model uuids!")
else:
    print("There are no repeated Note Model uuids! Everything is OK!")

General (non-permanent) fix

The following code will set an alternative (automatically generated) uuid where there's a collision/repetition:

from uuid import uuid1

uuids = []

for model in filter(lambda model: 'crowdanki_uuid' in model,
                    sorted(self.col.models.all(), key=lambda m: m["id"])):
    # we're sorting in the hope that note models with higher ids (which mostly corresponds to creation date) were created later, so we change the uuid of the copies, not the original
    cu = model["crowdanki_uuid"]
    if cu in uuids:
        print("Replacing UUID for note model " + model["name"] + "!")
        model["crowdanki_uuid"] = str(uuid1())
        self.col.models.save(model)
    else:
        uuids.append(cu)

(Since the UUIDs are generated automatically, this might not be ideal in the case of notes shared by multiple people, if several people use this at the same time on their copies of the same deck, but it shouldn't cause any major issue.)

By "non-permanent" I mean that this will need to be re-run every time a new note model is created by duplicating a note model which already has a crowdanki_uuid.

Tackling the root causes

Whenever a note model is cloned, we'd ideally want CrowdAnki to check whether the note model being cloned already has a crowdanki_uuid, and if it does, remove or change the crowdanki_uuid in the copy.

The code for cloning note models is in qt/aqt/models.py and pylib/anki/models.py. Unfortunately, unlike for deck configs, there's no nice "hook" for addons. We could monkey-patch the code or ask for a hook to be created.

An alternative would be to check for uniqueness of note model UUIDs whenever we carry out any operation (import, export, snapshot), and if there are any collisions, change the UUID of the note models with the higher id, in the hope that they're the copies rather than the original. However, this feels "wrong" and seems like it might be inefficient.

(I'll think about how best to deal with this, when I have more time.)


BTW thanks for the nice-looking and detailed note models!

@b3nj5m1n
Copy link
Author

Thanks a lot for the quick and detailed response!

I was able to verify that this was indeed the cause for my problem, and the fix worked fine as well, so thank you very much.

I'll leave this issue open to track progress on the root cause, hopefully you're able to find a clean solution.

@aplaice
Copy link
Collaborator

aplaice commented Mar 24, 2021

I'm glad that the "palliative" fix worked!

@caplett caplett mentioned this issue Jun 8, 2021
aplaice added a commit to aplaice/CrowdAnki that referenced this issue Jul 3, 2021
Fix Stvad#121, Stvad#123.

Unfortunately, this does not prevent a user from duplicating a note
model's crowdanki_uuid when cloning a note model on AnkiDroid or
AnkiMobile.  Hopefully most users create new note models on desktop!
aplaice added a commit to aplaice/CrowdAnki that referenced this issue Aug 22, 2021
Fix Stvad#121, Stvad#123.

Unfortunately, this does not prevent a user from duplicating a note
model's crowdanki_uuid when cloning a note model on AnkiDroid or
AnkiMobile.  Hopefully most users create new note models on desktop!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants