Import ecoinvent 3.6 cutoff fails with InvalidExchange exception

[levonamatuni]

Have recently imported (online) the 'cut-off36' Ecoinvent database into my recent project. It shows 0 records only and does not react on 'Delete database' function. Possibly I had to import the 'cutoff36' version (without the dash).

In general, importing a new Ecoinvent database proposes to download either cut-off Or cutoff version, which is confusing.

Thanks!

Levon

[dgdekoning]

This is partially similar to #289 which also has to do with database importing. The two different versions issue is resolved in the newest development builds (pull-request #290), cut-off is the correct one for 3.6, cutoff for previous versions.

Can you install activity-browser-dev instead?

Are there any error messages in the prompt where you opened the activity browser?

[dgdekoning]

@levonamatuni, did the newest development version solve your issue?

[levonamatuni]

@dgdekoning The newest version shows properly the names of the databases to be imported and allows to remove the empty database; however, 3.6 version is still impossible to import because of the discrepancy with some exchanges during its extraction.

[dgdekoning]

But only sometimes?

[dgdekoning]

So, a quick test on windows 10, python 3.7 shows the import succeeding in a 'clean' project, which only contains the required biosphere3 database.

It also succeeds importing in a new project with a previous version (3.5) of ecoinvent.

But at the same time I have seen it fail on my device under the same conditions before, and I have seen it fail on someone else's device with the same exception thrown. I am very close to calling shenanigans.

Possible things to consider:

• Decompressing into a temporary directory causes problems (a failed import leaves traces and causes all following imports to fail?) (possibly related to [Linux] PyQt errors when loading database #289)
• Interference from some existing settings / data in older projects.

Edit:
Tried to import in the 'default' project and was able to get the exact same exception again.

Partial log with exception:

# More extraction lines here.
Extracting  MasterData\Units.xml

Everything is Ok

Folders: 2
Files: 18141
Size:       2415495790
Compressed: 77643614
Extracted 18121 datasets in 293.16 seconds
Applying strategy: normalize_units
Applying strategy: update_ecoinvent_locations
Applying strategy: remove_zero_amount_coproducts
Applying strategy: remove_zero_amount_inputs_with_no_activity
Applying strategy: remove_unnamed_parameters
Applying strategy: es2_assign_only_product_with_amount_as_reference_product
Applying strategy: assign_single_product_as_activity
Applying strategy: create_composite_code
Applying strategy: drop_unspecified_subcategories
Applying strategy: fix_ecoinvent_flows_pre35
Applying strategy: drop_temporary_outdated_biosphere_flows
Applying strategy: link_biosphere_by_flow_uuid
Applying strategy: link_internal_technosphere_by_composite_code
Applying strategy: delete_exchanges_missing_activity
Applying strategy: delete_ghost_exchanges
Applying strategy: remove_uncertainty_from_negative_loss_exchanges
Applying strategy: fix_unreasonably_high_lognormal_uncertainties
Applying strategy: set_lognormal_loc_value
Applying strategy: convert_activity_parameters_to_list
Applying strategy: add_cpc_classification_from_single_reference_product
Applied 20 strategies in 7.08 seconds
Writing activities to SQLite3 database:
Traceback (most recent call last):
  File "C:\Users\koningdgde\workspace\activity-browser\activity_browser\app\ui\wizards\db_import_wizard.py", line 457, in run
    self.run_ecoinvent()
  File "C:\Users\koningdgde\workspace\activity-browser\activity_browser\app\ui\wizards\db_import_wizard.py", line 474, in run_ecoinvent
    self.run_import()
  File "C:\Users\koningdgde\workspace\activity-browser\activity_browser\app\ui\wizards\db_import_wizard.py", line 522, in run_import
    importer.write_database(backend='activitybrowser')
  File "C:\Users\koningdgde\AppData\Local\Continuum\miniconda3\envs\ab_dev\lib\site-packages\bw2io\importers\base_lci.py", line 240, in write_database
    db.write(existing)
  File "C:\Users\koningdgde\AppData\Local\Continuum\miniconda3\envs\ab_dev\lib\site-packages\bw2data\project.py", line 356, in writable_project
    return wrapped(*args, **kwargs)
  File "C:\Users\koningdgde\AppData\Local\Continuum\miniconda3\envs\ab_dev\lib\site-packages\bw2data\backends\peewee\database.py", line 260, in write
    self._efficient_write_many_data(data)
  File "C:\Users\koningdgde\workspace\activity-browser\activity_browser\app\ui\wizards\db_import_wizard.py", line 710, in _efficient_write_many_data
    super()._efficient_write_many_data(*args, **kwargs)
  File "C:\Users\koningdgde\AppData\Local\Continuum\miniconda3\envs\ab_dev\lib\site-packages\bw2data\backends\peewee\database.py", line 205, in _efficient_write_many_data
    index, key, ds, exchanges, activities
  File "C:\Users\koningdgde\workspace\activity-browser\activity_browser\app\ui\wizards\db_import_wizard.py", line 718, in _efficient_write_dataset
    return super()._efficient_write_dataset(*args, **kwargs)
  File "C:\Users\koningdgde\AppData\Local\Continuum\miniconda3\envs\ab_dev\lib\site-packages\bw2data\backends\peewee\database.py", line 156, in _efficient_write_dataset
    raise InvalidExchange
bw2data.errors.InvalidExchange

[dgdekoning]

Found a possible reason:

There are new biosphere3 flows. So the solution to the importing issue would be to either update the existing biosphere3 database in some way or to straight up remove all databases (create a new project) and reimport the biosphere3 flows.

[dgdekoning]

Alright, so I just noticed that #213 has the same underlying reasons as this issue.

My proposed solution to this is two-parted:

• Create a warning / exception that is triggered specifically when importing EcoInvent and receiving an InvalidExchange exception, basically notifying the user that their biosphere3 database is out of date.
• Create a new main-menu option which uses the bw2io functionality that adds new biosphere flows to the biosphere3 database to attempt and update the existing biosphere3 database to the latest version.