upf to json converter #3308

simonpintarelli · 2019-09-11T12:22:18Z

_prepare_json to upf class
added export, e.g. verdi data upf export PK to cli interface

yakutovicha · 2019-09-11T12:41:16Z

@giovannipizzi, @sphuber this export functionality is needed to enable SIRIUS support within the aiida-cp2k plugin. I think the same could be used to support SIRIUS under QE.

yakutovicha

Thanks, @simonpintarelli ,

Before I start the review please add a couple of tests (upf1->json, upf2->json) and fix the problems raised by the pre-commit run. Just run pre-commit run --all-files to see all of them

As soon as it is done, I think there shouldn't be a problem to merge it.

CasperWA · 2019-09-13T10:57:47Z

Before I start the review please add a couple of tests (upf1->json, upf2->json) and fix the problems raised by the pre-commit run. Just run pre-commit run --all-files to see all of them

Hi @simonpintarelli, I am unsure how familiar you are with pre-commit, so just to be more explicit (and if you already know it, then hopefully no harm done 😄 ):
To have pre-commit, AiiDA may be installed specifying 'dev_precommit', e.g.

pip install -U -e .[dev_precommit]

when in the folder of the git cloned 'aiida-core' folder, or

pip install -U aiida[dev_precommit]

if installing from PyPI and using AiiDA v0.12 (should not be the case here).

After this, still being in the git cloned 'aiida-core' folder, you can run

pre-commit install

and then, finally

pre-commit run --all-files

as @yakutovicha writes 😃

simonpintarelli · 2019-09-13T11:36:33Z

@CasperWA, I did pip install pre-commit, but apparently forgot to run pre-commit install.

giovannipizzi

For now we could include this into AiiDA core, even if eventually we want to move materials-science related classes in a different repository/plugin.

One question: do you have in mind of preparing a minimal library to manage UPF and then you could keep the code there and just depend on it, or this is the whole code for UPF that you have so it's easier to add it to aiida_core verbatim, as you are doing in this PR?

giovannipizzi · 2019-10-04T08:46:40Z

aiida/backends/tests/orm/data/test_upf.py

+        json_string, _ = self.pseudo_oxygen_upf1._prepare_json()
+        filepath_base = os.path.abspath(os.path.join(__file__, os.pardir, os.pardir, os.pardir, 'fixtures', 'pseudos'))
+        reference_json = json.dumps(json.load(open(os.path.join(filepath_base, 'O_raw.json'), 'r')))
+        self.assertEqual(json_string, reference_json.encode('utf-8'))


Could you compare the dictionaries rather than the strings?

giovannipizzi · 2019-10-04T08:48:01Z

aiida/orm/nodes/data/upf_to_json/__init__.py

@@ -0,0 +1,3 @@
+""" UPF converter """
+
+from .upf_to_json import upf_to_json


The right place for these tools (we're slowly moving also other things there) is not in aiida/orm/data but in aiida/tools/data/upf/.... Could you move the logic there?

@giovannipizzi Yes, this is the entire code we have for UPF to json conversion. Although moving it to a separate dependency might be a good idea.

Moving the files to aiida/tools/data/upf seems to trigger a circular import if I then import the function in orm/nodes/data/upf.py.

simonpintarelli · 2019-10-07T12:11:44Z

I've moved the code to aiida/tools/data/upf/...

It passes through prospector on my computer and and export upf to json via verdi works as desired. For some reason now travis keeps failing on both python3 & 2:

Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/bin/verdi", line 10, in <module>
    sys.exit(verdi())
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/aiida/cmdline/commands/cmd_setup.py", line 47, in setup
    from aiida import orm
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/aiida/orm/__init__.py", line 22, in <module>
    from .nodes import *
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/aiida/orm/nodes/__init__.py", line 16, in <module>
    from .data import *
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/aiida/orm/nodes/data/__init__.py", line 32, in <module>
    from .upf import UpfData
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/aiida/orm/nodes/data/upf.py", line 23, in <module>
    from aiida.tools.data.upf.to_json import upf_to_json
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/aiida/tools/__init__.py", line 28, in <module>
    from .data.array.kpoints import *
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/aiida/tools/data/array/kpoints/__init__.py", line 18, in <module>
    from aiida.orm import KpointsData, Dict
ImportError: cannot import name 'KpointsData'
The command ".ci/setup_profiles.sh" failed and exited with 1 during .

giovannipizzi · 2019-10-11T11:28:57Z

I have the feeling this might be a cyclic import error... can you investigate if you understand if this is the case, having this hint?

simonpintarelli · 2019-10-14T11:47:21Z

@giovannipizzi There was a circular dependency, I've moved the json converter now to a separate package.
The unit tests for checking the generated json seem to pass now, there were roundoff errors, so the dictionaries need to be compared manually.

There is one failure remaining. Where do I have to add a generator of a dummy instance?

======================================================================
ERROR: test_data_exporters (aiida.backends.tests.orm.data.test_data.TestData)
Verify that the return value of the export methods of all `Data` sub classes have the correct type.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/aiidateam/aiida-core/aiida/backends/tests/orm/data/test_data.py", line 75, in test_data_exporters
    instance = self.generate_class_instance(data_class)
  File "/home/travis/build/aiidateam/aiida-core/aiida/backends/tests/orm/data/test_data.py", line 57, in generate_class_instance
    'for this data class, add a generator of a dummy instance here'.format(data_class)
RuntimeError: no instance generator implemented for class `<class 'aiida.orm.nodes.data.upf.UpfData'>`. If you have added a `_prepare_*` method for this data class, add a generator of a dummy instance here

sphuber · 2019-10-14T12:05:29Z

You need to add a clause to aiida.backends.tests.orm.data.test_data.TestData.generate_class_instance class method. Add a clause if data_class is orm.UpfData where you return a UpfData instance. You can use one of the UPF files you use for testing the conversion between UPF and json formats as a dummy file.

simonpintarelli · 2019-10-14T13:28:57Z

@sphuber Thanks! I've rebased my PR to the latest develop branch, before there was no aiida.backends.tests.orm.data.test_data.TestData.

simonpintarelli · 2019-10-14T14:50:18Z

Travis is failing the last test, because conda cannot find upf_to_json (https://pypi.org/project/upf-to-json/).

Is it enough to publish the package on conda-forge?

sphuber · 2019-10-14T15:03:50Z

Yes, as long as it is on conda-forge, it should work. Note that even after pushing it to conda-forge it may take some time for it to become discoverable. But usually one hour after uploading should be sufficient.

simonpintarelli · 2019-10-20T12:09:36Z

@sphuber, thank you. PR is pending on conda-forge: conda-forge/staged-recipes#9862

sphuber

@simonpintarelli thanks a lot, this is looking great now. Except for a minor typo, this is looking good to merge. Could you please address the final change and rebase?

sphuber · 2019-10-29T07:24:22Z

aiida/cmdline/commands/cmd_data/cmd_upf.py

+@export_options
+@decorators.with_dbenv()
+def upf_export(**kwargs):
+    """Export StructureData object to file."""


Suggested change

"""Export StructureData object to file."""

"""Export `UpfData` object to file."""

Thanks. I've fixed it.

UPF format 1 and 2 are supported. SIRIUS requires pseudopotentials in json format.

Export a upf to json using: verdi data export upf

Requested hanges addressed

sphuber · 2019-10-29T08:51:05Z

Thanks a lot for the contribution @simonpintarelli

yakutovicha requested review from sphuber and giovannipizzi September 11, 2019 12:36

yakutovicha self-requested a review September 11, 2019 12:48

yakutovicha previously requested changes Sep 11, 2019

View reviewed changes

simonpintarelli force-pushed the upf-to-json branch from 38f584b to 44f7bd3 Compare September 11, 2019 13:45

simonpintarelli force-pushed the upf-to-json branch from 34fd304 to 5163179 Compare September 13, 2019 11:30

simonpintarelli force-pushed the upf-to-json branch 5 times, most recently from bbf31cf to b34ae22 Compare September 16, 2019 14:12

giovannipizzi reviewed Oct 4, 2019

View reviewed changes

simonpintarelli force-pushed the upf-to-json branch 2 times, most recently from 81fbc7a to 6be3a57 Compare October 6, 2019 21:15

simonpintarelli force-pushed the upf-to-json branch 3 times, most recently from 78d6cfe to ac923d1 Compare October 9, 2019 15:07

simonpintarelli force-pushed the upf-to-json branch 5 times, most recently from 5d67b4a to bc12816 Compare October 14, 2019 11:23

simonpintarelli force-pushed the upf-to-json branch from bc12816 to 5dedb5f Compare October 14, 2019 13:29

simonpintarelli force-pushed the upf-to-json branch from 5dedb5f to 1ea1ca0 Compare October 20, 2019 12:09

simonpintarelli force-pushed the upf-to-json branch 2 times, most recently from 767c84b to 2ffba11 Compare October 26, 2019 07:17

simonpintarelli force-pushed the upf-to-json branch from 2ffba11 to 3127495 Compare October 28, 2019 15:23

sphuber requested changes Oct 29, 2019

View reviewed changes

simonpintarelli added 5 commits October 29, 2019 09:21

add upf to json converter

23ea93e

UPF format 1 and 2 are supported. SIRIUS requires pseudopotentials in json format.

upf2json: add unit tests

0b4d77b

add upf export to cli

194eb0c

Export a upf to json using: verdi data export upf

add UPF for carbon to TestVerdiDataUpf

212cfdf

test_data_exporters: add a dummy instance for UpfData

5d325d3

simonpintarelli force-pushed the upf-to-json branch from 3127495 to 5d325d3 Compare October 29, 2019 08:21

sphuber approved these changes Oct 29, 2019

View reviewed changes

sphuber merged commit 2e3a9a7 into aiidateam:develop Oct 29, 2019

simonpintarelli deleted the upf-to-json branch October 29, 2019 09:01

unkcpz mentioned this pull request Nov 29, 2020

WIP: parse more info from upf header #4597

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upf to json converter #3308

upf to json converter #3308

simonpintarelli commented Sep 11, 2019

yakutovicha commented Sep 11, 2019

yakutovicha left a comment

CasperWA commented Sep 13, 2019

simonpintarelli commented Sep 13, 2019

giovannipizzi left a comment

giovannipizzi Oct 4, 2019

giovannipizzi Oct 4, 2019

simonpintarelli Oct 6, 2019

simonpintarelli Oct 7, 2019

simonpintarelli commented Oct 7, 2019

giovannipizzi commented Oct 11, 2019

simonpintarelli commented Oct 14, 2019

sphuber commented Oct 14, 2019

simonpintarelli commented Oct 14, 2019

simonpintarelli commented Oct 14, 2019

sphuber commented Oct 14, 2019

simonpintarelli commented Oct 20, 2019

sphuber left a comment

sphuber Oct 29, 2019

simonpintarelli Oct 29, 2019

sphuber commented Oct 29, 2019

		@@ -0,0 +1,3 @@
		""" UPF converter """

		from .upf_to_json import upf_to_json

	"""Export StructureData object to file."""
	"""Export `UpfData` object to file."""

upf to json converter #3308

upf to json converter #3308

Conversation

simonpintarelli commented Sep 11, 2019

yakutovicha commented Sep 11, 2019

yakutovicha left a comment

Choose a reason for hiding this comment

CasperWA commented Sep 13, 2019

simonpintarelli commented Sep 13, 2019

giovannipizzi left a comment

Choose a reason for hiding this comment

giovannipizzi Oct 4, 2019

Choose a reason for hiding this comment

giovannipizzi Oct 4, 2019

Choose a reason for hiding this comment

simonpintarelli Oct 6, 2019

Choose a reason for hiding this comment

simonpintarelli Oct 7, 2019

Choose a reason for hiding this comment

simonpintarelli commented Oct 7, 2019

giovannipizzi commented Oct 11, 2019

simonpintarelli commented Oct 14, 2019

sphuber commented Oct 14, 2019

simonpintarelli commented Oct 14, 2019

simonpintarelli commented Oct 14, 2019

sphuber commented Oct 14, 2019

simonpintarelli commented Oct 20, 2019

sphuber left a comment

Choose a reason for hiding this comment

sphuber Oct 29, 2019

Choose a reason for hiding this comment

simonpintarelli Oct 29, 2019

Choose a reason for hiding this comment

sphuber commented Oct 29, 2019