✨ NEW: _repr_pretty_ representation for `ProcessBuilder` #4970

mbercx · 2021-06-02T19:36:29Z

The representation of the ProcessBuilder (i.e. __repr__) is
difficult to read since it just prints the heavily nested mapping on one
line and doesn't show the contents of e.g. Dict nodes.

Here we add a _repr_pretty_ method to improve the readability
when using the ProcessBuilder in Jupyter notebooks and IPython.

mbercx · 2021-06-02T19:45:00Z

This is one I've been meaning to do for a while now, but I never knew about _repr_html_. 😅

@sphuber this is adapted from a method I stole from the aiida-quantumespresso-epfl repo. 😁
@louisponet I couldn't add you as a reviewer, but have a look and let me know what you think.

Some usage examples:

After using the builder for the PwCalculation for a bit:

builder = code.get_builder()
builder.structure = structure
builder.parameters = orm.Dict(dict={
  'CONTROL': {
    'calculation': 'scf',  # self-consistent field
  },
  'SYSTEM': {
    'ecutwfc': 30.,  # wave function cutoff in Ry
    'ecutrho': 240.,  # density cutoff in Ry
  }
})
builder

Gives

Process class: PwCalculation
Inputs:
"code": Remote code 'qe-v6.7-pw' on localhost, pk: 1, uuid: 47a1c120-47af-4b9e-80c2-5a647c99817e
"metadata": {
    "options": {
        "stash": {}
    },
},
"parameters": {
    "CONTROL": {
        "calculation": "scf"
    },
    "SYSTEM": {
        "ecutwfc": 30.0,
        "ecutrho": 240.0
    }
},
"pseudos": {}
"structure": Si2 (pk: 3)

PwBaseWorkChain obtained from get_builder_from_protocol() method:

code = orm.load_code(1)
structure = orm.load_node(3)

builder = PwBaseWorkChain.get_builder_from_protocol(
    code=code,
    structure=structure,
)
builder

Gives:

Process class: PwBaseWorkChain
Inputs:
"clean_workdir": True
"kpoints_distance": 0.15
"kpoints_force_parity": False
"metadata": {}
"pw": {
    "code": Remote code 'qe-v6.7-pw' on localhost, pk: 1, uuid: 47a1c120-47af-4b9e-80c2-5a647c99817e
    "metadata": {
        "options": {
            "max_wallclock_seconds": 43200
            "resources": {
                "num_machines": 1
            },
            "withmpi": True
        },
    },
    "parameters": {
        "CONTROL": {
            "calculation": "scf",
            "forc_conv_thr": 0.0001,
            "tprnfor": true,
            "tstress": true,
            "etot_conv_thr": 2e-05
        },
        "SYSTEM": {
            "nosym": false,
            "occupations": "smearing",
            "smearing": "cold",
            "degauss": 0.01,
            "ecutwfc": 30.0,
            "ecutrho": 240.0
        },
        "ELECTRONS": {
            "electron_maxstep": 80,
            "mixing_beta": 0.4,
            "conv_thr": 4e-10
        }
    },
    "pseudos": {
        "Si": uuid: 07a2d8db-ef3f-43ce-b200-bd68a7b4c6d2 (pk: 933)
    },
    "structure": Si2 (pk: 3)
},

mbercx · 2021-06-02T19:48:23Z

@chrisjsewell should I add something to ignore these mypy warnings in this file?

aiida/engine/processes/builder.py:97: error: Item "Port" of "Union[Port, PortNamespace]" has no attribute "serialize"  [union-attr]
aiida/engine/processes/builder.py:97: error: Item "PortNamespace" of "Union[Port, PortNamespace]" has no attribute "serialize"  [union-attr]

mbercx · 2021-06-02T19:52:59Z

Just for reference, this is what you would get for the PwBaseWorkChain.get_builder_from_protocol() method one without this representation:

{'metadata': {}, 'pw': {'metadata': {'options': {'resources': {'num_machines': 1}, 'max_wallclock_seconds': 43200, 'withmpi': True}}, 'pseudos': {'Si': <UpfData: uuid: 07a2d8db-ef3f-43ce-b200-bd68a7b4c6d2 (pk: 933)>}, 'code': <Code: Remote code 'qe-v6.7-pw' on localhost, pk: 1, uuid: 47a1c120-47af-4b9e-80c2-5a647c99817e>, 'structure': <StructureData: uuid: e5367a26-ebcd-479d-83db-73cf56ba1539 (pk: 3)>, 'parameters': <Dict: uuid: e6b64e56-2fec-4355-9b16-725914de9d58 (unstored)>}, 'clean_workdir': <Bool: uuid: 57f303c9-8a3e-4f0e-844f-910d4a437c08 (unstored) value: True>, 'kpoints_distance': <Float: uuid: 6055b4e0-5142-4163-9861-63971be7a0a6 (unstored) value: 0.15>, 'kpoints_force_parity': <Bool: uuid: d00c33b3-b27c-41de-b4c7-8fc47352e889 (unstored) value: False>}

(quoted here to wrap the super long line)

louisponet · 2021-06-02T19:54:01Z

So that's what it would look like in ipython and jupyter notebooks? I love it!

codecov · 2021-06-02T19:54:09Z

Codecov Report

Merging #4970 (cd91b0e) into develop (55a7bfe) will increase coverage by 4.93%.
The diff coverage is 91.31%.

@@             Coverage Diff             @@
##           develop    #4970      +/-   ##
===========================================
+ Coverage    77.21%   82.13%   +4.93%     
===========================================
  Files          533      533              
  Lines        38470    38485      +15     
===========================================
+ Hits         29700    31607    +1907     
+ Misses        8770     6878    -1892

Flag	Coverage Δ
django	`77.19% <91.31%> (-0.01%)`	⬇️
sqlalchemy	`76.52% <91.31%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiida/engine/processes/builder.py	`92.25% <91.31%> (-0.73%)`	⬇️
aiida/backends/general/migrations/utils.py	`85.28% <0.00%> (+0.51%)`	⬆️
aiida/engine/daemon/client.py	`76.39% <0.00%> (+1.01%)`	⬆️
aiida/backends/testbase.py	`93.90% <0.00%> (+1.53%)`	⬆️
aiida/manage/manager.py	`80.77% <0.00%> (+1.65%)`	⬆️
aiida/manage/tests/main.py	`88.94% <0.00%> (+2.22%)`	⬆️
aiida/backends/manager.py	`72.23% <0.00%> (+2.78%)`	⬆️
...orm/implementation/sqlalchemy/querybuilder/main.py	`85.08% <0.00%> (+4.68%)`	⬆️
aiida/backends/sqlalchemy/models/log.py	`96.30% <0.00%> (+7.41%)`	⬆️
... and 78 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 55a7bfe...cd91b0e. Read the comment docs.

mbercx · 2021-06-02T19:57:01Z

So that's what it would look like in ipython and jupyter notebooks? I love it!

Unfortunately, not in IPython. 😞 The console doesn't support HTML, notebooks do. See here.

louisponet · 2021-06-02T20:00:09Z

_repr_pretty_ I guess is the way to go for that then, probably not very hard to get a similar repr there.

mbercx · 2021-06-02T20:27:22Z

_repr_pretty_ I guess is the way to go for that then, probably not very hard to get a similar repr there.

👀 Indeed! Let me get right on that. 😁

mbercx · 2021-06-02T20:38:45Z

@louisponet there you go! I dumped the _repr_html_ version. It wasn't really adding anything so far. Now it works in both Jupyter notebooks as well as the IPython console.

mbercx · 2021-06-02T21:40:30Z

Note: the format_inputs method should probably be moved elsewhere. I think we can also use it for e.g. formatting overrides inputs, or the inputs of a process that has already run.

chrisjsewell · 2021-07-21T09:38:40Z

changing this to a draft while the tests are failing @mbercx. Get the tests passed then we can finalise this I guess 😄

mbercx · 2021-08-11T08:42:23Z

Hmm, so I can't seem to fix the pre-commit because if I actually try to ignore these issues:

aiida/engine/processes/builder.py:96: error: Incompatible types in assignment (expression has type "None", variable has type "Union[Port, PortNamespace]")  [assignment]
aiida/engine/processes/builder.py:98: error: Item "Port" of "Union[Port, PortNamespace]" has no attribute "serialize"  [union-attr]
aiida/engine/processes/builder.py:98: error: Item "PortNamespace" of "Union[Port, PortNamespace]" has no attribute "serialize"  [union-attr]
Found 3 errors in 1 file (checked 74 source files)

By adding # type: ignore[assignment] and # type: ignore[union-attr], mypy complains that these are not used:

aiida/engine/processes/builder.py:96: error: unused 'type: ignore' comment
aiida/engine/processes/builder.py:98: error: unused 'type: ignore' comment
Found 2 errors in 1 file (checked 1 source file)

Don't know how to fix this due to my mypy ignorance. Will look into it, but @chrisjsewell: you prob can tell me how to fix this off the top of your head? :)

sphuber · 2021-08-11T13:59:54Z

y adding # type: ignore[assignment] and # type: ignore[union-attr], mypy complains that these are not used:

Might be related to this issue: #5026 I have also suffered from this a few times and have no idea where it comes from nor how to fix it.

chrisjsewell · 2021-08-11T17:17:59Z

Will look into it, but @chrisjsewell: you prob can tell me how to fix this off the top of your head? :)

How do I never seem to have these issues 😆
I would suggest you make sure your version of mypy is up-to-date, in fact I would suggest you use tox -r -e py38-pre-commit -- --all mypy

mbercx · 2021-08-12T07:07:40Z

How do I never seem to have these issues 😆
I would suggest you make sure your version of mypy is up-to-date, in fact I would suggest you use tox -r -e py38-pre-commit -- --all mypy

sigh, alright! I'll finally start looking into using tox. 😝

mbercx · 2022-01-26T17:30:20Z

(oops, slippery fingers! 😅 )

mbercx · 2022-01-26T17:52:02Z

Actually, maybe it makes a lot more sense to output this in a YAML format... It's meant to be human readable after all. For the PwBandsWorkChain example:

See pretty representation of builder

Process class: PwBandsWorkChain
Inputs:
bands:
  metadata: {}
  pw:
    code: Plane wave Quantum ESPRESSO v6.6 - local compilation
    metadata:
      options:
        max_wallclock_seconds: 43200
        resources:
          num_machines: 1
        stash: {}
        withmpi: true
    parameters:
      CONTROL:
        calculation: scf
        etot_conv_thr: 2.0e-05
        forc_conv_thr: 0.0001
        tprnfor: true
        tstress: true
      ELECTRONS:
        conv_thr: 4.0e-10
        electron_maxstep: 80
        mixing_beta: 0.4
      SYSTEM:
        degauss: 0.01
        ecutrho: 240.0
        ecutwfc: 30.0
        nosym: false
        occupations: smearing
        smearing: cold
    pseudos:
      Si: ''
bands_kpoints_distance: 0.025
clean_workdir: true
metadata: {}
nbands_factor: 3.0
relax:
  base:
    kpoints_distance: 0.15
    kpoints_force_parity: false
    metadata: {}
    pw:
      code: Plane wave Quantum ESPRESSO v6.6 - local compilation
      metadata:
        options:
          max_wallclock_seconds: 43200
          resources:
            num_machines: 1
          stash: {}
          withmpi: true
      parameters:
        CELL:
          cell_dofree: all
          press_conv_thr: 0.5
        CONTROL:
          calculation: vc-relax
          etot_conv_thr: 2.0e-05
          forc_conv_thr: 0.0001
          tprnfor: true
          tstress: true
        ELECTRONS:
          conv_thr: 4.0e-10
          electron_maxstep: 80
          mixing_beta: 0.4
        SYSTEM:
          degauss: 0.01
          ecutrho: 240.0
          ecutwfc: 30.0
          nosym: false
          occupations: smearing
          smearing: cold
      pseudos:
        Si: ''
  base_final_scf:
    metadata: {}
    pw:
      metadata:
        options:
          stash: {}
      pseudos: {}
  max_meta_convergence_iterations: 5
  meta_convergence: true
  metadata: {}
  volume_convergence: 0.02
scf:
  kpoints_distance: 0.15
  kpoints_force_parity: false
  metadata: {}
  pw:
    code: Plane wave Quantum ESPRESSO v6.6 - local compilation
    metadata:
      options:
        max_wallclock_seconds: 43200
        resources:
          num_machines: 1
        stash: {}
        withmpi: true
    parameters:
      CONTROL:
        calculation: scf
        etot_conv_thr: 2.0e-05
        forc_conv_thr: 0.0001
        tprnfor: true
        tstress: true
      ELECTRONS:
        conv_thr: 4.0e-10
        electron_maxstep: 80
        mixing_beta: 0.4
      SYSTEM:
        degauss: 0.01
        ecutrho: 240.0
        ecutwfc: 30.0
        nosym: false
        occupations: smearing
        smearing: cold
    pseudos:
      Si: ''
structure: Si

sphuber

Thanks @mbercx just two minor comments remaining. And then if you are still up for it, I would take you up on your suggestion to use YAML instead of JSON.

tests/conftest.py

aiida/engine/processes/builder.py

tests/engine/processes/test_builder.py

The representation of the `ProcessBuilder` (i.e. `__repr__`) is difficult to read since it just prints the heavily nested mapping on one line. Here we add a `_repr_html_` method to improve the readability when using the `ProcessBuilder` in e.g. Jupyter notebooks.

for more information, see https://pre-commit.ci

sphuber · 2022-01-27T12:34:08Z

aiida/engine/processes/builder.py

+        """Pretty representation for in the IPython console and notebooks."""
+        return p.text(
+            f'Process class: {self._process_class.__name__}\n'
+            f'Inputs:\n{yaml.safe_dump(json.JSONDecoder().decode(PrettyEncoder().encode(self)))}'


Doing a full encode/decode cycle in JSON before encoding in YAML is a bit inefficient. Then again, writing serializers in YAML is a bit of a mess (like in aiida.orm.utils.serialize, and unfortunately we cannot even reuse that) and this should not be horribly costly, so let's leave this as is.

Yeah, I know it's a bit of a messy workaround, but didn't know how to write the serializer in YAML to be honest. I timed the line with %%timeit and it only took a couple dozen nanoseconds, so should be ok. ^^

comments addressed

sphuber · 2022-01-27T12:35:28Z

Thanks a lot @mbercx !

mbercx added priority/nice-to-have type/enhancement topic/processes labels Jun 2, 2021

mbercx requested review from chrisjsewell and sphuber June 2, 2021 19:36

mbercx force-pushed the fix/print-process branch from 4870584 to e0d14b4 Compare June 2, 2021 20:40

mbercx changed the title ~~ProcessBuilder: Add HTML representation~~ ProcessBuilder: Add pretty representation 😍 Jun 3, 2021

mbercx mentioned this pull request Jun 4, 2021

👌 IMPROVE: Minor layout and UX issues aiidalab/aiidalab-qe#53

Closed

19 tasks

chrisjsewell marked this pull request as draft July 21, 2021 09:37

mbercx force-pushed the fix/print-process branch 2 times, most recently from f93dffc to cb0a906 Compare August 11, 2021 07:29

chrisjsewell mentioned this pull request Aug 11, 2021

mypy keeps complaining error: unused "type: ignore" comment in pre-commit #5026

Closed

mbercx added this to the v2.0.0 milestone Sep 22, 2021

mbercx force-pushed the fix/print-process branch 2 times, most recently from c32df03 to 11aeaab Compare October 3, 2021 21:26

mbercx reopened this Jan 26, 2022

sphuber requested changes Jan 26, 2022

View reviewed changes

tests/conftest.py Outdated Show resolved Hide resolved

aiida/engine/processes/builder.py Outdated Show resolved Hide resolved

tests/engine/processes/test_builder.py Outdated Show resolved Hide resolved

tests/engine/processes/test_builder.py Outdated Show resolved Hide resolved

mbercx and others added 14 commits January 27, 2022 11:29

Switch to using _repr_pretty_

fc7926a

Fix pre-commit (now tested with tox!)

4f9b986

Attempt to appease the CodeCov gods

0431dde

More tests

297db0c

Apply reviewer suggestions

6530761

use Dict import; switch to using textwrap.indent

2b1fe53

Switch to using a JSONEncoder

91bdb97

fix pre-commit

1308421

Reorganise tests

2903eac

Fixed mypy (I hope)

27005b5

[pre-commit.ci] auto fixes from pre-commit.com hooks

54b9bda

for more information, see https://pre-commit.ci

more test coverage

b85abc6

Apply reviewer suggestions

ff48c17

mbercx force-pushed the fix/print-process branch from e94c13c to ff48c17 Compare January 27, 2022 10:29

mbercx and others added 2 commits January 27, 2022 11:46

Add types-PyYAML dependency for pre-commmit

9fddc58

[pre-commit.ci] auto fixes from pre-commit.com hooks

cd91b0e

for more information, see https://pre-commit.ci

mbercx requested a review from sphuber January 27, 2022 11:28

sphuber reviewed Jan 27, 2022

View reviewed changes

sphuber approved these changes Jan 27, 2022

View reviewed changes

sphuber merged commit 2b6b2e9 into aiidateam:develop Jan 27, 2022

mbercx deleted the fix/print-process branch January 27, 2022 12:55

chrisjsewell mentioned this pull request Mar 12, 2022

🔀 MERGE: Release v2.0.0b1 #5426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ NEW: _repr_pretty_ representation for `ProcessBuilder` #4970

✨ NEW: _repr_pretty_ representation for `ProcessBuilder` #4970

mbercx commented Jun 2, 2021 •

edited

Loading

mbercx commented Jun 2, 2021 •

edited

Loading

mbercx commented Jun 2, 2021

mbercx commented Jun 2, 2021

louisponet commented Jun 2, 2021

codecov bot commented Jun 2, 2021 •

edited

Loading

mbercx commented Jun 2, 2021

louisponet commented Jun 2, 2021

mbercx commented Jun 2, 2021

mbercx commented Jun 2, 2021

mbercx commented Jun 2, 2021

chrisjsewell commented Jul 21, 2021

mbercx commented Aug 11, 2021

sphuber commented Aug 11, 2021

chrisjsewell commented Aug 11, 2021

mbercx commented Aug 12, 2021

mbercx commented Jan 26, 2022

mbercx commented Jan 26, 2022

sphuber left a comment

sphuber Jan 27, 2022

mbercx Jan 27, 2022

sphuber commented Jan 27, 2022

✨ NEW: _repr_pretty_ representation for ProcessBuilder #4970

✨ NEW: _repr_pretty_ representation for ProcessBuilder #4970

Conversation

mbercx commented Jun 2, 2021 • edited Loading

mbercx commented Jun 2, 2021 • edited Loading

mbercx commented Jun 2, 2021

mbercx commented Jun 2, 2021

louisponet commented Jun 2, 2021

codecov bot commented Jun 2, 2021 • edited Loading

Codecov Report

mbercx commented Jun 2, 2021

louisponet commented Jun 2, 2021

mbercx commented Jun 2, 2021

mbercx commented Jun 2, 2021

mbercx commented Jun 2, 2021

chrisjsewell commented Jul 21, 2021

mbercx commented Aug 11, 2021

sphuber commented Aug 11, 2021

chrisjsewell commented Aug 11, 2021

mbercx commented Aug 12, 2021

mbercx commented Jan 26, 2022

mbercx commented Jan 26, 2022

sphuber left a comment

Choose a reason for hiding this comment

sphuber Jan 27, 2022

Choose a reason for hiding this comment

mbercx Jan 27, 2022

Choose a reason for hiding this comment

sphuber commented Jan 27, 2022

✨ NEW: _repr_pretty_ representation for `ProcessBuilder` #4970

✨ NEW: _repr_pretty_ representation for `ProcessBuilder` #4970

mbercx commented Jun 2, 2021 •

edited

Loading

mbercx commented Jun 2, 2021 •

edited

Loading

codecov bot commented Jun 2, 2021 •

edited

Loading