Improve reporting (phase 2)

[khaeru]

Superseded by #144; text below may be out of date.

---

This will implement the second phase of the reporting update.

See also:

• WIP: Improve reporting (phase 2) message_ix#176 for corresponding changes to message_ix.
• iiasa/message_data#41 for corresponding changes to the MESSAGE global model.
• WIP - do not merge: generic MESSAGEix reporting class message_ix#142 for phase 1.
• Improve reporting features message_ix#149 master issue for both phases.

Progress
Address requirements from wiki:

• A1. Provide equal support for any type of model that can be built on the ixmp.
• A2. Support use of exogenous data.
• A3. Support basic operations:
  • A3i. Weighted sums using model or exogenous weights.
  • A3ii. Disaggregation using model or exogenous shares.
  • A3iii. Interpolation: linear, exponential, etc.
• A4. Support operations using quantities at different levels of aggregation.
• A5. Support user-defined operations.
• A6. Duplicate or clone existing operations for multiple other sets of inputs or outputs.
• A7. Support renaming of quantities using multiple name mappings:
  • A7i. …given programmatically.
  • A7ii. …loaded from file.
• A8. Compute only a subset of quantities from a larger/complete report.
  • A8i. …for testing/development use-cases.
  • A8ii. …for high-performance use-cases.
  • A8iii. …using command-line arguments or other straightforward inputs to specify the subset.
• A9. Handle units for quantities.
  • Both quantities with provided units, and quantities without units.
  • Derive units for operations such as division and multiplication.
• A10. Provide automated description of how quantities are computed, as text (console output) or image.
• A11. Callable through retixmp (tests require Test rixmp using 'testthat' #135)

PR checklist:

• Tests added (ongoing)
• Documentation added — preview here.
• Description in RELEASE_NOTES.md added

Miscellaneous:

• Choose target release: 0.3 or 1.0?
• Decide: this adds dependencies on dask[array] (thus toolz), graphviz (for dask.visualize), pint, and xarray. Should these be non-optional requirements, or extras (ixmp[reporting])?
• Per @gidden comment below, provide user-friendly exception handling when bad units are encountered.

[khaeru]

From this point, the added dependency on xarray pulls in version 0.12, which drops support for Python 2; so Python 2 CI will fail. The last commit disables the Python 2 CI temporarily. There should probably be a separate PR that strips out all Python 2 compatibility hacks and removes the CI; this can be rebased after that's merged.

[gidden]

FWIW, I think we should freeze a release pretty soon (maybe after the first reporting feature @danielhuppmann is working on gets in). After that, we should drop py2 support in master and all subsequent releases.

…

On Tue, Apr 2, 2019 at 11:27 AM Paul Natsuo Kishimoto < ***@***.***> wrote: From this point, the added dependency on xarray pulls in version 0.12, which drops support for Python 2 <http://xarray.pydata.org/en/stable/whats-new.html#v0-12-0-15-march-2019>; so Python 2 CI will fail. The last commit disables the Python 2 CI *temporarily*. There should probably be a separate PR that strips out all Python 2 compatibility hacks and removes the CI; this can be rebased after that's merged. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#126 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVAEVvCJHkGOexWQnuHWN-_D9JS2O2Cks5vcyJsgaJpZM4bZCoh> .

[khaeru]

First light on visualization:

[khaeru]

Updated the PR description to show the requirements (copied from the wiki). Only a few remain.

[gidden]

Hey @khaeru would you mind rebasing this on latest master? I think that will fix the test failure and let me get started poking around. Thanks!

[gidden]

yay!

[khaeru]

Per discussion today with @volker-krey and @gidden, the plan is now:

1. Merge this into master more or less as-is,
2. Release it with 0.2 in the next week or two as clearly-marked ‘experimental’ code,
3. Refine with smaller PRs based on dogfooding with 2019 YSSPers and their supervisors, and
4. Promote from ‘experimental’ status in a major release (1.0, probably late 2019).

[khaeru]

• A3. Support basic operations:
  • A3iii. Interpolation: linear, exponential, etc.
• A7. Support renaming of quantities using multiple name mappings:
  • A7ii. …loaded from file.
• A11. Callable through retixmp (tests require Test rixmp using 'testthat' #135)

These will be addressed in separate PRs.

• Decide: this adds dependencies on dask[array] (thus toolz), graphviz (for dask.visualize), pint, and xarray. Should these be non-optional requirements, or extras (ixmp[reporting])?

@gidden would appreciate your input on this as part of your review. Currently they are added as mandatory requirements of the package.

[gidden]

Regarding your last point and prior experience, I think we want these to be mandatory deps

[gidden]

I will start adding notes here (and iiasa/message_ix#176 where appropriate) as I continue to fiddle with things.

TODO:

The current error provided when units are unknown to pint is not useful - it goes into dask internals and errors because a NoneType was provided. The entry point to the error is:

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-36-51e83ee23d33> in <module>

----> 1 rep.get(rep.full_key('out'))



~/.local/lib/python3.7/site-packages/ixmp-0.1.3.post0.dev99-py3.7.egg/ixmp/reporting/__init__.py in get(self, key)

   248         log.debug('Cull {} → {} keys'.format(len(self.graph), len(dsk)))

   249

--> 250         return dask_get(dsk, key)

We should catch these prior to calling dask_get() and return a more useful error message

[khaeru]

Superseded by #144.