Enhancing storage equations by considering `mode` and `lvl_temporal` #633

behnam-zakeri · 2022-07-28T13:33:38Z

This PR addresses #632 by adding mode to the mapping set of storage technologies (map_tec_storage) and also to storage equations.
More details:
This change means re-initializing the existing set of map_tec_storage with new index sets. Based on the discussion in #631, this re-initializing happens automatically by modifications in the file models.py, only if the mapping set is empty. If the set is not empty, an exception is raised, telling the user that they need to update this set.
Moreover, this PR improves the documentation of storage section to reflect the above changes and more than that, adding required details for each equation.

How to review

create a branch from the submitted PR; load an existing scenario (from Westeros, national or global models); clone a new scenario without carrying the solution (using the argument keep_solution=False); solve the cloned scenario again. The cloned scenario should solve without an error, if there is no content in the set map_tec_storage in that scenario. Otherwise, there should be an error asking you to update this set.
run test_feature_storage.py from this PR on the main branch; it should fail.
read the documentation of the storage section and migration notes, and make sure it's clear and complete.

PR checklist

Continuous integration checks all ✅
Test is enhanced (test_feature_storage.py) to reflect the new changes.
Expanded the documentation.
Update release and migration notes.

behnam-zakeri · 2022-07-28T14:57:54Z

message_ix/models.py

@@ -415,6 +419,61 @@ def enforce(scenario):
                scenario.remove_set(set_name, existing)
                scenario.add_set(set_name, expected)

+        # Enforcing new indexes for existing set and parameters


@khaeru, following the discussion under #631, I drafted a workflow to help reinitialize some existing sets and parameters here. The code checks if the item is empty before re-initializing, and if not raises an exception. I do believe this can be done much more compact and maybe under a separate method than enforce().

@khaeru Resuming working on this, I was wondering if you have time to have a look at the above point. In principle, it's a design issue; how we would like to alert users that they need to re-parameterize some existing sets and parameters in their scenario to make it work with the enhanced code.

@behnam-zakeri I've just pushed a couple commits with the requested changes.

Per this:

users that they need to re-parameterize some existing sets and parameters in their scenario to make it work with the enhanced code.

Did you already write some documentation or helper code about how they should do this? I think we ought to provide some…

I pushed another couple of commits to add a helper function expand_dims().

Thanks a lot, @khaeru both for expanding the doc, the elegant workaround for adding the re-initialized items, and the help function. It looks like very well-structured code and fits the purpose nicely.

behnam-zakeri · 2022-10-13T12:04:34Z

As discussed in the MESSAGEix meeting today (13 Oct), it will be useful if @khaeru can review the work-around suggested here to re-initialize an existing message_ix set or parameter.

behnam-zakeri · 2022-10-13T12:08:16Z

And it will be nice if @JulianHunt4 or @adrivinca can review this content-wise, i.e, related to the functionality of storage.

adrivinca · 2022-10-27T11:46:53Z

Thanks for the PR, it seems everything works well.
Concerning the 'how to review' procedure, I did it. But not sure if I got the second point correctly. Running the test_feature_storage.py on the main branch immediately gives errors because for instance expand_dims is not defined.

I wrote a comment to a point in the GAMs code where the mode component ,m, might have been forgotten.

And last I was wondering if the branch should be rebased to message_ix 3.6, or if, since there are no conflicts, it does not matter.

behnam-zakeri · 2022-10-27T21:32:46Z

Thanks @adrivinca. Where is your comment, I can't see that?

adrivinca · 2022-10-27T09:32:49Z

message_ix/model/MESSAGE/model_core.gms

@@ -51,8 +51,8 @@
 * :math:`REL_{r,n,y} \in \mathbb{R}`                       Auxiliary variable for left-hand side of relations (linear constraints)
 * :math:`COMMODITY\_USE_{n,c,l,y} \in \mathbb{R}`          Auxiliary variable for amount of commodity used at specific level
 * :math:`COMMODITY\_BALANCE_{n,c,l,y,h} \in \mathbb{R}`    Auxiliary variable for right-hand side of :ref:`commodity_balance`
-* :math:`STORAGE_{n,t,l,c,y,h} \in \mathbb{R}`             State of charge or content of storage at each sub-annual timestep
-* :math:`STORAGE\_CHARGE_{n,t,l,c,y,h} \in \mathbb{R}`     Charging of storage in each sub-annual timestep (negative for discharging)
+* :math:`STORAGE_{n,t,l,c,y,h} \in \mathbb{R}`             State of charge or content of storage at each sub-annual time slice


shouldn't you add the mode set here as well? or is it for version compatibility?

Thanks for noting this. I went through and added some missing indexes in the inline documentation.

behnam-zakeri · 2022-11-17T15:57:39Z

@khaeru if you don't have any additional concerns here, I can merge this.

khaeru · 2022-11-21T09:56:32Z

@khaeru if you don't have any additional concerns here, I can merge this.

GitHub says "This branch cannot be rebased due to conflicts". I'll try to rebase, then let's see if tests pass.

khaeru

Ready to merge once checks all pass.

behnam-zakeri · 2022-11-21T11:01:51Z

Ready to merge once checks all pass.

Thanks a lot @khaeru for finalizing this.

behnam-zakeri marked this pull request as draft July 28, 2022 13:33

behnam-zakeri self-assigned this Jul 28, 2022

behnam-zakeri added enh New features & functionality timeslice labels Jul 28, 2022

behnam-zakeri added a commit to behnam-zakeri/message_ix that referenced this pull request Jul 28, 2022

iiasa#633 Release notes updated

088ca79

behnam-zakeri force-pushed the storage-improvement branch from 37a55fd to 63737a2 Compare July 28, 2022 14:34

behnam-zakeri commented Jul 28, 2022

View reviewed changes

behnam-zakeri added a commit to behnam-zakeri/message_ix that referenced this pull request Oct 5, 2022

iiasa#633 Release notes updated

6b762b4

behnam-zakeri force-pushed the storage-improvement branch from 63737a2 to 66aaa6b Compare October 5, 2022 08:48

behnam-zakeri marked this pull request as ready for review October 5, 2022 09:01

behnam-zakeri force-pushed the storage-improvement branch from b444837 to ce47871 Compare October 5, 2022 09:52

behnam-zakeri requested a review from khaeru October 13, 2022 12:00

khaeru added a commit to behnam-zakeri/message_ix that referenced this pull request Oct 19, 2022

Edit release notes for iiasa#633; add expand_dims()

c435e96

khaeru force-pushed the storage-improvement branch from e94847c to cf979a7 Compare October 19, 2022 15:57

adrivinca reviewed Oct 28, 2022

View reviewed changes

behnam-zakeri added a commit to behnam-zakeri/message_ix that referenced this pull request Nov 8, 2022

iiasa#633 Release notes updated

a968773

khaeru pushed a commit to behnam-zakeri/message_ix that referenced this pull request Nov 21, 2022

iiasa#633 Release notes updated

b5dbce6

khaeru added a commit to behnam-zakeri/message_ix that referenced this pull request Nov 21, 2022

Edit release notes for iiasa#633; add expand_dims()

44f4d37

khaeru force-pushed the storage-improvement branch from 04d8c57 to 70267db Compare November 21, 2022 10:30

khaeru approved these changes Nov 21, 2022

View reviewed changes

khaeru added this to the 3.7 milestone Nov 21, 2022

khaeru linked an issue Nov 21, 2022 that may be closed by this pull request

Storage mapping sets do not consider mode of operation of technologies #632

Closed

behnam-zakeri force-pushed the storage-improvement branch from 53e020e to 70267db Compare November 23, 2022 13:17

behnam-zakeri and others added 18 commits November 23, 2022 14:24

Improve storage for "mode" of operation and level of temporal

a03472f

Improve the test of storage for representing "mode"

4a13431

Improve models.py to re-initialize some items with new index sets

9dc2bf0

Improve inline documentation and release notes

1c522f2

iiasa#633 Release notes updated

fdcc712

Apply black on the test file

890c190

Improve release notes and error message in models.py

01aadea

Update release notes

d8691d8

Update reporting network length manually

f2215bc

Remove extra printing lines in models.py

b15f0c6

Make use of "time slice" consistent in inline doc

1834e32

Add "mode" to missing index sets in the doc

7a3d039

Improve init/enforce of MESSAGE storage items

a7a6d8d

Test init/enforce of storage items

be1d0cb

Add .util.expand_dims and test

bd68e44

Edit release notes for iiasa#633; add expand_dims()

9214d78

Ensure scenario is checked out in MESSAGE.initialize()

cdb672f

Cleanup the migration notes

af375da

behnam-zakeri force-pushed the storage-improvement branch from 70267db to af375da Compare November 23, 2022 13:29

Remove repeating lines from Release notes

5bd09b8

behnam-zakeri merged commit 3b6651f into iiasa:main Nov 23, 2022

behnam-zakeri mentioned this pull request Nov 23, 2022

Re-initializing existing items #631

Closed

behnam-zakeri mentioned this pull request Dec 1, 2022

Extend user-defined relations to include the index of time #680

Draft

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancing storage equations by considering `mode` and `lvl_temporal` #633

Enhancing storage equations by considering `mode` and `lvl_temporal` #633

behnam-zakeri commented Jul 28, 2022 •

edited

Loading

behnam-zakeri Jul 28, 2022

behnam-zakeri Oct 5, 2022

khaeru Oct 19, 2022

khaeru Oct 19, 2022

behnam-zakeri Oct 20, 2022

behnam-zakeri commented Oct 13, 2022

behnam-zakeri commented Oct 13, 2022

adrivinca commented Oct 27, 2022

behnam-zakeri commented Oct 27, 2022

adrivinca Oct 27, 2022

behnam-zakeri Nov 8, 2022

behnam-zakeri commented Nov 17, 2022

khaeru commented Nov 21, 2022

khaeru left a comment

behnam-zakeri commented Nov 21, 2022

Enhancing storage equations by considering mode and lvl_temporal #633

Enhancing storage equations by considering mode and lvl_temporal #633

Conversation

behnam-zakeri commented Jul 28, 2022 • edited Loading

How to review

PR checklist

behnam-zakeri Jul 28, 2022

Choose a reason for hiding this comment

behnam-zakeri Oct 5, 2022

Choose a reason for hiding this comment

khaeru Oct 19, 2022

Choose a reason for hiding this comment

khaeru Oct 19, 2022

Choose a reason for hiding this comment

behnam-zakeri Oct 20, 2022

Choose a reason for hiding this comment

behnam-zakeri commented Oct 13, 2022

behnam-zakeri commented Oct 13, 2022

adrivinca commented Oct 27, 2022

behnam-zakeri commented Oct 27, 2022

adrivinca Oct 27, 2022

Choose a reason for hiding this comment

behnam-zakeri Nov 8, 2022

Choose a reason for hiding this comment

behnam-zakeri commented Nov 17, 2022

khaeru commented Nov 21, 2022

khaeru left a comment

Choose a reason for hiding this comment

behnam-zakeri commented Nov 21, 2022

Enhancing storage equations by considering `mode` and `lvl_temporal` #633

Enhancing storage equations by considering `mode` and `lvl_temporal` #633

behnam-zakeri commented Jul 28, 2022 •

edited

Loading