Features/integrate stochastic programming

[e-zolotarevskaya]

Integration of stochastic programming into oemof to optimize potential of providing ancillary services.
Resolves #778

• check if mpi-sppy works with pyomo version between 5.7.0 and 5.7.3
• custom ancillary services component
• custom constraint using mpi-sppy
• get tags for stages
• extract list of components/flows/objectives based on stage

[pep8speaks]

Hello @e-zolotarevskaya! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/oemof/solph/_groupings.py:

Line 91:1: E302 expected 2 blank lines, found 1
Line 94:45: E712 comparison to True should be 'if cond is True:' or 'if cond:'
Line 99:1: E305 expected 2 blank lines after class or function definition, found 1
Line 100:80: E501 line too long (80 > 79 characters)
Line 103:1: E302 expected 2 blank lines, found 1
Line 106:45: E712 comparison to True should be 'if cond is True:' or 'if cond:'
Line 111:1: E305 expected 2 blank lines after class or function definition, found 1
Line 112:80: E501 line too long (80 > 79 characters)
Line 115:1: E302 expected 2 blank lines, found 1
Line 121:1: E305 expected 2 blank lines after class or function definition, found 1

• In the file src/oemof/solph/_models.py:

Line 385:29: E231 missing whitespace after ','
Line 385:38: E231 missing whitespace after ','
Line 385:40: E231 missing whitespace after ','
Line 385:80: E501 line too long (83 > 79 characters)
Line 397:9: E303 too many blank lines (2)
Line 398:29: E231 missing whitespace after ','
Line 398:38: E231 missing whitespace after ','
Line 398:40: E231 missing whitespace after ','
Line 398:80: E501 line too long (89 > 79 characters)
Line 403:19: E231 missing whitespace after ','
Line 403:28: E231 missing whitespace after ','
Line 403:30: E231 missing whitespace after ','
Line 403:80: E501 line too long (80 > 79 characters)
Line 411:24: E231 missing whitespace after ','
Line 411:26: E231 missing whitespace after ','
Line 429:80: E501 line too long (89 > 79 characters)
Line 433:80: E501 line too long (99 > 79 characters)

• In the file src/oemof/solph/components/_generic_storage.py:

Line 1122:80: E501 line too long (85 > 79 characters)
Line 1131:80: E501 line too long (87 > 79 characters)

• In the file src/oemof/solph/components/experimental/_ancillary_services.py:

Line 13:1: E302 expected 2 blank lines, found 1
Line 24:1: W391 blank line at end of file

• In the file src/oemof/solph/flows/_flow.py:

Line 406:80: E501 line too long (85 > 79 characters)
Line 408:80: E501 line too long (94 > 79 characters)
Line 583:33: W504 line break after binary operator
Line 584:33: W504 line break after binary operator

• In the file src/oemof/solph/flows/_investment_flow.py:

Line 479:1: W391 blank line at end of file

Comment last updated at 2022-02-17 10:12:12 UTC

[jokochems]

Hi @e-zolotarevskaya: I'm happy to assist and review on this one.

I changed the base of your Pull Request to v0.5, meaning it will be merged into this branch. We discussed at the dev meeting that we want to merge every new feature into v0.5 as of now since this breaks the API compared to the dev branch. Looking at the commit history, it is all fine: You already created your additions based on v0.5, so there is no need for you to change anything.

I think, your draft component seems to be located correctly. You can check other experimental component classes for how to implement. Feel free to ask back if anything is unclear.

[e-zolotarevskaya]

@jokochems thank you!

Speaking of files location, what would be a good place for my test script? For now I made a nested directory in constraints folder, but didn't add it to git as I was unsure if it is right.

[jokochems]

@jokochems thank you!

Speaking of files location, what would be a good place for my test script? For now I made a nested directory in constraints folder, but didn't add it to git as I was unsure if it is right.

That depends on what you are referring to when you use the term "tests".

• Actual pytest scripts are located in the tests folder: https://github.com/oemof/oemof-solph/tree/v0.5/tests
  These will also be checked by the CI.
• Your ongoing development of the component and constraints you plan to add can happen directly in the components/experimental resp. constraints folder. For sole constraints, there is no experimental section foreseen as of now.
• When you use some kind of toy model setup for quick checks of your implementation, it is best not to track it in the repo. If you wish, you could version it elsewhere on your GitHub account.

[simnh]

Is there something new here?

@e-zolotarevskaya could you share the scripts you already used for your example (using pure pyomo)...it would then be easier to understand the conrete use case and find the abstract version and its implementation in solph...

[e-zolotarevskaya]

Hi! So far I've been mostly working on validating the concept in Julia. It's working quite well now, so I'm moving back to pyomo and oemof.
My python scripts are in this repository: https://github.com/e-zolotarevskaya/Ancillary_services_SP/tree/main/python_prototype
I decided not to add them here because they are still very much a work in progress. I got mpi-sppy stochastic problem solvers working on oemof, but the filter by stage is very simplistic and inflexible.

[simnh]

Hey,

this looks already quite great!

The variables you want i.e. for your first stage variables you have already found.

My first idea was to create a set with all flows (i.e. variables) of the first stage (there might be more, such as the storage level, or investment and binary (nonconvex) variables. But I think, all of them except the storage content, are associated with the flow. So from the API side something like this could be possible

    gen = solph.components.Source(
            label='gen',
            outputs={bus: solph.flows.Flow(
                     nominal_value=50,
                     variable_costs=10,
                     firststage=True)})

Then, internally, we build sets model.FIRSTSTAGE_VARS to build the first stage objective model.first_stage_objective and to easily collect the first stage vars to build the scenario tree:

    sputils.attach_root_node(
        model,
        model.first_stage_objective,
        [model.FIRSTSTAGE_PYOMO_VARS)

Collecting the first stage variables could also be done by users and we would not need to change the source code. Getting the first stage objective right is a little more tricky. But as every "Block" which guild constraints has a method ._objective_expression(), I think this would be the place where we distinguish between the expression for the first stage only and the rest. For a deterministic model, first stage and rest will simply be added, but for model.first_stage_objective can be used as shown above.

[simnh]

But we might have a call to discuss the details (what is required, what is possible, etc)

[simnh]

I added a first idea (only for the flow-associated variables).

Collecting the variables can acutally be easy once we have all our first stage cost expressions (as a pyomo-Expression instance)

For example like this with a model om:

from pyomo.core.expr import current as EXPR
[i for i in EXPR.identify_variables(om.FlowBlock.first_stage_gradient_costs)]

I suggest combining all first stage cost expressions in one big one om.first_stage_costs so its a one liner to geht the variables

[simnh]

This is obsolete..

[simnh]

I looked through the code and found the following places where cost expressions occur (they are all build in model.<blockname>._objective_expression()

Block	name
FlowBlock	variable_costs
FlowBlock	gradient_costs
InvestmentFlowBlock	investment_costs
NonConvexFlowBlock	startup_costs
NonConvexFlowBlock	shutdown_costs
NonConvexFlowBlock	activity_costs
NonConvexFlowBlock	gradient_costs
GenericInvestmentStorageBlock	investment_costs
SinkDSMDLRBlock	cost
SinkDSMDLRInvestmentBlock	cost
SinkDSMDIWBlock	cost
SinkDSMDIWInvestmentBlock	cost
SinkDSMOemofBlock	cost

I wonder if there is an elegant way of having only one file with all the method definitions "_objective_expression()" for the stochastic version and simply overriding the present method inside the block if e.g. the model has an attribute "stochastic" (or isinstance of solph.StochasticModel?

Of course, creating a whole new set of inherited classes, i.e. model.StochasticFlow(), where the method objective expression is different from the parent might be an option.

[e-zolotarevskaya]

Thank you for taking a look! I tried to do something similar to the set using filters and selecting flows associated with certain components, but this is much cleaner.

If we could have a call to discuss it in more detail I would appreciate it.

[simnh]

Just a small info, I rebuild the mpisppy farmer example with oemof-solph and results are the same. You can check it out here:

https://github.com/simnh/oemof-sp

[pdb5627]

What is the status of this work on stochastic programming integration? I'd be glad to help. I am not super deep into oemof or pyomo modeling, but I have a project I'm working on that I want to extend to include stochasticity and also integrate with and/or contribute models back to an energy modeling framework so that there's a better chance for my work to be useful to others.