Convert prices to Moving Avg fixed for < cm_startyear to ref run; keep single-period marginals as |Rawdata

[orichters]

• Let a run have cm_startyear = 2025. Then, the Price|*|Moving Avg values that are used in the AR6 mapping for 2020 are p_average(2020) = (p(2015) + 2p(2020) + p(2025))/4 as calculated in remind2::reportPrices() by using magclass::lowpass(). And as the 2025 prices differ between the scenarios, so does the reported 2020 price, which it should not.
• See discussion here.

Solution:

• The marginal prices directly obtained from the model are saved as Price|…|Rawdata. This is for example used in compareScenarios2
• A moving average is calculated as Price|…|Moving Avg as before
• The variable Price|… will get a cleaned-up version that should be used for reporting
  • First, make sure all prices are non-negative
  • Calculate a moving average to smooth out spikes
  • For t < cm_startyear, use the Price values from the reference gdx_ref

Minor improvements:

• Added a check whether variable names contain NA, || or something like that. There are still some variables with NA but I haven't figured out what the unit should be
• fixed units for Res|Extraction|…|Cumulated which was NA
• fix fegat being reported as NA, see reportPrices reports NA for fegat in transport #405

For the record:

• My suggestion was first to replace Moving Avg by such a adapted calculation, but this was rejected because it would not be a moving average anymore and therefore confusing.
• Therefore, I implemented the version @Renato-Rodrigues suggested below…
• I asked whether there are other places where we have to call reportPrices with the gdx_ref: I will figure out with @dklein-pik whether convGDX2MIF_REMIND2MAgPIE() as called in the REMIND reporting should also get that added. Not sure whether MAgPIE uses these averages prices…

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

• Let a run have cm_startyear = 2025. Then, the Price|*|Moving Avg values that are used in the AR6 mapping for 2020 are p_average(2020) = (p(2015) + 2p(2020) + p(2025))/4 as calculated in remind2::reportPrices() by using magclass::lowpass(). And as the 2025 prices differ between the scenarios, so does the reported 2020 price, which it should not.

It might be that the 2020 prices in files using the AR6 mapping should not differ, but that is not the same as having Price|*|Moving Avg variables not being a moving averages in 2020, but something else.
Where else are those moving averages used?

[orichters]

It might be that the 2020 prices in files using the AR6 mapping should not differ, but that is not the same as having Price|*|Moving Avg variables not being a moving averages in 2020, but something else. Where else are those moving averages used?

The only places the Moving Avg variables are used are to the knowledge of myself and reposearch:

• plotting and comparison scripts such as AriadneComparison
• export templates in piamInterfaces (NAVIGATE, ECEMF, AR6, ARIADNE)

So this is a variable added only for the database export.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

So this is a variable added only for the database export.

You cannot infer the reason something has been introduced for in the past from a limited view of what it is used for in the present.

• plotting and comparison scripts such as AriadneComparison
• export templates in piamInterfaces (NAVIGATE, ECEMF, AR6, ARIADNE)

So this change would affect at least all those other projects, and they should have a say in it.

If memory serves, moving average prices where used instead of real prices because the latter where always weirdly spiky. That situation might have changed with remindmodel/remind#1238. Have you checked if you can use proper prices instead?

[robertpietzcker]

I would be fine with / in favor of the approach "don't change values in time steps before cm_startyear, even if we are talking about moving averages that theoretically would have to be changed for 100% conceptual consistency".

I have a hard time imagining somebody complaining that "your 2020 prices aren't real moving averages"
(I can much better imagine "why are your prices so weird and spiky", or "how can prices in the past change?" :-) )

[orichters]

FYI:

• the moving averages seem to be introduced in this PR by @fschreyer.
• they are generally used in the templates since this PR by myself. An alternative would be to revert that.

[Renato-Rodrigues]

I am against changing the moving averages calculation to add special rules, as I stated in the mattermost message I sent before.

If you want to post-process any model specific variable (not only prices), you should make very clear what your output reporting variable is doing.
Price variables are a mess as they are, and adding special rules that violate their own naming conventions only contributes to more confusion in the future.

I started working on a more structured price variable reporting some time ago, following strict rules like:

1. adding the tag MARGINAL to prices directly taken from the model without any post-processing.
2. make sure that additional tags clearly identify the embedded assumption behind specific post-processings, for cases like the Moving Average tag.
3. use pure price variable names, without any additional tags, only for our "official" reporting standards that include all necessary post-processing that are necessary to show our results to the outside World.

Meanwhile something like this is not fully implemented, as it is on hold due to other priorities, I would recommend to avoid infringing those principles, and if you have a too specific case, just do any project specific post-processing in project specific scripts.

[dklein-pik]

FYI: In the coupling MAgPIE does not use moving average prices but "Price|Carbon (US$2005/t CO2)". The same for the other Kyoto gases.

[robertpietzcker]

FYI: In the coupling MAgPIE does not use moving average prices but "Price|Carbon (US$2005/t CO2)". The same for the other Kyoto gases.

I think this discussion is all about energy prices, which tend to fluctuate in REMIND quite a bit. Carbon prices are (to my knowledge) never averaged

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

I think this discussion is all about energy prices, which tend to fluctuate in REMIND quite a bit. Carbon prices are (to my knowledge) never averaged

This is true, but completely conditional on where in the function prices are averaged. Push that down by 100 lines and Carbon prices get their Moving Avg buddies.

[dklein-pik]

FYI: In the coupling MAgPIE does not use moving average prices but "Price|Carbon (US$2005/t CO2)". The same for the other Kyoto gases.

I think this discussion is all about energy prices, which tend to fluctuate in REMIND quite a bit. Carbon prices are (to my knowledge) never averaged

It was just a reaction to Olli's question at the top.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Have you checked if you can use proper prices instead?

Well, still positively alpine in parts …

[orichters]

To me, it looks like:

• not using Moving Avg in the submission is no good option because of the spikes shown by Michaja
• changing the calculation of Moving Avg faces opposition
• Robert and myself want to avoid questions like "why are your prices so weird and spiky", or "how can prices in the past change?"

I think we can:

1. add variables Price|*|for Submission or whatever (suggestions welcome) and use that at least in the project mappings that want it. This name gives us more leeway on what that means, and could also allow in the future to fix the prices to other data, given we sometimes have problem with negative or spiking prices pre-2020
2. rename |Moving Avg to |for Submission if nobody needs the original
3. fix that somewhere in the post-processing (say: piamInterfaces), but that implies you have to be very careful passing the cm_startyear relations at a point where you usually only work with the mif files, and no direct relation to the gdx files is present anymore where you could look them up.

Given the three options I could imagine, my preference would be 1 or 2.

[Renato-Rodrigues]

May I ask why you ignored in your possible options the structured rules that I mentioned above, and that makes use of the already existent MARGINAL tag?

I would say that the best alternative instead would be to follow the already started process of:

1. Using the already existent MARGINAL tag to reflect any price directly taken from the model. This tag is only necessary to be added to prices that require some kind of post-processing.
2. Adapt, if necessary, the moving average variables to use these marginal price variables instead.
3. Use the clear price variable names to store our "release ready" values, after all post-processing is done. This includes adding any necessary moving average assumption; your change on the fixed years enforcement; and additional changes like for example enforcing minimum and maximum acceptable price values for corner cases when the solver marginals are at extreme values (this rule should be applied before the moving average post-processing in cases it is necessary).

This proposed solution follow the structure already in place, do not break existent mappings that use the current moving average calculation, and make sure that in the future we can use price variable names close to the templates used in projects without the need for selecting prices with special tags. We would always have our "clean" price naming convention reserved for our best interpretation of what would be the correct prices to report externally.

There is no need for a new special tag, named for Submission or whatever else, creating even more duplication and increasing the number of our reported variables. For most of the cases, the already existent variables would be more than enough to correctly report model and post-processed results.

[orichters]

Sorry, @Renato-Rodrigues, I honestly don't know why my mind forgot about your comment when I was thinking about the alternatives. Weird. I actually think your proposal is the most convincing one.

Did you already prepare something related to this process that I can build on?

If not, I would suggest to rename all the Price|* variables for which now a Price|*|Moving Avg is calculated to Price|*|Marginal, calculate the Price|*|Moving Avg as lowpass of the Marginals, and use Price|* for the adapted moving average? All other prices would stay unchanged… What about compareScenarios, MAgPIE coupling etc., should we use the Marginals there or the new reporting-"proof" prices?

[robertpietzcker]

What about compareScenarios, MAgPIE coupling etc., should we use the Marginals there or the new reporting-"proof" prices?

I think the compScen should always at least show non-smoothed prices so we always see how spiky prices are currently. (if somebody wants to see the averaged as well, fine with me)

[orichters]

Okay, implemented a first attempt of what I described above:

Some statistics, comparing old and new results of reportPrices:

• Without fixing, old variables that differ from new Marginal ones: 0
• With fixing, old variables that differ from new Marginal ones: 0
• With fixing, Moving Avg variables that differ from old ones: 0
• Without fixing, Moving Avg that differ from reporting variables: 0
• With fixing, Moving Avg that differ from reporting variables > 2020: 0
• With fixing, Moving Avg that do not differ from reporting variables in 2020: 629
• With fixing, Moving Avg that differ from reporting variables < 2020: 100

Doesn't look too bad. I think, in theory, the values below 2020 should not differ, if the fixing was perfect, but 100 data points is not too bad. If affects:

[1] "Price|Final Energy|Buildings|Gases|Natural Gas|Total LCOE|Moving Avg"
 [2] "Price|Final Energy|Buildings|Liquids|Oil|Fuel Cost|Moving Avg"
 [3] "Price|Final Energy|Buildings|Liquids|Oil|Total LCOE|Moving Avg"
 [4] "Price|Final Energy|Industry|Gases|Natural Gas|Total LCOE|Moving Avg"
 [5] "Price|Final Energy|Industry|Liquids|Oil|Fuel Cost|Moving Avg"
 [6] "Price|Final Energy|Industry|Liquids|Oil|Total LCOE|Moving Avg"
 [7] "Price|Final Energy|Industry|Solids|Coal|Total LCOE|Moving Avg"
 [8] "Price|Final Energy|Industry|Solids|Moving Avg"
 [9] "Price|Final Energy|Transport|Gases|Natural Gas|Total LCOE|Moving Avg"
[10] "Price|Final Energy|Transport|Liquids|Diesel|Fuel Cost|Moving Avg"
[11] "Price|Final Energy|Transport|Liquids|Diesel|Total LCOE|Moving Avg"
[12] "Price|Final Energy|Transport|Liquids|Petroleum|Fuel Cost|Moving Avg"
[13] "Price|Final Energy|Transport|Liquids|Petroleum|Total LCOE|Moving Avg"
[14] "Price|Secondary Energy|Liquids|Fossil|Moving Avg"
[15] "Price|Secondary Energy|Liquids|Moving Avg"

The difference is < 3% except for

    region variable                                          period  MovA    Rep
 49 REF    Pr|FE|Ind|Solids|Moving Avg                         2005  3.94  0.701
 55 REF    Pr|FE|Ind|Solids|Moving Avg                         2010  3.40  2.32

If somebody wants to investigate, mif files and scripts are lying around in /p/tmp/oliverr/remind2-prices.

[orichters]

We have to check inside remind2:

• inst/markdown/compareScenarios2/cs2_02_macro.Rmd -> use |Rawdata?
• R/reportCharts.R -> I would leave it as is and plot the new values to be reported
• R/reportCosts.R -> Use Rawdata, see answer below by David
• R/reportPrices.R -> what to do with Price|Marginal calculations already present for regipol?

outside remind2:

• piamInterfaces -> use Price|* without |Moving Avg if |Rawdata is in the data and therefore the reporting was made with remind2 version containing these changes here -> flexibly switch between Price|* and Price|*|Moving Avg piamInterfaces#123
• remindmodel: AriadneComparison -> deleting |Moving Avg.
• mip: plotstyle.csv: make sure also |Rawdata values are plotted correctly -> add Prices|*|Rawdata to plotstyles mip#70

[orichters]

@dklein-pik wrote me about reportCosts:

I think I would first try to stay as close as possible to the unchanged results, if the resulting quality of the results calculated by reportCosts allows it, i.e. they do not also vary strongly. The marginal is currently read in reportCosts for some quantities directly from the model and not obtained via reportPrices, as e.g. here

remind2/R/reportCosts.R

Line 132 in e66ab2b

price_fe_trans_emiMkt <- abs(demFeTrans.m[,y1,]/(budget.m[,y1,]+1e-10)) * 1000 / pm_conv_TWa_EJ # price of final energy in transportation per emission market (US$2005/GJ)

. This would have to be changed if we do not want to use the marginal, so that we are consistent within reportCosts.

[bs538]

After some discussion with Oliver I'd like to suggest a further addition to the price postprocessing:

Issue: For non-SSP2 scenarios, we currently have wild FE price fluctuations in 2025 (sometimes several 100%), which are caused by the fixing of non-SSP2 scenarios to SSP2-NDC until 2020. We discussed it in more detail in the Macro channel: https://mattermost.pik-potsdam.de/rd3/pl/g156uh5mgt8upjcxmgzocug7fe
For an example of results, see most FE prices here: /p/projects/piam/SHAPE_runs/coupled_2023-05-03_final/remind/compScen-SSP-SDP-PkBudg650-newWithTaxInc-2023-05-23_13.28.57-REMIND-MAgPIE.pdf

As a temporary fix, while we come up with a new way of handling the fixing (hopefully as part of the switch to NPi calibrations), we could implement one of the following options:

• A: for all non-SSP2 scenarios, replace 2025 (=cm_startyear) prices with average of 2020 and 2030. This needs to be done prior to calculating the moving average (akin to the current removal of negative prices) to prevent the spikes propagating into 2020 and 2030

• B: replace only prices that fluctuate beyond a current threshold, e.g. set p_tstart = min(max(0.5 * pavg, p_tstart), 1.5 * pavg), where pavg = (p_(tstart-5) + p_(tstart+5)) / 2 (suggestion by Oliver). This would also happen prior to the moving average.

• C: modify the moving average calculation so that 2025 prices are discarded (through adjusting the weights) - either for all non-SSP2 scenarios (as option A) or if they fluctuate beyond a threshold (as option B). This has the advantage that cleaning and moving average happen in one go (no double-averaging for 2025 price), but sounds a bit fiddly as one needs to modify the weighting scheme used in magclass::lowpass() for specific years and scenarios

In my view, Option A would be preferable, as it targets only the prices we know to be affected, and does so without having to specify an arbitrary threshold how much fluctuation is OK.

@Renato-Rodrigues @0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q @LaviniaBaumstark any thoughts?

[orichters]

@bs538: Another alternative would be to add something like that after this line.

out.reporting[, cm_startyear, ] <- 0.5 * ( priceRef[getRegions(out), cm_startyear, getNames(out)] +
                                           out.reporting[, cm_startyear, ] )

So the fixing would not be in time, but rather you avoid diverging too much from the reference run.

[bs538]

@orichters : thanks, interesting idea. But I think using the averaging in time domain would be more appropriate in this case, because we do actually expect a strong deviation from the reference run when the carbon price comes in in cm_startyear (especially for ambitious policy scenarios). I quickly looked at FE price changes from 2020 to 2025 in SSP2-PkBudg650 scenario (which has no fixing/adjustment issue), and here we also have price jumps of 50+ % for some FE|Sector|Carriers.

[orichters]

I mean, that is the general problem: I don't think we can easily differentiate legitimate price jumps from the others.

So indeed maybe the best temporary option is to identify from the gdx whether it is no SSP2 setting, and apply one of the options above. Modifying the weighting scheme seems tedious to me, so I would go with either A or B, depending on whether you actually like to keep part of the price changes or not.

The question is whether this is best done here or in some sort of post-processing. I have no strong opinion on that.

And I think we should put this issue high on the agenda to fix these problems at the root (if that is the root): remindmodel/remind#1068

[bs538]

I mean, that is the general problem: I don't think we can easily differentiate legitimate price jumps from the others.

So indeed maybe the best temporary option is to identify from the gdx whether it is no SSP2 setting, and apply one of the options above. Modifying the weighting scheme seems tedious to me, so I would go with either A or B, depending on whether you actually like to keep part of the price changes or not.

Fully agree on the general problem. That's why any sort of threshold approach is problematic. So I would pick Option A (because we know that non-SSP2 scenarios have a price problem in 2025).

The question is whether this is best done here or in some sort of post-processing. I have no strong opinion on that.

As we already fix price issues (such as negative prices, and smoothing out spikes) here, this seems to be the right place to add this further fix. If we do it in a separate post-processing, we need to go back to the Rawdata / Marginal prices, fix the problem there, and then re-apply the MovingAvg, so a lot of duplication.

And I think we should put this issue high on the agenda to fix these problems at the root (if that is the root): remindmodel/remind#1068

Again fully agree.

[bs538]

@orichters : thanks!

@0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q @Renato-Rodrigues @fbenke-pik quick review would be appreciated as I would like to use this improvement to the price reporting (both general and in particular d3b1a54 ) for final project scenarios quickly. thanks!

[fbenke-pik]

Sorry, I don't think I can be of help when it comes to validating rules for reporting variables. Whenever I extend the reporting (adding/adjusting reporting variables), it happens under supervision of felix, robert or renato. I just do the tech, but don't know anything about the logic behind the calculations.

[orichters]

Find attached a number of plots where I compare

• the old reporting
• the new reporting without any reference (nothing should have changed)
• the new reporting with a base run to which it is fixed until 2020
• the reference run

It all looks good from my side. (I moved the lines a bit left and right to make everything visible)

alldata.pdf

You were right, I adapted the code such that Moving Avg are still moving averages, but the plain variables are now used for reporting.