Scenario analysis application is not showing all data

[DenoBeno]

This is what I see on https://csis.myclimateservice.eu/study/35/step/1528/view/scenario-analysis

Error message seems to be bogus.

[DenoBeno]

Hm wait, the data in question is really not shown in the table. However, the bar charts are missing some other data too, e.g. rcp 26 for 2071-2100 period.

the table component is still not showing anything, so I can't say what is really wrong here. https://csis.myclimateservice.eu/study/35/step/1528/view/table

[p-a-s-c-a-l]

Yes, this error occurs from time to time because the data retrieved from the EMIKAT RA/IA API does not contain all values. I don't know why this happens. BTW, this is different view (https://service.emikat.at/EmiKatTst/api/scenarios/3189/feature/view.2994/table/data) than the one used by the table component. It aggregates the values used by the table component. So maybe there's simply no result for

• RCP26 (2041-2070), Event Frequency: Rare
• RCP45 (2011-2040). Event Frequency: Frequent

available, for scenario 3189? Maybe @humerh can have a look?

[humerh]

Yes, you are right. These Events does not exist.

See the table:

1201	37,98	23,73			1	HW	Baseline	Baseline	Occasional	38_0.366d	34,7
1192	37,98	23,73			1	HW	Baseline	Baseline	Rare	39.5_0.066d	34,7
1210	37,98	23,73			1	HW	Baseline	Baseline	Frequent	35.5_3.133d	34,7
1193	37,98	23,73			1	HW	rcp26	20110101-20401231	Rare	40_0.133d	34,7
1211	37,98	23,73			1	HW	rcp26	20110101-20401231	Frequent	36.5_3.233d	34,7
1202	37,98	23,73			1	HW	rcp26	20110101-20401231	Occasional	39.5_0.266d	34,7
1212	37,98	23,73			1	HW	rcp26	20410101-20701231	Frequent	37.5_2.066d	34,7
1203	37,98	23,73			1	HW	rcp26	20410101-20701231	Occasional	40.5_0.366d	34,7
1205	37,98	23,73			1	HW	rcp26	20710101-21001231	Occasional	41_0.300d	34,7
1214	37,98	23,73			1	HW	rcp26	20710101-21001231	Frequent	37_3.333d	34,7
1195	37,98	23,73			1	HW	rcp26	20710101-21001231	Rare	42_0.133d	34,7
1199	37,98	23,73			1	HW	rcp45	20110101-20401231	Occasional	40.5_0.266d	34,7
1191	37,98	23,73			1	HW	rcp45	20410101-20701231	Rare	43.5_0.066d	34,7
1200	37,98	23,73			1	HW	rcp45	20410101-20701231	Occasional	41_0.266d	34,7
1209	37,98	23,73			1	HW	rcp45	20410101-20701231	Frequent	38_2.133d	34,7
1206	37,98	23,73			1	HW	rcp45	20710101-21001231	Occasional	42_0.266d	34,7
1196	37,98	23,73			1	HW	rcp45	20710101-21001231	Rare	42.5_0.066d	34,7
1215	37,98	23,73			1	HW	rcp45	20710101-21001231	Frequent	38_3.400d	34,7
1194	37,98	23,73			1	HW	rcp85	20110101-20401231	Rare	39.5_0.133d	34,7
1204	37,98	23,73			1	HW	rcp85	20110101-20401231	Occasional	39_0.333d	34,7
1213	37,98	23,73			1	HW	rcp85	20110101-20401231	Frequent	37_2.300d	34,7
1216	37,98	23,73			1	HW	rcp85	20410101-20701231	Frequent	39_2.466d	34,7
1197	37,98	23,73			1	HW	rcp85	20410101-20701231	Rare	42.5_0.133d	34,7
1207	37,98	23,73			1	HW	rcp85	20410101-20701231	Occasional	41.5_0.600d	34,7
1217	37,98	23,73			1	HW	rcp85	20710101-21001231	Frequent	41_2.733d	34,7
1198	37,98	23,73			1	HW	rcp85	20710101-21001231	Rare	45.5_0.066d	34,7
1208	37,98	23,73			1	HW	rcp85	20710101-21001231	Occasional	44_0.266d	34,7

[p-a-s-c-a-l]

These Events does not exist.

Any why? Is that a problem with the input hazard data or something else?

[humerh]

This question has to be answered by Robert (@RobAndGo).
I suppose, that there are not enough statistical data inputs available for this event.

[DenoBeno]

If that's a "normal" behaviour, so be it. But I don't understand it.

@RobAndGo, please explain.

[RobAndGo]

I wasn't able to see what problem you are all referring to, but I do know that there are cases where for a given grid point, specific heat wave events may not occur. e.g. it is possible that for a rcp85 scenario, a rare or occasional event may occur, but a frequent event is not registered because in the statistics, this occurred at an interval of either much greater than once per year, or much lower than once per year.

Just a reminder of the event definitions:
frequent: 1 event per year
occasional: 1 event in 5 years
rare: 1 event in 20 years.

For the 30-year periods on which this is based, these values correspond to:
frequent: 30 events per 30-year period
occasional: 6 events per 30-year period
rare: 1.5 events per 30-year period.

Using these strict definitions lead to many events being missed, so I searched for ranges to these values, namely:
frequent: 25-45 events per 30-year period
occasional: 4-8 events per 30-year period
rare: 1-3 events per 30-year period

However, there were still cases where events weren't present, and at the time I wasn't sure how significant a problem this would be. I could adjust the search ranges above such that every grid point has an event.

[DenoBeno]

A, so we may very easily miss the 2-year events, I see. But I don't think that this is the issue we have at hand. Look at https://github.com/clarity-h2020/scenario-analysis/issues/17#issue-502564514 again. The whole RPC26 is missing for the period 2041-2070, not just the rare event.

Unless the lack of the "rare" event makes the generation fail for the others too.

[DenoBeno]

By the way: it would be good if the scenario analysis app would only show the data for the "presets" (we can allow users to define more than one preset). In that way, there would be less data sets to show and the labels would be easier for the users since users can freely choose them.

[RobAndGo]

OK, now I think I understand - you mean that there is no data from the RCP26 2041-2070 scenario. I thought I uploaded all the files in a previous github post - I'll try to find it now to see if maybe I forgot this one.

I can't seem to find the original post, so I'll upload the missing dataset here:
europe_rcp26_20410101-20701231.csv.txt
It's a csv file so one just needs to remove the .txt extension.

As an aside, does anyone know how to search on github for particular posts, e.g. I wanted to search for posts containing the links for these data files which I previously posted.

[p-a-s-c-a-l]

It's a csv file so one just needs to remove the .txt extension.

OK. And who is responsible for making the data available via a service? Is it METEOGRID or @humerh? Is it directly imported into EMIKAT or fist into geo server then queried by EMIKAT?

As an aside, does anyone know how to search on github for particular posts, e.g. I wanted to search for posts containing the links for these data files which I previously posted.

@RobAndGo here

[DenoBeno]

Is this done now?

[p-a-s-c-a-l]

Is this done now?

No: "Could not load indicator file RCP45 (2011-2040). It contains no indicator data for indicatorFrequent"

[RobAndGo]

@p-a-s-c-a-l do you mean this is a problem with the original data set that I uploaded, because I do have frequent heat wave events for RCP45 (2011-2040), here is the file once again:
europe_rcp45_20110101-20401231.csv.txt

[p-a-s-c-a-l]

I don't know. This data comes from the EMIKAT RA/IA API, so maybe @humerh knows why the events are still missing.

[DenoBeno]

@humerh , WDYT?

[p-a-s-c-a-l]

Events are still missing.
Bug or Feature?

[p-a-s-c-a-l]

still not resolved yet: Could not load indicator file RCP45 (2011-2040). It contains no indicator data for indicatorFrequent

[p-a-s-c-a-l]

Could not load indicator file RCP45 (2011-2040). It contains no indicator data for indicatorFrequent

[p-a-s-c-a-l]

API Call with missing Events: https://service.emikat.at/EmiKatTst/api/scenarios/3189/feature/view.2994/table/data

[RobAndGo]

I'll upload the file again (I initially uploaded the rcp26 file for some reason):
europe_rcp45_20110101-20401231.csv.txt

[RobAndGo]

After discussing the issue further with @humerh, and seeing that @humerh did have all the data already correctly imported into Emikat, we determined that the problem is that which I addressed in the post above. That is, it is normal that a grid point may not have all three types of events (frequent, occasional, rare) for a particular RCP-scenario/future time period.

E.g. for the grid point at latitude:32.215336 and longitude: -4.425540, the rcp45, 2071-2100 does not have a rare event, but a rare event does exist for the rcp45 2011-2040 period.

latitude;longitude;urban area;country;RCP;Period;Tx95p;Frequent(1.0);Occasional(0.2);Rare(0.05)
32.215336;-4.425540;;;rcp45;20710101-21001231;36.5;40_3.533d;44.5_0.300d;
32.215336;-4.425540;;;rcp45;20110101-20401231;36.5;38.5_3.866d;42.5_0.366d;43_0.066d

[humerh]

CSIS-Application can call EmiKat to get all defined event types for a specific Study:

https://service.emikat.at/EmiKatTst/api/scenarios/3189/feature/tab.CLY_HAZARD_EVENTS_STUDY.2036/table/data
where 3189 is the <emikat_Id> of the study.

[p-a-s-c-a-l]

Finally, this has been resolved in clarity-h2020/scenario-analysis#19