Support for Exposure Data [EE API] #10

p-a-s-c-a-l · 2019-07-01T14:36:00Z

We need access to the Exposure Data stored in EMIKAT. Currently it supports Population Exposure including ~~age group distribution for whole Europe~~. There is also finer grained local population exposure data for the city of Naples available which can be served transparently via the same API.

The API could support a parameter $exposureType which contains the id of the respective Taxonomy.

The API will be accessed by the table-state-rest-api and then by the related EE Table Component. Additionally, we need an WMS Endpoint for the EE Map Component.

The respective EE Data Package Resource has to be updated accordingly and example request / response URLs (including WMS URLs for RA/IA maps) have to be added to the Wiki.

The text was updated successfully, but these errors were encountered:

p-a-s-c-a-l · 2019-07-08T07:51:51Z

I seems that "The population are no longer grouped according to age classes due to lack of sufficient data for the age groups.". @humerh can you please clarify this with @stefanon and @alecapolupo ?

ghost · 2019-07-08T10:28:39Z

Yes, the population will not be grouped in three classes since the data at european level are not available in that format. Copernicus uploaded the population data already spatialized, therefore they should just be converted in the grid format (https://land.copernicus.eu/local/urban-atlas/urban-atlas-2012).

p-a-s-c-a-l · 2019-07-08T10:35:14Z

the data at european level are not available in that format.

Yes, but Heinrich somehow managed to derive the age groups from other European datasets and to merge it with the Copernicus population data. Right, @humerh?

ghost · 2019-07-08T10:39:42Z

It is not necessary to apply that procedure because we have just vulnerability function for all the population.

p-a-s-c-a-l · 2019-07-10T05:55:30Z

Exposure data (Population) calculated from EMIKAT:

Methode:
We use the Populationen Data from EUROSTAT (1km*1km) and split this data to the 500m-Grid by using Landcover data from Copernicus. This data are available for all Europe.

REST API: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_FAKE.2016/table/data?rownum=1000

or:

https://service.emikat.at/EmiKatTst/api/scenarios/$emikatId/feature/tab.CLY_EL_POPULATION_FAKE.2016/table/data

Output Example:

{
  "name": "CLY_EL_POPULATION_FAKE",
  "description": "Beschreibung fürCLY_EL_POPULATION_FAKE",
  "columnnames": [
    "GRID_ID",
    "POPULATION_TOTAL",
    "POPULATION_LT14",
    "POPULATION_14TO64",
    "POPULATION_GT64"
  ],
  "rows": [
    {
      "rownum": 0,
      "values": [
        "500mE46730N19800",
        395,
        52,
        254,
        89
      ]
    },
    {
      "rownum": 1,
      "values": [
        "500mE46730N19805",
        394,
        52,
        254,
        88
      ]
    },
    {
      "rownum": 2,
      "values": [
        "500mE46735N19800",
        394,
        52,
        254,
        88
      ]
    },

p-a-s-c-a-l · 2019-07-10T06:36:33Z

@humerh

Thank you, some questions and remakes:

CLY_EL_POPULATION_FAKE.2016 should be renamed to CLY_EL_POPULATION_INTERPOLATED.2016 or something like that :-).
Population Data contains age groups which are not needed any more. We can keep them but they are probably just relevant when dealing with local exposure data.
We need a WMS Interface for the EE Map Component and ideally a WCS interface for the table-state-rest-api. Alternatively, a tabular format that is not as verbose as the JSON is preferred.

Alternatively we could keep the source data that is needed for visualisation purposes only on (METEOGRID) Geoserver and expose it via WCS/WMS and duplicate the data that is needed for Impact/Adaptation Calculation in EMIKAT. However, I would prefer to keep the data in one central place (EMIKAT).

Here are some samples WCS/WMS URLs:

Note that these URLs currently point to local Exposure Data obtained from ISTAT (Italian National Institute of Statistics)! The structure and format exposed at the APIs must be the same for the European level exposure data obtained from Copernicus. I think it is more than obvious that we cannot support different data formats for the same data type. We need to be able to switch data packages, e.g. perform the same type of study with European Level Data Package and the Local Data Package.

humerh · 2019-07-10T12:47:43Z

On your request I changed my Population evaluation and remove the age distribution.
I did also a renaming of table.

So my interfaces are now the followings (replace 2846 by $emikatId):

WMS: https://service.emikat.at/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity:view.2955&bbox=4673500.0,1980000.0,4691500.0,1988000.0&width=768&height=341&srs=EPSG:3035&format=image/gif&CQL_FILTER=SZ_ID=2846

p-a-s-c-a-l · 2019-07-10T13:35:08Z

Thanks! WMS doesn't seem to work, though:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE ServiceExceptionReport SYSTEM "https://service.emikat.at/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd"> <ServiceExceptionReport version="1.1.1" >   <ServiceException code="internalError">
      Rendering process failed
3
</ServiceException></ServiceExceptionReport>

Are you going to provide a WCS endpoint, too?

For the Data Package, we need some meta-information about the Exposure Data, especially to properly credit the original data provider (Copernicus). Is the description of the source data already available in our CKAN? If not, where can I find the the original data?

humerh · 2019-07-10T13:47:22Z

Additional Services available:
EMIKAT REST: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

or EMIKAT GeoJSON: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/geojson?rownum=1000

or CSV: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/csv?rownum=1000

or WMS:
WMS: https://service.emikat.at/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity:view.2955&bbox=4673500.0,1980000.0,4691500.0,1988000.0&width=768&height=341&srs=EPSG:3035&format=image/gif&CQL_FILTER=SZ_ID=2846

A WCS endpoint is not available.

p-a-s-c-a-l · 2019-07-10T13:55:19Z

For the Data Package, we need some meta-information about the Exposure Data, especially to properly credit the original data provider (Copernicus). Is the description of the source data already available in our CKAN? If not, where can I find the the original data?

Is the local Naples Exposure Data in EMIKAT available too, so that we can test with the table rest API?

humerh · 2019-07-10T14:15:47Z

What is the problem with WMS service? Is it still present?

p-a-s-c-a-l · 2019-07-10T14:25:54Z

WMS works now. Thanks!

p-a-s-c-a-l · 2019-07-16T13:17:35Z

EMIKAT REST: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

Hm, same question as here: Is it possible to return only those records that are within the area of the current study (2846)? Alternatively, does the API support additional filter parameters, e.g. by bounding box so that we could include the '$studyArea' in the request?

p-a-s-c-a-l · 2019-07-31T07:19:57Z

Hm, same question as here: Is it possible to return only those records that are within the area of the current study (2846)? Alternatively, does the API support additional filter parameters, e.g. by bounding box so that we could include the '$studyArea' in the request?

humerh · 2019-07-31T08:51:47Z

Yes, you will get only those records, which are in the area of the study, I have implemented it already.
As the content of the records will change due to the discussion about the duration I have not published it.

It will be like this:

https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_HAZARD_EVENTS_STUDY.2036/table/data?rownum=1000

p-a-s-c-a-l · 2019-08-07T08:55:52Z

When I remove the filter CQL_FILTER=SZ_ID=$emikat_id from the Population Exposure WMS URL, I'm, still getting only the exposure for the study area:

I would expect to get exposure data for whole Europe. Why? Because we want to show the visualisation widgets (maps and table) not only in the study views but also in the Data Package and resource views. See clarity-h2020/data-package#26

Therefore it would be helpful to be able to obtain the data for whole Europe via for the REST API, too. E.g. instead of

https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

we may use an optional query parameter, e.g.

https://service.emikat.at/EmiKatTst/api/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?scenario=2846

In any case, while this might work for (static) population data, it will probably not be available for HC-LE data and IA Data, because they are calculated on the fly for each new study, right?

humerh · 2019-08-07T09:18:45Z

When I remove the filter CQL_FILTER=SZ_ID=$emikat_id from the Population Exposure WMS URL, I'm, still getting only the exposure for the study area:

Currently we have only ONE Study defined and calculated in EmiKat, so the result is the same. If more than one Study exists, this parameter is essential.

... showing data for all the Europe ...

It was a strategic decision to calculate all the dynamic tables only for the area of the study. The recalculation time is therefore in a very good range (seconds or minutes). Recalculation time for Europe needs in many cases hours or days.

The examples of population distribution is currently handled with the same algorithms and is calculated only for the area of the study.
Locally I have the information of whole Europe available, so it would be possible to calculate also this layer on a European level. The input layers are also CONSTANTs and so this calculated population layer need not be recalculated for each Study.
But I need for my algorithms also this projected view to the study area, so it would be an adiitional task to make this European layer available.

A compromise could be, that we provide only the 1km*1km input layer of population on European level. Is this sufficient?

p-a-s-c-a-l · 2019-08-07T09:26:00Z

A compromise could be, that we provide only the 1km*1km input layer of population on European level. Is this sufficient?

Since this European population layer is then used for visualisation in CSIS only and not for calculation in EMIKAT, it would be better to follow the same approach as for the Hazard indices layers:

@ghilbrae should make these layers available on METEOGRID Geoserver.

ghilbrae · 2019-08-13T07:58:00Z

I'm working on this and I hope to have something today or tomorrow. I'd like to know of you @humerh need 2006 and 2011 or only 2011. I'm working on 2011 right now.

ghilbrae · 2019-08-13T10:50:38Z

@humerh the population for 2011 is available in our geoserver (https://clarity.meteogrid.com/geoserver/). The name of the layer is: europe:EXT_Population_GEOSTAT_1K_2011 You can check it in the layer preview.

Here is a link to the GetMap: https://clarity.meteogrid.com/geoserver/europe/wms?service=WMS&version=1.1.0&request=GetMap&layers=europe%3AEXT_Population_GEOSTAT_1K_2011&bbox=1896628.6179337814%2C1097454.5685083764%2C7058042.777624368%2C6827128.019856293&width=691&height=768&srs=EPSG%3A3035&format=application/openlayers

It may take a while to load but the layer is quite heavy. Let me know if this works for you or if there's something missing or strange.

humerh · 2019-08-21T07:01:39Z

I can access this layer as WMS service. I would prefer to have a WCS service for this layer, because I need the data and not the picture.
A WCS service would be the most efficent access to get the data of the study area.

Currently I continue to use my local copy of data till a WCS service is avalable.

humerh · 2019-08-25T17:19:23Z

As I need the data (not the map) is it possible to enable also WCS service for this layer?

p-a-s-c-a-l · 2019-08-26T04:32:58Z

Maybe @LauraMTG can enable the WCS?

LauraMTG · 2019-08-27T14:38:56Z

@humerh and @p-a-s-c-a-l , It is not possible to access SHAPE data through a WCS service (only for raster files). For this you need to use a WFS service and the procedure is the same.

LauraMTG · 2019-08-28T10:48:45Z

Hi @humerh and @p-a-s-c-a-l

I have transformed the shapefile into a raster file and it is already in geoserver with the name "geostat_GRID_POPULATION_1K_2011". Now you have the possibility to access the European population layer through the WCS service, if it is useful for you.

humerh · 2019-08-28T16:50:51Z

Thank you, it looks fine.
I will switch to this data source in the next days.

DenoBeno · 2019-09-24T09:37:55Z

I suppose this is resolved, except that no $exposureType filter exists and/since the data is only available for the population?

p-a-s-c-a-l · 2019-09-25T12:24:02Z

No Exposure Data seems to be available for this study (Alba Iulia): https://service.emikat.at/EmiKatTst/api/scenarios/3209/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

humerh · 2019-09-25T14:16:01Z

Study with EmikatId=3209 was not triggered for calculation. The reason is in my eyes on the drupal side.
So you get no data.

p-a-s-c-a-l · 2019-09-25T14:19:10Z

Hm, but there is an EMIKAT id assigned, hmm

DenoBeno · 2019-10-14T15:51:22Z

This works now. Guess we can close the issue.
Several issues to consider for later

the table is not very useful IMO.

GRID_ID is cryptic for the users, so per-grid data is generally not very useful in the tables. Unless we would have table & map linked on the same page.
=> maybe we should show average/min/max for the project area in a table?

How could we add "future exposure"?

p-a-s-c-a-l added enhancement New feature or request SHOWSTOPPER Feature or bug, that, if not addressed, renders the CSIS essentially useless BB: Catalogue of ER & AO Catalogue of Elements at Risk and Adaptation Options Building Block labels Jul 1, 2019

p-a-s-c-a-l added this to the D1.4 CLARITY CSIS v2 milestone Jul 1, 2019

p-a-s-c-a-l assigned humerh Jul 1, 2019

p-a-s-c-a-l changed the title ~~Exposure [EE API]~~ Support for Exposure Data [EE API] Jul 1, 2019

This was referenced Jul 1, 2019

1st Data Package: 03 Exposure Layers clarity-h2020/data-package#10

Closed

Exposure - Switch to EMIKAT API clarity-h2020/table-state-rest-api#7

Closed

Implement EE MAP clarity-h2020/map-component#32

Closed

p-a-s-c-a-l assigned ghilbrae, ghost and maesbri Jul 10, 2019

humerh closed this as completed Jul 10, 2019

p-a-s-c-a-l reopened this Jul 10, 2019

p-a-s-c-a-l mentioned this issue Jul 15, 2019

Meta-Data for 1st Data Package clarity-h2020/ckan#18

Closed

2 tasks

p-a-s-c-a-l modified the milestones: D1.4 CLARITY CSIS v2, EMIKT APIs Aug 7, 2019

p-a-s-c-a-l assigned LauraMTG Aug 26, 2019

p-a-s-c-a-l mentioned this issue Sep 5, 2019

CLARITY Data Package: Exposure Maps Resource specification clarity-h2020/csis#15

Closed

p-a-s-c-a-l mentioned this issue Sep 25, 2019

Studies not getting the EMIKAT ID assigned #20

Closed

p-a-s-c-a-l closed this as completed Oct 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Exposure Data [EE API] #10

Support for Exposure Data [EE API] #10

p-a-s-c-a-l commented Jul 1, 2019 •

edited

Loading

p-a-s-c-a-l commented Jul 8, 2019

ghost commented Jul 8, 2019

p-a-s-c-a-l commented Jul 8, 2019

ghost commented Jul 8, 2019

p-a-s-c-a-l commented Jul 10, 2019 •

edited

Loading

p-a-s-c-a-l commented Jul 10, 2019 •

edited

Loading

humerh commented Jul 10, 2019

p-a-s-c-a-l commented Jul 10, 2019

humerh commented Jul 10, 2019 •

edited

Loading

p-a-s-c-a-l commented Jul 10, 2019 •

edited

Loading

humerh commented Jul 10, 2019

p-a-s-c-a-l commented Jul 10, 2019

p-a-s-c-a-l commented Jul 16, 2019

p-a-s-c-a-l commented Jul 31, 2019

humerh commented Jul 31, 2019

p-a-s-c-a-l commented Aug 7, 2019 •

edited

Loading

humerh commented Aug 7, 2019

p-a-s-c-a-l commented Aug 7, 2019

ghilbrae commented Aug 13, 2019

ghilbrae commented Aug 13, 2019

humerh commented Aug 21, 2019

humerh commented Aug 25, 2019

p-a-s-c-a-l commented Aug 26, 2019

LauraMTG commented Aug 27, 2019

LauraMTG commented Aug 28, 2019

humerh commented Aug 28, 2019

DenoBeno commented Sep 24, 2019

p-a-s-c-a-l commented Sep 25, 2019 •

edited

Loading

humerh commented Sep 25, 2019

p-a-s-c-a-l commented Sep 25, 2019

DenoBeno commented Oct 14, 2019

Support for Exposure Data [EE API] #10

Support for Exposure Data [EE API] #10

Comments

p-a-s-c-a-l commented Jul 1, 2019 • edited Loading

p-a-s-c-a-l commented Jul 8, 2019

ghost commented Jul 8, 2019

p-a-s-c-a-l commented Jul 8, 2019

ghost commented Jul 8, 2019

p-a-s-c-a-l commented Jul 10, 2019 • edited Loading

p-a-s-c-a-l commented Jul 10, 2019 • edited Loading

humerh commented Jul 10, 2019

p-a-s-c-a-l commented Jul 10, 2019

humerh commented Jul 10, 2019 • edited Loading

p-a-s-c-a-l commented Jul 10, 2019 • edited Loading

humerh commented Jul 10, 2019

p-a-s-c-a-l commented Jul 10, 2019

p-a-s-c-a-l commented Jul 16, 2019

p-a-s-c-a-l commented Jul 31, 2019

humerh commented Jul 31, 2019

p-a-s-c-a-l commented Aug 7, 2019 • edited Loading

humerh commented Aug 7, 2019

p-a-s-c-a-l commented Aug 7, 2019

ghilbrae commented Aug 13, 2019

ghilbrae commented Aug 13, 2019

humerh commented Aug 21, 2019

humerh commented Aug 25, 2019

p-a-s-c-a-l commented Aug 26, 2019

LauraMTG commented Aug 27, 2019

LauraMTG commented Aug 28, 2019

humerh commented Aug 28, 2019

DenoBeno commented Sep 24, 2019

p-a-s-c-a-l commented Sep 25, 2019 • edited Loading

humerh commented Sep 25, 2019

p-a-s-c-a-l commented Sep 25, 2019

DenoBeno commented Oct 14, 2019

p-a-s-c-a-l commented Jul 1, 2019 •

edited

Loading

p-a-s-c-a-l commented Jul 10, 2019 •

edited

Loading

p-a-s-c-a-l commented Jul 10, 2019 •

edited

Loading

humerh commented Jul 10, 2019 •

edited

Loading

p-a-s-c-a-l commented Jul 10, 2019 •

edited

Loading

p-a-s-c-a-l commented Aug 7, 2019 •

edited

Loading

p-a-s-c-a-l commented Sep 25, 2019 •

edited

Loading