More Documentation on Asset creation

[nilutz]

Feature Request

More Documentation on Asset creation

Description

In a current use case we want to use an Asset and set its dataAddress to allow proxyQueryParams. As an example consider the following, our Asset has a baseUrl of https://example.com and the user can use queryParams to request different underlying data like so: https://example.com?query=hello&id=123. The question now arises how the assets in EDC would support such a behaviour. After some research we found out that one can enable flags to enable certain behaviour on producer side. So in order to create an asset, one would need to enable proxyQueryParams like so:

"dataAddress": {
    "properties": {
        "type": "HttpData",
        "baseUrl": "https://example.com",
        "method": "GET",
        "proxyQueryParams": "true",
    }
}

But how would the consumer now tell the producer which params to proxy to the dataAdress baseUrl ? I tried with the following request data:

"dataDestination": {
    "properties": {
      "type": "HttpData",
      "baseUrl": "https://destination-url.com",
      "assetId": "asset-id-123",
    },
  },
  "properties" : {
    "queryParams":  {
        "query":"hello",
        "id":123
    }
}

But had no luck. In addition the documentation or openapi.yaml has no information on what could go into the "properties" section on how to control the "dataAdress" or "dataDestination" behaviour.

Why Is the Feature Desired?

In order to for developers to use the EDC to hook up any data source this part of the documentation is crucial

[SebastianOpriel]

Thanks for your question! This is a feature of the data-plane-http extension. Please have a look for parameters here and here. You may use additionally ´pathSegments´ and ´queryParams´.

In a short search I found following snippet, but need to admit, that we do not support the feature above the current implementation of Core EDC.

  "properties": {
      "pathSegments": "search",
      "method": "GET",
      "queryParams": "q=sovity"
    },

[nilutz]

I tried the pathSegments and queryParams. In the following way:

"dataDestination": {
    "properties": {
      "type": "HttpData",
      "baseUrl": http://destination,
      "assetId": "asset-123",
    },
  },
  "properties" : {
    "method" : "PUT",
    "pathSegments": "@12,34",
    "mediaType": "text/javascript",
    "body": "hello",
    "queryParams":"query=hello"
}

Everything seem to work except that when using the method "GET" the "pathSegments" are not getting attached (only the queryParams). I guess that is also a bug in the EDC Core. Can you tell me where I would raise an Issue to have this flagged ? Thanks

[efiege]

Can you try using path instead of pathSegments? It seems that pathSegments is only used if proxyPath is activated.

Here you can find the constants, which are used to build the DataAddress: HttpDataAddress of the EDC.

[nilutz]

Just for future reference. I found the correct combination. In the dataAdress field:

 "dataAddress": {
        "properties": {
            "type": "HttpData",
            "baseUrl": baseUrl,
            "method": "GET",
            "header:Authorization": f"Bearer {bearer}",
            "proxyQueryParams":True,
            "proxyPath": True
        }
    }

and in the corresponding data request:

"dataDestination": {
    "properties": {
      "type": "HttpData",
      "baseUrl": data_transfer_destination,
      "assetId": contract_offer["asset"]["id"],
    },
  },
  "properties" : {
    "method" : "GET",
    "pathSegments": "@48.723413,9.169551",
    "queryParams":"radius=200000&cause=BUM&next_page_token=04352f1d-9087-43e8-a9aa-465e7c50b9e0"
}

So in order to enable "pathSegments" one must enable "proxyPath".

[kris-mart]

Do you also found something about headers passed as they are also required (for auth)?
(They had been in QueryInput-Class.)

[nilutz]

I found the following non-intuitive "header:Authorization":"Bearer some-super-secret-bearer-token" - then the json looks like so:

"dataAddress": {
        "properties": {
            "type": "HttpData",
            "baseUrl": "https://example.com",
            "method": "GET",
            "header:Authorization":"Bearer some-super-secret-bearer-token",
            "proxyQueryParams":True,
        }
    }

[kris-mart]

They should to be proxied as well via Transportrequest as Bearer quickly expires - It has been in IDS:
https://github.com/FraunhoferISST/DataspaceConnector/blob/main/src/main/java/io/dataspaceconnector/common/net/QueryInput.java

[SebastianOpriel]

You are referring to another connctor implementation, not the one which is used here. You are here in an EDC repository, not DSC (which you referred to)

[efiege]

You can pass additional headers, like @nilutz mentioned, by passing an additional property "header:headerName": "HeaderValue".

The Authorization Header is handled in a separate way:

• authKey will set the name of the header, which contains the authorization information
• authCode will set the secret
• secretName will set the name of the secret, which has been previously stored in the vault. When the EDC needs to access the data it will first look into the vault in resolve the secret. Furthermore the vault is just evaluated if authCode has not been set.

Here is an example for directly passing the secret:

"dataAddress": {
        "properties": {
            "type": "HttpData",
            "baseUrl": "https://example.com",
            "method": "GET",
            "header:MyCustomHeader":"HeaderValue",
            "authKey": "X-Api-Key",
            "authCode":, "my super secret key"
        }
    }

Here is an example for the vault access: (setting the X-Api-Key header to the content of the vault entry)

"dataAddress": {
        "properties": {
            "type": "HttpData",
            "baseUrl": "https://example.com",
            "method": "GET",
            "header:MyCustomHeader":"HeaderValue",
            "authKey": "X-Api-Key",
            "secretName":, "vault-entry-name"
        }
    }

Here is a pointer to the HttpDataAddress of the EDC.

[kris-mart]

For documentation completeness:
I have created Pull-Request how we solved the issue
https://github.com/sovity/EDC/pull/163