The ENG Execution Core Container, based on the IDS Base Connector, is the core component of an IDS Connector enabling:
- the data exchange between connectors, using HTTPS, WS over HTTPS, IDSCP2 (beta)
- interaction with the AISEC Fraunhofer DAPS Service for requiring and validating a token
- communication with the Fraunhofer Clearing House for registering transactions
The configuration should be performed customizing the following variables in the docker-compose file:
- DATA_APP_ENDPOINT=https://localhost:8083/data DataAPP endpoint for receiving data (F endpoint in the above picture)
- MULTIPART=mixed DataAPP endpoint Content Type (choose mixed for Multipart/mixed or form for Multipart/form-data)
- Edit external port if need (default values: 8086 for web sockets over HTTPS, 8090 for http, 8887 for A endpoint and 8889 for B endpoint)
- Forward-To protocol validation can be enabled by setting the property application.enableProtocolValidation to true. If you have this enabled please refer to the following step.
- Forward-To protocol validation can be changed by editing application.validateProtocol. Default value is true and Forward-To URL must be set like http(https,wss)://example.com, if you choose false Forward-To URL can be set like http(https,wss)://example.com or just example.com and the protocol chosen (from application.properties)will be automatically set (it will be overwritten! example: http://example.com will be wss://example if you chose wss in the properties).
If you want to use your own certificate for the AISEC Fraunhofer DAPS server:
- Put DAPS certificates into the cert folder and edit related settings (i.e., application.keyStoreName, application.keyStorePassword) into the resources/application.properties file
Finally, run the application:
- Execute
docker-compose up
The Execution Core Container will use two ports (http and https) as described by the Docker Compose File. It will expose the following endpoints (both over https):
Following endpoints are open for all (credentials are not required), since in real scenario, they will be accessible only from dataApp:
/incoming-data-app/multipartMessageBodyBinary to receive data (MultiPartMessage) with binary body from Data App (the A endpoint in the above picture)
/incoming-data-app/multipartMessageBodyFormData to receive data (MultiPartMessage) with form-data body from Data App (the A endpoint in the above picture)
/incoming-data-app/multipartMessageHttpHeader to receive data (MultiPartMessage) represented with IDS-Message using http headers and payload in body from Data App (the A endpoint in the above picture)
/data to receive data (IDS Message) from a sender connector (the B endpoint in the above picture)
while 'B-endpoint' is public, and exposed to the world.
/data
More information about security and user credentials can be found in this link
Furthermore, just for testing it will expose (http and https):
/about/version
returns business logic version.
The ECC supports three different way to exchange data:
- REST endpoints enabled if IDSCP2=false and WS_OVER_HTTPS=false
- IDSCP2 enabled if IDSCP2=true and WS_INTERNAL=false (use https on the edge) or IDSCP2=true and WS_INTERNAL=true (use WS on the edge)
- Web Socket over HTTPS enabled if WS_OVER_HTTPS=true and IDSCP2=false
Execution Core Container allows setting up HttpFirewall through Spring Security. To turn it on/off, please take a look at following property:
#Firewall
application.firewall.isEnabled=true
If firewall is enabled, it will read properties defined in firewall.properties
file which easily can be modified by needs of setup.
#Set which HTTP header names should be allowed (if want to allow all header names, keep it empty)
allowedHeaderNames=
#Set which values in header names should have the exact value and allowed (if want to allow any values keep it empty)
allowedHeaderValues=
#Set which HTTP methods should be allowed (if want to allow all header names, keep it empty)
allowedMethods=GET,POST
#Set if a backslash "\" or a URL encoded backslash "%5C" should be allowed in the path or not
allowBackSlash=true
#Set if a slash "/" that is URL encoded "%2F" should be allowed in the path or not
allowUrlEncodedSlash=true
#Set if double slash "//" that is URL encoded "%2F%2F" should be allowed in the path or not
allowUrlEncodedDoubleSlash=true
#Set if semicolon is allowed in the URL (i.e. matrix variables)
allowSemicolon=true
#Set if a percent "%" that is URL encoded "%25" should be allowed in the path or not
allowUrlEncodedPercent=true
#if a period "." that is URL encoded "%2E" should be allowed in the path or not
allowUrlEncodedPeriod=true
IMPORTANT: If you're not an expert, the strong advice is to keep values at their default values. If you decide to change values, pay special attention to allowHeaderNames and allowHeaderValues, since those set values are exclusive and considered as only values that should be present in the header.
In order to communicate with UI, CORS (Cross-Origin Resource Sharing) settings should be configured in application.properties
file. This allows you to specify which origins, methods, and headers are permitted when making cross-origin requests to your application.
application.cors.allowed.origins=
application.cors.allowed.methods=
application.cors.allowed.headers=
application.cors.allowed.origins
: Specifies the allowed origins. If empty, all origins (*) are allowed.application.cors.allowed.methods
: Specifies the allowed HTTP methods. If empty, all methods (*) are allowed.application.cors.allowed.header
s: Specifies the allowed headers. If empty, all headers (*) are allowed.
Example configuration:
# Allow specific origins
application.cors.allowed.origins=https://example.com,https://another-example.com
# Allow specific HTTP methods
application.cors.allowed.methods=GET,POST,PUT,DELETE
# Allow specific headers
application.cors.allowed.headers=
The reachability could be verified using the following endpoints:
- http://{IP_ADDRESS}:{HTTP_PUBLIC_PORT}/about/version
Keeping the provided docker-compose will be:
- http://{IP_ADDRESS}:8090/about/version
The sender DataApp should send a request using the following schema, specifying in the Forward-To header the destination connector URL:
curl --location --request POST 'https://{IPADDRESS}:{A_ENDPOINT_PUBLIC_PORT}/incoming-data-app/multipartMessageBodyBinary' \
--header 'Content-Type: multipart/mixed; boundary=CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6' \
--header 'Forward-To: {RECEIVER_IP_ADDRESS}:{B_ENDPOINT_PUBLIC_PORT}/data' \
--data-raw ' --CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6
Content-Disposition: form-data; name="header"
Content-Type: application/json; charset=UTF-8
Content-Length: 1293
{
"@context" : {
"ids" : "https://w3id.org/idsa/core/",
"idsc" : "https://w3id.org/idsa/code/"
},
"@type" : "ids:ArtifactRequestMessage",
"@id" : "https://w3id.org/idsa/autogen/artifactRequestMessage/36cdbc3c-993d-4efe-a9bd-a88f400ff3f6",
"ids:transferContract" : {
"@id" : "http://w3id.org/engrd/connector/examplecontract"
},
"ids:correlationMessage" : {
"@id" : "http://w3id.org/artifactRequestMessage/1a421b8c-3407-44a8-aeb9-253f145c869a"
},
"ids:securityToken" : {
"@type" : "ids:DynamicAttributeToken",
"@id" : "https://w3id.org/idsa/autogen/dynamicAttributeToken/4f81873e-ca33-47ea-b777-b5485fc53253",
"ids:tokenValue" : "DummyTokenValue",
"ids:tokenFormat" : {
"@id" : "https://w3id.org/idsa/code/JWT"
}
},
"ids:modelVersion" : "4.2.7",
"ids:issued" : {
"@value" : "2021-11-24T15:09:01.276+01:00",
"@type" : "http://www.w3.org/2001/XMLSchema#dateTimeStamp"
},
"ids:issuerConnector" : {
"@id" : "http://w3id.org/engrd/connector"
},
"ids:senderAgent" : {
"@id" : "http://sender.agent/sender"
},
"ids:recipientAgent" : [ ],
"ids:requestedArtifact" : {
"@id" : "http://w3id.org/engrd/connector/artifact/test1.csv"
},
"ids:recipientConnector" : [ ]
}
--CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6
Content-Disposition: form-data; name="payload"
Content-Type: application/json
Content-Length: 50
{"catalog.offers.0.resourceEndpoints.path":"/pet2"}
--CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6--'
Keeping the provided configuration:
Multipart mixed request
curl --location --request POST 'https://localhost:8887/incoming-data-app/multipartMessageBodyBinary' \
--header 'Content-Type: multipart/mixed; boundary=CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6' \
--header 'Forward-To: https://localhost:8889/data' \
--data-raw ' --CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6
Content-Disposition: form-data; name="header"
Content-Type: application/json; charset=UTF-8
Content-Length: 1293
{
"@context" : {
"ids" : "https://w3id.org/idsa/core/",
"idsc" : "https://w3id.org/idsa/code/"
},
"@type" : "ids:ArtifactRequestMessage",
"@id" : "https://w3id.org/idsa/autogen/artifactRequestMessage/36cdbc3c-993d-4efe-a9bd-a88f400ff3f6",
"ids:transferContract" : {
"@id" : "http://w3id.org/engrd/connector/examplecontract"
},
"ids:correlationMessage" : {
"@id" : "http://w3id.org/artifactRequestMessage/1a421b8c-3407-44a8-aeb9-253f145c869a"
},
"ids:securityToken" : {
"@type" : "ids:DynamicAttributeToken",
"@id" : "https://w3id.org/idsa/autogen/dynamicAttributeToken/4f81873e-ca33-47ea-b777-b5485fc53253",
"ids:tokenValue" : "DummyTokenValue",
"ids:tokenFormat" : {
"@id" : "https://w3id.org/idsa/code/JWT"
}
},
"ids:modelVersion" : "4.2.7",
"ids:issued" : {
"@value" : "2021-11-24T15:09:01.276+01:00",
"@type" : "http://www.w3.org/2001/XMLSchema#dateTimeStamp"
},
"ids:issuerConnector" : {
"@id" : "http://w3id.org/engrd/connector"
},
"ids:senderAgent" : {
"@id" : "http://sender.agent/sender"
},
"ids:recipientAgent" : [ ],
"ids:requestedArtifact" : {
"@id" : "http://w3id.org/engrd/connector/artifact/test1.csv"
},
"ids:recipientConnector" : [ ]
}
--CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6
Content-Disposition: form-data; name="payload"
Content-Type: application/json
Content-Length: 50
{"catalog.offers.0.resourceEndpoints.path":"/pet2"}
--CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6--'
curl --location --request POST 'https://{IPADDRESS}:{A_ENDPOINT_PUBLIC_PORT}/incoming-data-app/multipartMessageBodyFormData' \
--header 'Content-Type: multipart/mixed; boundary=CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6' \
--header 'Forward-To: {RECEIVER_IP_ADDRESS}:{B_ENDPOINT_PUBLIC_PORT}/data' \
--form 'header="{
\"@context\" : {
\"ids\" : \"https://w3id.org/idsa/core/\",
\"idsc\" : \"https://w3id.org/idsa/code/\"
},
\"@type\" : \"ids:ArtifactRequestMessage\",
\"@id\" : \"https://w3id.org/idsa/autogen/artifactRequestMessage/36cdbc3c-993d-4efe-a9bd-a88f400ff3f6\",
\"ids:transferContract\" : {
\"@id\" : \"http://w3id.org/engrd/connector/examplecontract\"
},
\"ids:correlationMessage\" : {
\"@id\" : \"http://w3id.org/artifactRequestMessage/1a421b8c-3407-44a8-aeb9-253f145c869a\"
},
\"ids:securityToken\" : {
\"@type\" : \"ids:DynamicAttributeToken\",
\"@id\" : \"https://w3id.org/idsa/autogen/dynamicAttributeToken/4f81873e-ca33-47ea-b777-b5485fc53253\",
\"ids:tokenValue\" : \"DummyTokenValue\",
\"ids:tokenFormat\" : {
\"@id\" : \"https://w3id.org/idsa/code/JWT\"
}
},
\"ids:modelVersion\" : \"4.2.7\",
\"ids:issued\" : {
\"@value\" : \"2021-11-24T15:09:01.276+01:00\",
\"@type\" : \"http://www.w3.org/2001/XMLSchema#dateTimeStamp\"
},
\"ids:issuerConnector\" : {
\"@id\" : \"http://w3id.org/engrd/connector\"
},
\"ids:senderAgent\" : {
\"@id\" : \"http://sender.agent/sender\"
},
\"ids:recipientAgent\" : [ ],
\"ids:requestedArtifact\" : {
\"@id\" : \"http://w3id.org/engrd/connector/artifact/test1.csv\"
},
\"ids:recipientConnector\" : [ ]
}"' \
--form 'payload="{\"catalog.offers.0.resourceEndpoints.path\":\"/pet2\"}";type=application/json; charset=UTF-8'
Keeping the provided configuration:
Multipart form request
curl --location --request POST 'https://localhost:8887/incoming-data-app/multipartMessageBodyFormData' \
--header 'Content-Type: multipart/mixed; boundary=CQWZRdCCXr5aIuonjmRXF-QzcZ2Kyi4Dkn6' \
--header 'Forward-To: https://localhost:8889/data' \
--form 'header="{
\"@context\" : {
\"ids\" : \"https://w3id.org/idsa/core/\",
\"idsc\" : \"https://w3id.org/idsa/code/\"
},
\"@type\" : \"ids:ArtifactRequestMessage\",
\"@id\" : \"https://w3id.org/idsa/autogen/artifactRequestMessage/36cdbc3c-993d-4efe-a9bd-a88f400ff3f6\",
\"ids:transferContract\" : {
\"@id\" : \"http://w3id.org/engrd/connector/examplecontract\"
},
\"ids:correlationMessage\" : {
\"@id\" : \"http://w3id.org/artifactRequestMessage/1a421b8c-3407-44a8-aeb9-253f145c869a\"
},
\"ids:securityToken\" : {
\"@type\" : \"ids:DynamicAttributeToken\",
\"@id\" : \"https://w3id.org/idsa/autogen/dynamicAttributeToken/4f81873e-ca33-47ea-b777-b5485fc53253\",
\"ids:tokenValue\" : \"DummyTokenValue\",
\"ids:tokenFormat\" : {
\"@id\" : \"https://w3id.org/idsa/code/JWT\"
}
},
\"ids:modelVersion\" : \"4.2.7\",
\"ids:issued\" : {
\"@value\" : \"2021-11-24T15:09:01.276+01:00\",
\"@type\" : \"http://www.w3.org/2001/XMLSchema#dateTimeStamp\"
},
\"ids:issuerConnector\" : {
\"@id\" : \"http://w3id.org/engrd/connector\"
},
\"ids:senderAgent\" : {
\"@id\" : \"http://sender.agent/sender\"
},
\"ids:recipientAgent\" : [ ],
\"ids:requestedArtifact\" : {
\"@id\" : \"http://w3id.org/engrd/connector/artifact/test1.csv\"
},
\"ids:recipientConnector\" : [ ]
}"' \
--form 'payload="{\"catalog.offers.0.resourceEndpoints.path\":\"/pet2\"}";type=application/json; charset=UTF-8'
curl --location --request POST 'https://{IPADDRESS}:{A_ENDPOINT_PUBLIC_PORT}/incoming-data-app/multipartMessageHttpHeader' \
--header 'Content-Type: text/plain' \
--header 'Forward-To: {RECEIVER_IP_ADDRESS}:8889/data' \
--header 'IDS-Messagetype: ids:ArtifactRequestMessage' \
--header 'IDS-Id: https://w3id.org/idsa/autogen/artifactResponseMessage/eb3ab487-dfb0-4d18-b39a-585514dd044f' \
--header 'IDS-Issued: 2021-11-24T13:09:42.306Z' \
--header 'IDS-IssuerConnector: http://w3id.org/engrd/connector/' \
--header 'IDS-ModelVersion: 4.2.7' \
--header 'IDS-RequestedArtifact: http://w3id.org/engrd/connector/artifact/1' \
--header 'IDS-SecurityToken-Id: https://w3id.org/idsa/autogen/958a6a2a-5a94-4cf9-ad72-b39c59ee8955' \
--header 'IDS-SecurityToken-TokenFormat: https://w3id.org/idsa/code/JWT' \
--header 'IDS-SecurityToken-TokenValue: DummyTokenValue' \
--header 'IDS-SecurityToken-Type: ids:DynamicAttributeToken' \
--header 'IDS-SenderAgent: http://sender.agent.com/' \
--data-raw '{"catalog.offers.0.resourceEndpoints.path":"/pet2"}'
Keeping the provided configuration:
Http header request
curl --location --request POST 'https://localhost:8887/incoming-data-app/multipartMessageHttpHeader' \
--header 'Content-Type: text/plain' \
--header 'Forward-To: https//localhost:8889/data' \
--header 'IDS-Messagetype: ids:ArtifactRequestMessage' \
--header 'IDS-Id: https://w3id.org/idsa/autogen/artifactResponseMessage/eb3ab487-dfb0-4d18-b39a-585514dd044f' \
--header 'IDS-Issued: 2021-11-24T13:09:42.306Z' \
--header 'IDS-IssuerConnector: http://w3id.org/engrd/connector/' \
--header 'IDS-ModelVersion: 4.2.7' \
--header 'IDS-RequestedArtifact: http://w3id.org/engrd/connector/artifact/1' \
--header 'IDS-SecurityToken-Id: https://w3id.org/idsa/autogen/958a6a2a-5a94-4cf9-ad72-b39c59ee8955' \
--header 'IDS-SecurityToken-TokenFormat: https://w3id.org/idsa/code/JWT' \
--header 'IDS-SecurityToken-TokenValue: DummyTokenValue' \
--header 'IDS-SecurityToken-Type: ids:DynamicAttributeToken' \
--header 'IDS-SenderAgent: http://sender.agent.com/' \
--data-raw '{"catalog.offers.0.resourceEndpoints.path":"/pet2"}'
The receiver connector will receive the request to the specified "Forward-To" URL, process data and finally send data to the DATA_APP_ENDPOINT as specified in its docker-compose. The data will be sent to the Data App using a body request as specified by the MULTIPART environment variable in the docker-compose.
IDSCP2 is used only between ECCs. Follow the REST endpoint or WS examples, put the server hostname/ip address in the Forward-To header (wss/https://{RECEIVER_IP_ADDRESS/Hostname}:{WS_PUBLIC_PORT}).
Follow the REST endpoint examples, taking care to use wss://{RECEIVER_IP_ADDRESS}:{WS_PUBLIC_PORT} in the Forward-To header.
Information on how to interact with a Broker, can be found on following link
To manage your Self Description Document please check following link
Audit events logging can be configured following this document
Connector health check functionality can be found in this document
Requirements:
Java11
Apache Maven
To build the execution core container you will have to do one of the following:
Solution 1
Use provided libraries on GitHub Package. To do so, you will have to modify Apache Maven settings.xml file like following:
Add in servers section:
<servers>
<server>
<id>github</id>
<username>some_username</username>
<password>{your GitHub Personal Access Token}</password>
</server>
</servers>
How to get GH PAT, you can check following link
Solution 2
- Clone Multipart Message Library
- Once this project is cloned, run
mvn clean install
This will install an internal library that is needed by the execution core container project.
After that you can run mvn clean package
in the root of the execution core container project, to build it.
NOTE: If you proceed with Solution 2, pay attention to the Multipart Message Library version in pom.xml file, and check if the same version is used in the execution core container pom.xml, if not modify them according to the one from the clone repository.
If you wish to run ECC from IDE, please address following prerequisite:
-
Add src/main/resources directory to class path; either by making this change in Run configuration in IDE or comment out exclude section in pom.xml. This resource directory is removed from final jar, to externalize property files from docker image.
-
Copy ssl-server.jks file from TRUE Connector repo into some directory on the file system
-
Configure property application.targetDirectory= to point to the file containing ssl-server.jks file
-
Configure DAPS related properties, if DAPS will be used.
application.dapsUrl=
application.keyStoreName=
application.keyStorePassword=
application.keystoreAliasName=
application.trustStoreName=
application.trustStorePassword=
application.connectorUUID=
application.dapsJWKSUrl=
- Choose one of 2 profiles: SENDER or RECEIVER.
- Set environment variable with following name: AES256-SECRET-KEY with any value (MyPassword for example). This will be used for encrypting AuditLog column in DB.
- Start application
Once you build the execution core container, if required, you can build docker image, by executing following command, from terminal, inside the root of the project:
docker build -t some_tag .
This repository implements following branch management:
and has several GitHub action files to support such functionality. Those files are located in:
*.github\workflows*
Used when code is pushed to branch prefixed with feature/ or hotfix/
Customize GHA to fit your needs. For now, just run mvn clean package
Executed when code is pushed to develop branch.
Manual trigger of the GHA.
Perform mvn release:prepare and mvn release:perform.
Input parameters:
release version
next development version
tag version
Manual trigger of the GHA.
Build docker image, and push it to dockerhub.
Sign with cosign.
Input parameters:
versionName
tagMessage
Create issue in Issue tab in GitHub repo, before starting to work on new functionality. It would be nice to provide task breakdown, with estimation. Try not to have activities that are longer than 8 hours. If such activity is present in task breakdown, please split this activity in 2 or more sub activities, trying that new sub activities are not bigger than 8 hours.
Example could be like following:
Implement new ServiceA - 16h
Implement method A in ServiceA 4h
Implement method B in ServiceA 3h
Implement method C in ServiceA 7h
Implement method D in ServiceA 2h
Write unit tests for ServiceA - 6 hours
Update documentation - 2 hours
If you need to work on new feature, be sure first to pull changes from origin, and create new branch, following the naming convention from develop branch.
When creating pull request, double check if PR will merge to develop branch and not master. If this is not the case, change destination branch to be develop.
Code coverage is checked by using jacoco plugin.
For more up to date information about code coverage, you can check report after you build a project. Report can be found in
target\site\jacoco\index.html