- 1. Introduction
- 2. Changes
- 3. Configuration
- 4. How it works
- 5. API
- 5.1. PUT /add-host
- 5.2. PUT /add-identifier{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?uri}
- 5.3. PUT /add/{objectType}/{nameSpace}/{idNumber}
- 5.4. PUT /add/{nameSpace}/{objectType}/{versionNumber}/{idNumber}
- 5.5. PUT /add-uri-to-identifier{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?uri}{?preferred}
- 5.6. POST /bulk-add-identifiers
- 5.7. POST /bulk-remove-identifiers
- 5.8. GET /current-identity{?uri}
- 5.9. Delete /delete-identifier{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?reason}
- 5.10. GET /links/{objectType}/{nameSpace}/{idNumber}
- 5.11. POST /move-identity
- 5.12. GET /preferred-host
- 5.13. GET /preferred-link/{objectType}/{nameSpace}/{idNumber}
- 5.14. Delete /remove-identifier-from-uri{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?uri}
- 5.15. Put /set-preferred-host
- 5.16. GET /stats
Mapper moved to Micronaut framework.
The Mapper maps URI identifiers to resources based on content negotiation. It is the place an ID is created and mapped to a resource.
From that point on the mapped URI becomes a permalink that will always resolve.
The mapper manages de-duplication of resources by allowing you to move a mapping from one resource identifier to another. It also manages deletion of resources allowing you to provide a reason for deleting a resource, which is returned with a 410 GONE http response.
You can map a URI to different servers based on the content type requested, e.g. JSON, XML, RDF, HTML.
-
The broker endpoint is only for resolving IDs. In V1 preferred host and link were also obtained from the
/broker/preferredLink
endpoint. This complicated the URL resolution for straight redirection brokering (e.g. a resource proxy) and the API. -
all API calls are now under the
/api/blah
-
endpoints are moving from camel case to skewer case ie.
/broker/preferredHost
→/api/preferred-host
-
only serving up JSON, we will not respond with xml on our mapper API as it’s not used
-
get version of add-identifier has been deprecated, there are two new put versions of add-identifier
-
removed bulkRemoveByUri as it wasn’t used
-
changed addURI to add-uri-to-identity
-
removed addIdentifierToUri as functionally equivalent to add uri to identifier
The broker uses Regex to pull apart the resolution requests. The Regex used for the broker expects there to be a /broker in the context path. If you have a proxy in front of the mapper you may need to adjust the context path in this Regex.
You can set it in the nsl-mapper-config-mn.groovy or change the application.yml in the classpath. set the mapper.brokerRegex
in the configuration (see below).
The urlRegex
shouldn’t need to be changed, that is used by the mapper to get identities from a given URL.
mapper:
brokerRegex: ^(https?://[^/]*)?/broker/(.*?)(/api/.*?)?(\.json|\.xml|\.rdf|\.html)?$
urlRegex: ^(https?://[^/]*)?/?(.*?)(/api/.*?)?(\.json|\.xml|\.rdf|\.html)?$
The application.yml defines the default datasource and this can be compiled in or changed the the external groovy configuration file.
datasources:
default:
url: jdbc:postgresql://localhost:5432/nsl
username: nsl
password: nsl-s33kr1t
driverClassName: org.postgresql.Driver
autoCommit: false
schema: mapper
socketTimeout: 30
you probably only want to override the url and username/password in the external config, e.g.
datasources {
'default' {
url = 'jdbc:postgresql://localhost:5432/nsl'
username = 'nsl'
password = 'v3ryS3cr3t'
}
}
An external configuration file is used to configure the resource service to map a URL to.
import au.org.biodiversity.mapper.Identifier
mapper {
resolverURL = 'http://localhost:8080/mapper' //(1)
defaultProtocol = 'http' //(2)
Map serviceHosts = [ //(3)
bpni: 'http://bpni.com',
apni: 'http://apni.com/nsl',
ausmoss: 'http://ausmoss.com/nsl',
algae: 'http://algae.net/thing',
fungi: 'http://fungi.foo',
foa : 'http://test.biodiversity.org.au/'
]
Closure htmlResolver = { Identifier ident -> //(4)
String host = serviceHosts[ident.nameSpace]
if (ident.objectType == 'treeElement') {
return "${host}/services/rest/${ident.objectType}/${ident.versionNumber}/${ident.idNumber}"
}
return "${host}/services/rest/${ident.objectType}/${ident.nameSpace}/${ident.idNumber}"
}
Closure jsonResolver = { Identifier ident ->
String host = serviceHosts[ident.nameSpace]
if (ident.objectType == 'treeElement') {
return "${host}/services/json/${ident.objectType}/${ident.versionNumber}/${ident.idNumber}"
}
return "${host}/services/json/${ident.objectType}/${ident.nameSpace}/${ident.idNumber}"
}
format { //(5)
html = htmlResolver
json = jsonResolver
xml = htmlResolver
rdf = {Identifier ident -> return null}
}
auth = [ //(6)
'TEST-services': [
secret: 'buy-me-a-pony',
application: 'services',
roles : ['admin'],
],
'TEST-editor': [
secret: 'I-am-a-pony',
application: 'editor',
roles : ['admin'],
]
]
db { //(7)
url = 'jdbc:postgresql://localhost:5432/nsl'
username = 'nsl'
password = 'nsl'
}
}
-
the URL to reach the mapper from the internet e.g. https://id.biodiversity.org.au
-
the protocol to use when connecting to a host, e.g. https to stop being redirected to the secure endpoint then redirected to the service
-
using a locally defined Map of hosts to simplify things. i.e. you can use plain old groovy code
-
A Groovy Closure to resolve a request for a html resource. This is just plain old Groovy used in the format section.
-
The mapping of formats to a Closure that can do the work. The closure will be called with the Identifier as it’s argument and you provide a URL to redirect to to get the resource. Returning null will cause a
404 Not Found
to be returned. -
Authentication list mapped by the service username. Only 'admin' role is currently supported.
-
set the database connection details here
Note
|
Only HTML, JSON, XML and RDF content types are supported in this version of the mapper. See the ContentNegService .
|
The mappers job is to permanently link URLs (URIs) to resources through content negotiation. It acts as a service broker when given a resolvable URL.
The aim is to use a URL as the ID for a resource. Being a URL it is resolvable, and depending on the content type requested different services may provide the resultant resource. The following sequence diagram shows the sequence of a content resolution request.
To do it’s job the Mapper needs to know the URLs to map to a particular resource. The mapper defines two entities to describe these:
-
Match: The URI string.
-
Identifier: The data used to uniquely identify a resource.
The Match and Identifier are linked as Many to Many relationships, so an Identifier can have many Matches (the usual case) and a Match can have many Identifiers. The second case, where a Match has many Identifiers is used where there are many resources that describe a particular ID. The resources all together describe the thing Identified. An example may be a Taxanomic Name, which may have many concepts described in many publications by many authors. e.g.
which returns a series of resources:
https://biodiversity.org.au/nsl/services/rest/name/apni/442093 https://biodiversity.org.au/nsl/services/rest/instance/apni/975779 https://biodiversity.org.au/nsl/services/rest/instance/apni/975776
In the interests of speed and the typical use case, The Broker in this version of the mapper will redirect you to the first resource found. The idea is that it would be better to have a single summary resource that directed you to the other resources.
The mapper also maps a Host to the URI (to make a URL). This allows historical URLs to work when a change of preferred host occurs. We at IBIS have gone through the following host changes:
-
biodiversity.org.au
-
biodiversity.org.au/boa
-
www.anbg.gov.au - original
-
id.biodiversity.org.au - currently preferred
The change in host can happen for many reasons, and once a host is published in a URL we need to maintain it if possible.
Hosts are linked to specific Matches because the reverse proxy may only be able to resolve certain patterns. For example
the old link for Doodia R.Br. https://biodiversity.org.au/apni.name/16512
uses the biodiversity.org.au host and works
because the reverse proxy can match the apni.name
pattern.
A Match can have many hosts. When the mapper is asked for all the links to a resource it will mark the preferred link in the list (see 1 below).
[
{
"link": "http://id.biodiversity.org.au/name/apni/70914",
"resourceCount": 1,
"preferred": true, //(1)
"deprecated": false,
"deleted": false
},
{
"link": "http://biodiversity.org.au/boa/name/apni/70914",
"resourceCount": 1,
"preferred": false,
"deprecated": false,
"deleted": false
},
{
"link": "http://id.biodiversity.org.au/70914",
"resourceCount": 1,
"preferred": false,
"deprecated": false,
"deleted": false
},
{
"link": "http://www.anbg.gov.au/cgi-bin/apni?taxon_id=16512",
"resourceCount": 1,
"preferred": false,
"deprecated": true,
"deleted": false
},
{
"link": "http://biodiversity.org.au/apni.name/16512",
"resourceCount": 1,
"preferred": false,
"deprecated": false,
"deleted": false
},
{
"link": "http://id.biodiversity.org.au/Doodia R.Br.",
"resourceCount": 15,
"preferred": false,
"deprecated": false,
"deleted": false
}
]
Here is the data structure of the mapper Database.
The brokers job is to redirect you to a service that can give you the resource you want. It does that using a 303 redirect
called a See Other
. If you ask for a deprecated URI, one that we don’t want you to use anymore, you will get a
Moved Permanently
or 301 redirect
. In theory a service seeing a 301 redirect will update it’s link to the new link
then request that one in the future. People hitting that link in the browser won’t notice, they’ll just get the resource.
The broker uses the configuration file to work out how to redirect to a service that will serve the resource. See the Configure mapping section for a description of configuring the mapping from an Identifier to a service based on format.
Tip
|
The API is available via swagger-ui/index.hml as testable documentation. Unfortunately that won’t work if you’re reading this on GitHub. I’ll update with a more permanent documentation site soon. |
The API provides information and administration endpoints. To change information you need to be authenticated. The Authentication is done via a JSON call to the /api/login endpoint with the username and password of the service calling the API. A JSON Web Token (see https://jwt.io/) is returned and must be presented as a Bearer Token in the Authorization header. See the ApiControllerSpec for examples of how to do this using Micronaut:
private String login() {
HttpRequest request = POST('/login', '{"username":"TEST-services","password":"buy-me-a-pony"}')
HttpResponse<BearerAccessRefreshToken> rsp = client.toBlocking().exchange(request, BearerAccessRefreshToken)
assert rsp.status == HttpStatus.OK
return rsp.body().accessToken
}
private Map httpPostCallMap(String uri, Map body, String accessToken) {
Flowable<HttpResponse<Map>> call = client.exchange(
POST(uri, body)
.header(HttpHeaders.AUTHORIZATION, "Bearer $accessToken")
, Map.class
)
return call.blockingFirst().body()
}
Login in using curl:
> curl -i -v "http://localhost:8080/api/login" -H 'Content-Type: application/json' -d $'{"username":"TEST-services","password":"buy-me-a-pony"}'
...
{"username":"TEST-services","roles":["admin"],"access_token":"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJURVNULXNlcnZpY2VzIiwibmJmIjoxNTY5MjEyMDMxLCJyb2xlcyI6WyJhZG1pbiJdLCJpc3MiOiJtYXBwZXItbW4iLCJleHAiOjE1NjkyMTU2MzEsImlhdCI6MTU2OTIxMjAzMX0.ctLAxA0Jsb_HfKY7M3JaUSwscPDb2iBGfz-TsjE7XQk","refresh_token":"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJURVNULXNlcnZpY2VzIiwibmJmIjoxNTY5MjEyMDMxLCJyb2xlcyI6WyJhZG1pbiJdLCJpc3MiOiJtYXBwZXItbW4iLCJpYXQiOjE1NjkyMTIwMzF9.9xP_JmTFG120M_fbiAKwTOE7YTjTpxwK3tOtO0UMKaM","token_type":"Bearer","expires_in":3600}
Once you have the JWToken you pass it in the Authorization header as Authorization: Bearer eyJhblahblahblah…
when you
make a request.
The above token will time out after 3600 seconds, you can refresh the token using the refresh token. The refresh token is not changed, so save it in a safe place. Have a look in ApiControllerSpec "test auth" test to see how to refresh your token.
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Put("/add-host")
Add a new Host to the mapper. You need to be authenticated to call this.
PUT request to /add-host with JSON body containing hostname.
{
"hostName": "mcneils.net"
}
-
Deprecated
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Put("/add-identifier{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?uri}")
Add a new identifier to the mapper with optional uri. A default URI is made if none is supplied. The Version Number is optional.
Sending HTTP Request: PUT /api/add-identifier?nameSpace=electronics&idNumber=555&objectType=timer Authorization: Bearer eyJh... content-type: application/json content-length: 2 Request Body {}
Response: content-type: text/json Response Body { "identifier": { "id": 21, "nameSpace": "electronics", "objectType": "timer", "idNumber": 555, "deleted": false, "updatedAt": 1569212311999, "updatedBy": "TEST-services" }, "uri": "timer/electronics/555" }
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Put("/add/{objectType}/{nameSpace}/{idNumber}")
Add a non versioned Identifier with the default URI, or supply the uri in the body
Sending HTTP Request: PUT /api/add/timer/electronics/555 Authorization: Bearer eyJh.... content-type: application/json Request Body {"uri":"dual-timer/556"}
Status Code: 200 OK content-type: text/json Response Body { "identifier": { "id": 21, "nameSpace": "electronics", "objectType": "timer", "idNumber": 555, "deleted": false, "updatedAt": 1569212311999, "updatedBy": "TEST-services" }, "uri": "timer/electronics/555" }
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Put("/add/{nameSpace}/{objectType}/{versionNumber}/{idNumber}")
Add a versioned Identifier with default URI, or specify the uri in the JSON body.
Sending HTTP Request: PUT /api/add/apni/treeElement/222/111 Authorization: Bearer eyJhbG... content-type: application/json Request Body {}
Status Code: 200 OK content-type: text/json Response Body { "identifier": { "id": 25, "nameSpace": "apni", "objectType": "treeElement", "idNumber": 111, "versionNumber": 222, "deleted": false, "updatedAt": 1569212312153, "updatedBy": "TEST-services" }, "uri": "treeElement/222/111" }
5.5. PUT /add-uri-to-identifier{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?uri}{?preferred}
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Put("/add-uri-to-identifier{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?uri}{?preferred}")
Adds a new or existing uri to an existing Identifier as specified by the query parameters.
Sending HTTP Request: PUT /api/add-uri-to-identifier?nameSpace=apni&idNumber=54433&uri=54433%2Fapni%2Fname&objectType=name Authorization: Bearer eyJhbG… content-type: application/json Request Body
{}
Status Code: 200 OK content-type: text/json Response Body
{ "success": true, "message": "uri added to identity", "match": { "id": 72109, "uri": "54433/apni/name", "deprecated": false, "updatedAt": 1569212332076, "updatedBy": "TEST-services" }, "identifier": { "id": 10, "nameSpace": "apni", "objectType": "name", "idNumber": 54433, "deleted": false, "updatedAt": 1568002341950, "updatedBy": "pmcneil" } }
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Post("/bulk-add-identifiers")
Adds multiple identifiers and the preferred URI. The body of the post is a JSON object containing a list of identifier objects that look like this:
{
"s": "apni", //nameSpace
"o": "treeElement", //object type
"i": 51215341, //id number
"v": 51313427, //version number
"u": "tree/51313427/51215341" //uri
}
Sending HTTP Request: POST /api/bulk-add-identifiers Authorization: Bearer eyJhbG... content-type: application/json content-length: 3099457 Request Body {"identifiers":[{"s":"apni","o":"treeElement","i":51215341,"v":51313427,"u":"tree/51313427/51215341"},...]}
Status Code: 200 OK content-type: text/json Response Body {"success":true,"message":"36040 identities added."}
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Post("/bulk-remove-identifiers")
Removes multiple Identifiers as specified in the JSON body. The body of the post is a JSON object containing a list of identifier objects that look like this:
{
"s": "apni", //nameSpace
"o": "treeElement", //object type
"i": 51215341, //id number
"v": 51313427 //version number
}
Sending HTTP Request: POST /api/bulk-remove-identifiers Authorization: Bearer eyJhbG... content-type: application/json content-length: 3099457 Request Body {"identifiers":[{"s":"apni","o":"treeElement","i":51215341,"v":51313427,"u":"tree/51313427/51215341"},...]}
Status Code: 200 OK content-type: text/json Response Body {"success":true,"message":"36040 identities removed."}
-
PermitAll
-
Produces(MediaType.TEXT_JSON)
-
Get("/current-identity{?uri}")
Gets the current identity associated with this URL
Sending HTTP Request: GET /api/current-identity?uri=http%3A%2F%2Flocalhost%3A8080%2Fname%2Fapni%2F54433 Status Code: 200 OK content-type: text/json Response Body [ { "id": 10, "nameSpace": "apni", "objectType": "name", "idNumber": 54433, "versionNumber": 0, "deleted": false, "updatedAt": 1568002341950, "updatedBy": "pmcneil" } ]
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Delete("/delete-identifier{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?reason}")
Sending HTTP Request: DELETE /api/delete-identifier?reason=just+for+kicks&nameSpace=animals&idNumber=1&objectType=rat Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJURVNULXNlcnZpY2VzIiwibmJmIjoxNTY5Mjg2ODE4LCJyb2xlcyI6WyJhZG1pbiJdLCJpc3MiOiJtYXBwZXItbW4iLCJleHAiOjE1NjkyOTA0MTgsImlhdCI6MTU2OTI4NjgxOH0.WrfFn8K6xBuC6fh0maY8CxySJShKlv4rscXHdMzi9bo content-type: application/json Request Body {}
Status Code: 200 OK content-type: text/json Response Body {"success":true,"message":"Deleted Identifier.","identifier":{"id":72118,"nameSpace":"animals","objectType":"rat","idNumber":1,"deleted":true,"reasonDeleted":"just for kicks","updatedAt":1569286818762,"updatedBy":"fred"}}
-
PermitAll
-
Produces(MediaType.TEXT_JSON)
-
Get("/links/{objectType}/{nameSpace}/{idNumber}")
Sending HTTP Request: GET /api/links/name/apni/54433 Authorization: Bearer null
Status Code: 200 OK content-type: text/json Response Body [ { "link": "http://localhost:8080/name/apni/54433", "resourceCount": 1, "preferred": true, "deprecated": false, "deleted": false }, { "link": "http://localhost:8080/cgi-bin/apni?taxon_id=230687", "resourceCount": 1, "preferred": false, "deprecated": true, "deleted": false } ]
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Post("/move-identity")
Sending HTTP Request: POST /api/move-identity Authorization: Bearer eyJhbG... content-type: application/json Request Body { "fromNameSpace": "animals", "fromObjectType": "dog", "fromIdNumber": 24, "toNameSpace": "animals", "toObjectType": "dog", "toIdNumber": 23 }
Status Code: 200 OK content-type: text/json Response Body { "success": true, "message": "Identities moved.", "from": { "id": 72113, "nameSpace": "animals", "objectType": "dog", "idNumber": 24, "deleted": false, "updatedAt": 1569286818517, "updatedBy": "TEST-services" }, "to": { "id": 72111, "nameSpace": "animals", "objectType": "dog", "idNumber": 23, "deleted": false, "updatedAt": 1569286818487, "updatedBy": "TEST-services" } }
-
Produces(MediaType.TEXT_JSON)
-
Get("/preferred-host")
Sending HTTP Request: GET /api/preferred-host Authorization: Bearer null
Status Code: 200 OK content-type: text/json Response Body {"host":"http://localhost:8080"}
-
PermitAll
-
Produces(MediaType.TEXT_JSON)
-
Get("/preferred-link/{objectType}/{nameSpace}/{idNumber}")
Sending HTTP Request: GET /api/preferred-link/name/apni/54433 Authorization: Bearer null
Status Code: 200 OK content-type: text/json Response Body {"link":"http://localhost:8080/name/apni/54433"}
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Delete("/remove-identifier-from-uri{?objectType}{?nameSpace}{?idNumber}{?versionNumber}{?uri}")
Sending HTTP Request: DELETE /api/remove-identifier-from-uri?nameSpace=animals&idNumber=1&uri=doggies%2F1&objectType=dog Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJURVNULXNlcnZpY2VzIiwibmJmIjoxNTY5Mjg2ODE4LCJyb2xlcyI6WyJhZG1pbiJdLCJpc3MiOiJtYXBwZXItbW4iLCJleHAiOjE1NjkyOTA0MTgsImlhdCI6MTU2OTI4NjgxOH0.WrfFn8K6xBuC6fh0maY8CxySJShKlv4rscXHdMzi9bo content-type: application/json Request Body {}
Status Code: 200 OK content-type: text/json Response Body { "success": true, "message": "Identifier removed from URI.", "identifier": { "id": 72115, "nameSpace": "animals", "objectType": "dog", "idNumber": 1, "deleted": false, "updatedAt": 1569286818633, "updatedBy": "fred" } }
-
RolesAllowed('admin')
-
Produces(MediaType.TEXT_JSON)
-
Put("/set-preferred-host")
Sending HTTP Request: PUT /api/set-preferred-host Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJURVNULXNlcnZpY2VzIiwibmJmIjoxNTY5Mjg2Nzk4LCJyb2xlcyI6WyJhZG1pbiJdLCJpc3MiOiJtYXBwZXItbW4iLCJleHAiOjE1NjkyOTAzOTgsImlhdCI6MTU2OTI4Njc5OH0.qhoVUrEIO2Gx_BaOnOaHKlOZ2WemLG7Tufbt14n8RUU content-type: application/json Request Body {"hostName":"mcneils.net"}
Status Code: 200 OK content-type: text/json Response Body { "host": { "id": 28, "hostName": "mcneils.net", "preferred": true } }
-
PermitAll
-
Produces(MediaType.TEXT_JSON)
-
Get("/stats")
{ "identifiers": 17546853, "matches": 19594817, "hosts": 4, "orphanMatch": 612, "orphanIdentifier": 0 }