Mapping rules specification

[unleashed]

3scale supports mapping HTTP requests to the concept of methods in a REST-style API (ie. checking HTTP verb, path and query string parameters, although it's not clear whether body and/or trailers could be looked at).

This is a tracking issue for the specification of 3scale's HTTP REST mapping rules.

Tasks:

• This library's implementation (Mapping rules implementation #78).
• Implementation missing bits - there are some fixes and enhancements coming to complete the impl (PR?)
• Description of the specification in the current documentation.
• [NEW] Description of the Porta validation rules.
• Description of what Apicast does.
• Integration of this library's implemenation with Apicast.
• Comparison/differences with what this library's implementation does. See Mapping rules integration and verification with threescalers library 3scale/APIcast#1309 for a comparison.
• Discussion with Apicast/Porta developers.
• Aftermath:
  • Implementation changes?
  • Documentation changes.

The Porta validation rules dictate how the patterns will look like regardless of what the implementations will accept. Those rules are also openly in contradiction with the documentation.

Documentation

Glossary

The entry for mapping rule reads:

1.14.2. mapping rule

Rule that associates incoming calls from specific endpoints to the corresponding methods and metrics created in 3scale. Usage tracking, endpoint access and limits are based on the methods and metrics that are configured with mapping rules. See also Section 1.6.1, “endpoint”, Section 1.14.3, “method”, Section 1.14.4, “metric”.

But there's no content on how to write them and what the expected behavior is.

Admin Portal Guide

The current documentation in the Admin Portal Guide (porta/system) reads:

See 19.4 Adding mapping rules to methods and metrics to learn how to map the endpoints of your API to the methods added here.

Section 19.4.3 and 19.4.4 in the above link read:

3. The Verb field is pre-populated with the HTTP method, GET, however you can select other options from the dropdown list.
4. In the Pattern field, add a valid URL that starts with an forward slash /. The URL can be from a wildcard you specified inside curly brackets {}.

The above is everything there is to how to use these two core parts of mapping rules it in the Admin Portal documentation, with no more detail to learn more other than pointing to the Getting Started Guide. Section 31.5. Troubleshooting techniques when analytics are missing for 3scale APIs mentions that you should check that the mapping rules are correctly mapped to the metrics, but nothing on how to determine that nor any further reference.

Getting Started Guide

Reading the Getting Started Guide we can get to section 1.8 Defining Mapping Rules where it reads:

Specify the following settings:

Verb: The HTTP request verb (GET, POST, DELETE, or PUT).
Pattern: The pattern to match. For example, /hello.
[...]

Again there is no specifics about the pattern, and it would appear the verb is limited to the above 4 options. There's no more detail in this section, but section 2.3.2. Configuring API access policies with application plans shows an example (included only as an image which seems problematic for accessibility) under the subsection Apicast deployment that shows a pattern /{version}/test as well as other patterns /v1 and /v2. There is no more explanation in this subsection at all (:raised_eyebrow:), but the Code plugin deployment subsection contains the description of a few example calls with the settings in the image, for which you can infer that a pattern rule with /{version}/test matches the path /noversion/test, and a literal pattern /v2 matches the path /v2.

There is no other specification and the text even hints at the possibility of a custom implementation that differs in behavior: [...] depending on how you have implemented the mapping for your plugin. The rest of references to mapping rules basically talk about how they interact with the APIs as a Product feature, but still no content that can be used to infer further behavior.

So a user is basically left with no way to know how mapping rules work.

Operating 3scale

This document has references to mapping rules in the context of dealing with a CRUD entity, but again it won't explain how they work. The examples in 5.4.4. Defining backend mapping rules and 5.5.7. Defining product mapping rules only use simple literal patterns.

Subsection 9.1.1.2. APIcast self-managed under 9.1.1. Integration issues links to a "3scale debug header" in the Administering the API Gateway document to troubleshoot mapping rules not matching, but otherwise has no further information.

Administering the API Gateway

This document provides more information about mapping rules than any of the others, though some of the information provided might be better off located in the Admin Portal Guide.

Secion 1.2. How APIcast applies mapping rules for capturing usage of 3scale APIs seems to deal with the information we are looking for.

There is some interesting information about catch-all patterns:

This rule [GET /] means that any GET requests that start with / increment the metric hits by 1. This rule matches any request to your API.

So now we know that GET with a pattern of / matches any request to the API. That seems counterintuitive, as a POST would be matched when using GET in the rule, so it would seem the real meaning is that the pattern part means any GET request would match.

The text goes on to talk about the interaction with APIs as a Product (reproducing content similar to that in the Getting Started Guide) and then the subsection Matching of mapping rules seems to come back to our topic, reading:

3scale applies mapping rules based on prefixes. The notation follows the OpenAPI and ActiveDocs specifications.

There is no link or reference to what part of the OpenAPI specification (OAS) is being followed here, not where to find the ActiveDocs specification, but an example of using the OpenAPI specification is found in the document Providing APIs in the Developer Portal section 4.2. OpenAPI document example: Petstore API. The document also mentions that ActiveDocs is not an specification but an instantiation of OAS (Section 4.1. Setting up 3scale ActiveDocs and OAS), and given other references in the text to ActiveDocs following OAS/Swagger, it seems as if the quote means that looking up the notation described in OAS should be enough.

Reading the example referred to above (quoted just below) and the associated text it would appear as if path parameters are specified by surrounding them curly braces, as in {id}, but this is not explicitly called out. However, it seems to match the few bits of information about curly braces parameters referred to in the Admin Portal Guide. What these parameters match exactly is also not specified in the document itself.

The Petstore API is composed of 4 methods:

GET /api/pets - returns all pets from the system
POST /api/pets - creates a new pet in the store
GET /api/pets/{id} - returns a pet based on a single ID
DELETE /api/pets/{id} - deletes a single pet based on the ID

OpenAPI Specification

Path templating

Looking at the Open API Spec v2.0, which is the version referred to in the text we can find a relevant definition, Path Templating, reading:

Path templating refers to the usage of curly braces ({}) to mark a section of a URL path as replaceable using path parameters.

OAS v3.0 slightly rewords the above:

Path templating refers to the usage of template expressions, delimited by curly braces ({}), to mark a section of a URL path as replaceable using path parameters.

Each template expression in the path MUST correspond to a path parameter that is included in the Path Item itself and/or in each of the Path Item's Operations.

The Paths Object's Patterned Fields looks relevant, with the following description:

A relative path to an individual endpoint. The field name MUST begin with a slash. The path is appended to the basePath in order to construct the full URL. Path templating is allowed.

Further down we can read in Parameter Object:

Path - Used together with Path Templating, where the parameter value is actually part of the operation's URL. This does not include the host or base path of the API. For example, in /items/{itemId}, the path parameter is itemId.

The section for Paths Object in OAS v3 has been extended with examples worth checking out here. The description of Patterned Fields is extended:

A relative path to an individual endpoint. The field name MUST begin with a forward slash (/). The path is appended (no relative URL resolution) to the expanded URL from the Server Object's url field in order to construct the full URL. Path templating is allowed. When matching URLs, concrete (non-templated) paths would be matched before their templated counterparts. Templated paths with the same hierarchy but different templated names MUST NOT exist as they are identical. In case of ambiguous matching, it's up to the tooling to decide which one to use.

The addition about non-templated paths seem to specify a mechanism for determining an order of evaluation, similar (but different) to the order in which 3scale applies mapping rules via rule's position and last parameters.

Additionally, the Path Templating Matching subsection includes examples:

• The path /pets/mine would both be matched by:

/pets/{petId}
/pets/mine

• The following paths are considered identical:

/pets/{petId}
/pets/{name}

• The following paths might lead to ambiguous resolution:

/{entity}/me
/books/{id}

The Path Item object used for evaluating these references the Parameter Object definition, which talks about patterns that by default would not match URI Reserved Characters, that is, :/?#[]@!$&'()*+,;=, though there is a provision to allow just that via an allowReserved setting.

Query string parameters

Query string parameters also accept a pattern field in Parameter Object, validated via JSON schema.

OAS v2 refers to JSON Schema Validation section 5.2.3, pattern reading:

The value of this keyword MUST be a string. This string SHOULD be a valid regular expression, according to the ECMA 262 regular expression dialect.

OAS 3.0 refers to (another) JSON Scheme Validation section 5.8, pattern which read identically but adds:

A string instance is considered valid if the regular expression matches the instance successfully. Recall: regular expressions are not implicitly anchored.

In essence query string parameters are covered, just like path parameters, by the Parameter Object). Like its predecessor, OAS 3.0 allows defining a pattern based on ECMA 262 regular expressions.

Back to Administering the API Gateway

Given the OAS details, one still does not know how to interpret them in the context of 3scale mapping rules. Here there is no explicit word on whether the OAS JSON objects are accepted by 3scale in the mapping rules, or even if query string parameters are accepted at all. Adhering to OAS would use JSON for the mapping rule, which is not the case in the 3scale documentation examples.

Continuing with the subsection Matching of Mapping Rules, we will read on a list of rules towards filling in the patterns:

• A mapping rule must start with a forward slash (/).

This directly follows OAS.

• Perform a match on the path over a literal string, which is a URL, for example, /hello.

Literal patterns should match literal values.

• Mapping rules can include parameters on the query string or in the body, for example, /{word}?value={value}).

This introduces query string parameters but also body parameters, and the example used seems to follow the Path Templating from OAS, which is defined by default to not accept URI Reserved Characters, that is, :/?#[]@!$&'()*+,;=, but that can be changed on a case-by-case basis.

This example is however surprising because it seems to mimic a URI with a query string parameter but does not follow the OAS Parameter Object pattern field definition. Instead the query string parameter appears to use path templating.

The other slightly unrelated surprise is stating that parameters referenced there could be present in the body, which requires inspection and content-type interpretation.

• APIcast fetches the parameters in the following ways:
  • GET method: From the query string.
  • POST, DELETE, or PUT method: From the body.

This ignores the fact that POST, DELETE and PUT can also contain query string parameters even though it is not standardized nor a best practice. Additionally this seems to leave out a lot of other methods on purpose.

• Mapping rules can contain named wildcards, for example, /{word}. This rule matches anything in the placeholder {word}, which makes requests such as /morning match the mapping rule. Wildcards can appear between slashes or between a slash and a dot. Parameters can also include wildcards.

OAS does not specify what the contents of the wildcard/placeholder should be other than they should match the name of a Parameter Object. So in this case word is not well defined, although this seems to imply only word characters are accepted.

This rule then appears to restrict where these wildcards can appear by forcing them to be located in between slashes or a slash and a dot. The verb "can" does not denote obligation, though. The sentence "in between ... a slash or a dot" does not specify in which order, although it might be implying the order should be first a slash then a dot. Ironically the very example used has a wildcard that is not placed in between slashes or in between a slash and a dot.

Lastly, this rule mentions that parameters can also include wildcards, which one can deduce refers to query string parameters, but there is no specification as to whether these wildcards can appear anywhere, in the parameter name, or the value, and whether any restrictions apply, as it seems out-of-place to require them to exist in between slashes or a slash and a dot in the query string.

• By default, all mapping rules are evaluated from first to last, according to the sort order you specified. If you add a rule "/v1", it matches requests whose paths start with "/v1", for example, "/v1/word" or "/v1/sentence".

While this rule looks clear enough, and we know "/v1" should match according to the rule dealing with literal patterns, there's however no example not following /v1 with a forward slash. According to the text, something like /v1beta1 should match.

• You can add a dollar sign ($) to the end of a pattern to specify exact matching. For example, "/v1/word" matches only "/v1/word" requests, and does not match "/v1/word/hello" requests. For exact matching, you must also ensure that the default mapping rule that matches everything (/) has been disabled.

This does not address the interaction with query string parameters: whether a pattern like this /v1?value={value}$ would perform an exact match on the path, on the query string parameter's value, or both. The intuition seems to be that the intended result could be achieved via /v1$?value={value}.

There's also no consideration about what the following pattern would do /v1\$?value={value}, which could be reasonably interpreted as escaping $ and matching /v1$?value=v. Note that $ and other sub-delim characters can appear in path segments.

The last sentence appears to not make sense in this context, but in the context of having a single rule match.

More than one mapping rule can match the request path, but if none matches, the request is discarded with an HTTP 404 status code.

This is behavior related to how to react to no matches, but reproduced here for completing this documentation subsection.

Section 10.3. URL versioning shows another example via an image where there are the following mapping rules: /v2/, /v1/, /{*}/song, /{*}/author. The example implies that the placeholder will match both /v1/song and /v2/song.

There's nothing else in the document discussing the behavior of these patterns.

Porta mapping rule validation (implementation)

Apicast implementation

Threescalers implementation