diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4bb17a4337..ccb293a808 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -61,3 +61,7 @@ For new features that affect test coverage, use a `coverage/*` branch in this re 2. Ensure that documentation reflects the new changes. 3. Check that CI tests pass before merging a Pull Request. 4. Ensure that your Pull Request has at least one approval before merging. + +## Writing a new test + +See the [Integration Tests CONTRIBUTING.md](./packages/openactive-integration-tests/CONTRIBUTING.md) for guidance on writing a new test. diff --git a/packages/openactive-integration-tests/CONTRIBUTING.md b/packages/openactive-integration-tests/CONTRIBUTING.md new file mode 100644 index 0000000000..22bc71b022 --- /dev/null +++ b/packages/openactive-integration-tests/CONTRIBUTING.md @@ -0,0 +1,271 @@ +# Contributing + +## Architecture + +The testing architecture is documented in [ARCHITECTURE.md](./architecture.md). This is fairly in depth so is only necessary if making changes to the core testing framework. + +## Structure + +The hierarchy for tests is: + +* **Test**: A single test, that tests a given flow e.g. the happy path for a successful free booking. +* **Feature**: A collection of **Test**s that must all pass for a given feature to be considered successfully implemented. Features may be turned on or off in the [`integrationTests` configuration](./README.md#configuration-for-integrationtests-within-confignode_envjson). + + Example features: + + * `free-opportunities`: Booking of free Opportunities + * `customer-requested-cancellation`: Customer-requested Cancellations + * `proposal-amendment`: Ability to amend proposals in the Approval Flow + + A feature contains both **implemented** and **non-implemented** tests: + + * **Implemented** tests are tests that are expected to pass if the feature is turned on. + * **Non-implemented** tests are tests that are expected to pass if the feature is turned off. + + An example non-implemented test is `no-paid-bookable-sessions` within the `non-free-opportunities` feature. This test checks that the Booking System under test does contain any non-free bookable sessions, and therefore can validly skip testing for non-free opportunities. +* **Category**: A collection of **Features**. Categories are only used to group features together for documentation purposes. + + Example categories: + + * `payment`: Payment-related features + * `cancellation`: Cancellation-related features + +Tests exist in a path that includes identifiers for the Category, Feature, and Test; whether it is implemented or non-implemented; and a postfix of `-test.js`, as follows: + +`test/features////-test.js` + +### Example + +The following test: + +```javascript +FeatureHelper.describeFeature(module, { + testCategory: 'payment', + testFeature: 'simple-book-with-payment', + testFeatureImplemented: true, + testIdentifier: 'with-payment-property', +``` + +Will exist in the path `test/features/payment/simple-book-with-payment/implemented/with-payment-property-test.js`. + +### feature.json + +For each feature, there also exists a `feature.json` file, which contains metadata about the feature. This is used by the [Documentation Generator](./ARCHITECTURE.md#documentation-generator) to automatically generate the documentation for that feature. It exists at the following path: + +`test/features///feature.json` + +Here is an example `feature.json` file: + +```json +{ + "category": "payment", + "identifier": "non-free-opportunities", + "name": "Opportunities with a non-zero price", + "description": "The most simple form of booking with payment. Does not check for leases.", + "explainer": "", + "specificationReference": "https://www.openactive.io/open-booking-api/EditorsDraft/#step-by-step-process-description", + "required": false, + "coverageStatus": "complete", + "links": [ + { + "name": ".NET Tutorial", + "href": "https://tutorials.openactive.io/open-booking-sdk/quick-start-guide/storebookingengine/day-5-b-and-delete-order" + } + ], + "requiresOneOfIfImplemented": ["prepayment-optional", "prepayment-required", "prepayment-unavailable"] +} +``` + +## Anatomy of a test + +Most tests are configured using `FeatureHelper.describeFeature(..)`. A simple example usage of this is: + +```js +FeatureHelper.describeFeature(module, { + testCategory: 'core', + testFeature: 'availability-check', + testFeatureImplemented: true, + testIdentifier: 'availability-confirmed', + testName: 'Occupancy in C1 and C2 matches feed', + testDescription: 'Runs C1 and C2 for a known opportunity from the feed, and compares the results to those attained from the feed.', + testOpportunityCriteria: 'TestOpportunityBookable', + controlOpportunityCriteria: 'TestOpportunityBookable', +}, +function (configuration, orderItemCriteriaList, featureIsImplemented, logger) { +``` + +An explanation of the main properties: + +* `testCategory`, `testFeature`, `testIdentifier`: Identifiers for the **Category**, **Feature** and **Test** respectively. For info on what these are, see [Structure](#structure). +* `testFeatureImplemented`: Whether or not this test is **implemented** or **non-implemented**. For info on what this means, see [Structure](#structure). +* `testName`, `testDescription`: Short- and long-form human-readable descriptions of the text. These are used for automatic Documentation Generation and for the test report. +* `testOpportunityCriteria`: The **Opportunity Criteria** to use for the primary OrderItem under test. For info on criteria, see [Opportunity Criteria](#opportunity-criteria). +* `controlOpportunityCriteria`: The **Opportunity Criteria** to use for any "control" OrderItems. For info on criteria, see [Opportunity Criteria](#opportunity-criteria). + + When the test is run with multiple opportunities, a "control" opportunity will be added. With this, one can test that the feature still works if combined with other opportunities. + + For example, if testing an error, set the controlOpportunityCriteria to TestOpportunityBookable to ensure that the correct error is returned even though one of the opportunities in the Order is valid. + + If a control isn't appropriate for a particular test, this can be set to the same value as `testOpportunityCriteria`. + +This is not an exhaustive list of properties which `FeatureHelper.describeFeature(..)` accepts. See the code documentation for the full picture. + +For more information on `FeatureHelper`, see [the ARCHITECTURE doc](./ARCHITECTURE.md#feature-helper). + +### Opportunity Criteria + +The definition of an Opportunity Criteria is documented in [the test-interface-criteria docs](../test-interface-criteria/README.md#criteria). + +Each test is configured to only run for Opportunities which conform to some criteria. This is primarily specified by the `testOpportunityCriteria` property of `FeatureHelper.describeFeature(..)`. + +In this example, the test will only run for Opportunities which are bookable, cancellable, and where the current time falls within the cancellation window: + +```js +FeatureHelper.describeFeature(module, { + testName: 'Successful booking and cancellation within window.', + testDescription: 'A successful end to end booking including cancellation within the cancellation window.', + testOpportunityCriteria: 'TestOpportunityBookableCancellableWithinWindow', +``` + +This is particularly useful for this test, as it checks that these types of Opportunities can be successfully booked and cancelled. If the test were to run for Opportunities which are not cancellable, then the test would fail, but this would not be a valid failure. + +## Writing a new test + +### 1. New feature? + +If creating a new feature, create a new folder within `test/features/` with the name of the feature. Within that folder, create a `feature.json` file as described in [feature.json](#featurejson). + +If this new test is being added to an existing feature, then skip this step. + +### 2. New Test Interface Criteria? + +Each test has a set of Criteria configured (see [Opportunity Criteria](#opportunity-criteria)). Check whether or not this test can use a Criteria that already exists or if a new one is needed. If a new Criteria is needed, be sure to create it in the [Test Interface](https://openactive.io/test-interface/) spec, as well as in the [`test-interface-criteria`](../test-interface-criteria/) package. + +### 3. Create the test + +In most cases, the test should be configured using `FeatureHelper.describeFeature(..)` as described in [Anatomy of a test](#anatomy-of-a-test). Other `FeatureHelper` functions can be used for other cases – see the code documentation and usage of each for more info. + +Here is an example of a simple test, split into annotated parts: + +#### 3.1. Configure the test + +Configure the test using `FeatureHelper.describeFeature(..)`, which helps generate documentation, test reports, and sets up the tests. + +```js +const { expect } = require('chai'); +const { FeatureHelper } = require('../../../../helpers/feature-helper'); +const { FlowStageRecipes, FlowStageUtils } = require('../../helpers/flow-stages'); + +// Configuration for the test, which helps generate documentation, test reports, and sets up the tests. +FeatureHelper.describeFeature(module, { + testCategory: 'payment', + testFeature: 'free-opportunities', + testFeatureImplemented: true, + testIdentifier: 'opportunity-free', + testName: 'Successful booking without payment property', + testDescription: 'A successful end to end booking without the `payment` property included.', + testOpportunityCriteria: 'TestOpportunityBookableFree', + // This must also be TestOpportunityBookableFree as the payment property is only disallowed if ALL items are free. + controlOpportunityCriteria: 'TestOpportunityBookableFree', +}, +(configuration, orderItemCriteriaList, featureIsImplemented, logger) => { +``` + +#### 3.2. Set up Flow Stages + +Flow Stages are explained in more detail in [their docs](./test/helpers/flow-stages/README.md). + +Here, we're using a [FlowStageRecipes recipe](./test/helpers/flow-stages/README.md#flowstagerecipes) to initialise the following [Flow Stage Runnables](./test/helpers/flow-stages/README.md#FlowStageRunnable): + +1. **Fetch Opportunities**: Fetches the Opportunities and Offers (with the given **Criteria**) from the [Broker Microservice](../openactive-broker-microservice/README.md) which are needed to run the test. +2. **C1-and-Assert-Capacity**: Calls C1 for the selected Opportunity-Offer combos. Additionaly, checks, via the Broker Microservice, that the capacity of the Opportunity is as expected after calling C1 (it should be unchanged unless anonymous leasing is supported). +3. **C2-and-Assert-Capacity**: Similar to C1 but for C2. +4. **Book-and-Assert-Capacity**: How this behaves depends on whether Simple Booking Flow or Approval Booking Flow is under test (See: [Test Matrix](./ARCHITECTURE.md#test-matrix)): + * **Simple Booking Flow**: Similarly to C1 and C2, calls B and checks that the capacity is as expected. + * **Approval Booking Flow**: Calls: P -> Order Proposal Feed Update -> B, and checks that the capacity is as expected after each of P and B. + +If you need to test a different flow from this, there are many other Flow Stage Recipes available, and it's always possible to create custom flows from scratch. + +Note that the Flow Stages are just initiated after this code is run. None of the API calls are made yet. + +```js + // ## Initiate Flow Stages + const { fetchOpportunities, c1, c2, bookRecipe } = FlowStageRecipes.initialiseSimpleC1C2BookFlow( + orderItemCriteriaList, + logger, + // Overridable options for the Flow Stages. + { + // Set the request template to 'standardFree' for the B (or P) request + bookReqTemplateRef: 'standardFree' + } + ); +``` + +In this example, we use the `standardFree` Book Request Template in order to ensure that the free booking is successful. Request templates are a useful way to configure the requests at each Flow Stage. For more info, see [Request Templates](./test/templates/README.md). + +#### 3.3 Run Tests + +Now that the Flow Stages have been set up, we use [FlowStageUtils](./test/helpers/flow-stages/README.md#flowstageutils) to create Jest test hooks (including `describe(..)`, etc) for each of the Flow Stages. + +It is in these Jest hooks where the API calls are made, outputs from each Flow Stage input into the next one, and then the results are checked. + +Except for `fetchOpportunities`, each invocation of `FlowStageUtils.describeRunAndCheckIsSuccessfulAndValid(..)` includes an additional callback, in which custom tests are included. Each of these custom tests must be run within `it(..)` hooks. See that `itShouldHavePrepayment(..)` is a function which creates an `it(..)` hook. If you are not familiar with Jest hooks, please see the [Jest docs](https://jestjs.io/docs/setup-teardown). + +```js + // ## Set up tests + FlowStageUtils.describeRunAndCheckIsSuccessfulAndValid(fetchOpportunities); + // This does the following: + // - Runs the C1 FlowStage, which calls C1 in the Booking System under test + FlowStageUtils.describeRunAndCheckIsSuccessfulAndValid(c1, () => { + itShouldHavePrepayment('https://openactive.io/Unavailable', () => c1.getStage('c1').getOutput().httpResponse.body); + }); + FlowStageUtils.describeRunAndCheckIsSuccessfulAndValid(c2, () => { + itShouldHavePrepayment('https://openactive.io/Unavailable', () => c2.getStage('c2').getOutput().httpResponse.body); + }); + FlowStageUtils.describeRunAndCheckIsSuccessfulAndValid(bookRecipe, () => { + if (bookRecipe.isApproval()) { + itShouldHavePrepayment('https://openactive.io/Unavailable', () => bookRecipe.p.getOutput().httpResponse.body); + } + itShouldHavePrepayment('https://openactive.io/Unavailable', () => bookRecipe.b.getOutput().httpResponse.body); + }); +}); + +/** + * @param {string | null} expected + * @param {() => unknown} getOrder + */ +function itShouldHavePrepayment(getOrder) { + it(`should return \`totalPaymentDue.openBookingPrepayment\` \`https://openactive.io/Unavailable\``, () => { + const order = getOrder(); + expect(order).to.have.nested.property('totalPaymentDue.openBookingPrepayment', expected); + }); +} +``` + +##### Error Scenarios + +When testing error scenarios, we would instead use `FlowStageUtils.describeRunAndCheckIsValid(..)` on the [Flow Stage Runnable](./test/helpers/flow-stages/README.md#FlowStageRunnable) which is expected to fail, which forgoes checking that the HTTP response is successful. Here's an example of that (note that this is NOT an extension of the previous example, but a snippet from a different hypothetical test): + +```js +FlowStageUtils.describeRunAndCheckIsValid(c2, () => { + itShouldReturnAnOpenBookingError('IncompleteBrokerDetailsError', 400, () => c2.getOutput().httpResponse); +}); +``` + +This uses a helper function `itShouldReturnAnOpenBookingError(..)` which checks that the HTTP response is an [Open Booking Error](https://openactive.io/open-booking-api/EditorsDraft/#oa-openbookingerror) with the expected `@type` and `statusCode`. + +#### Further Complexity + +The above example is a simple test. But much more complex types of tests are supported by the framework. The best way to learn about these is to look at the existing tests. All the tests are in [`./test/features`](./test/features/). + +## TypeScript + +The code is written in native JS, but uses TypeScript to check for type errors. TypeScript uses JSDoc annotations to determine types (See: [Type Checking JavaScript Files](https://www.typescriptlang.org/docs/handbook/type-checking-javascript-files.html)) from our native .js files. + +Type checking is performed by running: + +```sh +npx tsc +``` + +This is included in the test script so that this check is performed by CI. diff --git a/packages/openactive-integration-tests/DEVELOPMENT.md b/packages/openactive-integration-tests/DEVELOPMENT.md deleted file mode 100644 index b2761b3100..0000000000 --- a/packages/openactive-integration-tests/DEVELOPMENT.md +++ /dev/null @@ -1,102 +0,0 @@ -# Development - -## Architecture - -The testing architecture is documented in [ARCHITECTURE.md](./architecture.md). This is fairly in depth so is only necessary if making changes to the core testing framework. - -## Structure - -Tests exist in a path that includes identifiers for the Category, Feature, and Test, whether it is implemented or non-implemented, and a postfix of `-test.js`, as follows: - -`test/features////-test.js` - -### Example - -The following test: - -```javascript -FeatureHelper.describeFeature(module, { - testCategory: 'payment', - testFeature: 'simple-book-with-payment', - testFeatureImplemented: true, - testIdentifier: 'with-payment-property', -``` - -Will exist in the path `test/features/payment/simple-book-with-payment/implemented/with-payment-property-test.js`. - -## Implemented / non-implemented - -Implemented and non-implemented tests are the test modes, these are configured within the test suite by the user and this defines whether their implementation implements a feature or not. Some features are purely optional, in this case these may simply have tests to simply ensure that the service is not advertising as supporting a feature. - -## FeatureHelper - -This is a class that abstracts away much of the above. This implements the `describe` blocks and initiates the logger. This allows describing the current test, along with the criteria required for it. - -## Approach - -For each feature, implement the following `feature.json`: - -```json -{ - "category": "payment", - "identifier": "simple-book-with-payment", - "name": "Simple Booking of paid opportunities", - "description": "The most simple form of booking with payment. Does not check for leases.", - "explainer": "", - "specificationReference": "https://www.openactive.io/open-booking-api/EditorsDraft/#step-by-step-process-description", - "required": false, - "coverageStatus": "partial", - "links": [ - { - "name": ".NET Tutorial", - "href": "https://tutorials.openactive.io/open-booking-sdk/quick-start-guide/storebookingengine/day-5-b-and-delete-order" - } - ] -} -``` - -Check the [`test-interface-criteria`](../test-interface-criteria/) package includes any Criteria that are needed for a new test, and add any based on the [Test Interface specification](https://openactive.io/test-interface/) as necessary. - - -In each test file within that feature, implement the following (as a simple example): - -```js -FeatureHelper.describeFeature(module, { - testCategory: 'payment', - testFeature: 'prepayment-required', - testFeatureImplemented: true, - testIdentifier: 'opportunity-paid', - testName: 'Successful booking with payment property', - testDescription: 'A successful end to end booking with the `payment` property included.', - // The primary opportunity criteria to use for the primary OrderItem under test - testOpportunityCriteria: 'TestOpportunityBookableNonFreePrepaymentRequired', - // The secondary opportunity criteria to use for multiple OrderItem tests - controlOpportunityCriteria: 'TestOpportunityBookable', -}, (configuration, orderItemCriteria, featureIsImplemented, logger) => { - // # Set-up Flow Stages - const { fetchOpportunities, c1, c2, bookRecipe } = FlowStageRecipes.initialiseSimpleC1C2BookFlow(orderItemCriteriaList, logger); -``` - -For each Stage of the test (e.g. C1, or C2, or ..etc), a `describe(..)` block should be created. The `FlowStageUtils.describeRun..` functions will do this automatically e.g.: - -```js - // # Test flow stages - FlowStageUtils.describeRunAndCheckIsSuccessfulAndValid(fetchOpportunities); - FlowStageUtils.describeRunAndCheckIsSuccessfulAndValid(c1, () => { - it('should return `totalPaymentDue.openBookingPrepayment` "Required"', () => { - expect(c1.getOutput().httpResponse.body).to.have.nested.property('totalPaymentDue.openBookingPrepayment', 'https://openactive.io/Required'); - }); - }); -``` - -## TypeScript - -The code is written in native JS, but uses TypeScript to check for type errors. TypeScript uses JSDoc annotations to determine types (See: [Type Checking JavaScript Files](https://www.typescriptlang.org/docs/handbook/type-checking-javascript-files.html)) from our native .js files. - -Type checking is performed by running: - -```sh -npx tsc -``` - -This is included in the test script so that this check is performed by CI. diff --git a/packages/openactive-integration-tests/test/templates/README.md b/packages/openactive-integration-tests/test/templates/README.md new file mode 100644 index 0000000000..485f417a23 --- /dev/null +++ b/packages/openactive-integration-tests/test/templates/README.md @@ -0,0 +1,3 @@ +# Request Templates + +TODO diff --git a/packages/test-interface-criteria/README.md b/packages/test-interface-criteria/README.md index eee6f75e2e..725c583408 100644 --- a/packages/test-interface-criteria/README.md +++ b/packages/test-interface-criteria/README.md @@ -21,6 +21,44 @@ const TestOpportunityBookableCriteria = TestInterfaceCriteria.criteriaMap.get('T const { matchesCriteria, unmetCriteriaDetails } = TestInterfaceCriteria.testMatch(TestOpportunityBookableCriteria, opportunity); ``` +## Concepts + +### Opportunity-Offer + +An Opportunity-Offer is a combination of an [Opportunity](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity) and one of its [Offers](https://openactive.io/modelling-opportunity-data/#describing-offers-code-schema-offer-code-). This combination is selected by a customer in order to make a booking in real life. Similarly, Test Suite tests select one or more Opportunity-Offers – packaging each into an [OrderItem](https://openactive.io/open-booking-api/EditorsDraft/#schema-orderitem) – before testing a booking flow. + +### Criteria + +A Criteria is a set of Constraints that an **Opportunity-Offer** may or may not satisfy. For example, the `TestOpportunityBookableNotCancellable` criteria is only satisfied for an Opportunity-Offer that can be booked but cannot be cancelled. + +The Criteria are specified in the [Test Interface](https://openactive.io/test-interface) spec. Each of the "Enumeration Values" with type `test:TestOpportunityCriteriaEnumeration` is one of the supported Criteria. + +This package, `test-interface-criteria`, provides, for each of the Criteria specified in the spec, the logic for determining whether or not an Opportunity-Offer satisfies that Criteria. In this package, this logic is split into composable **Constraints**. + +### Constraints + +Each **Criteria** is composed of multiple Opportunity Constraints and Offer Constraints. Each of these is a predicate function that returns true if the Opportunity or Offer satisfies the Constraint, and false otherwise. Here is an example Opportunity Constraint: + +```js +/** +* @type {OpportunityConstraint} +*/ +function endDateMustBeInThePast(opportunity) { + return DateTime.fromISO(opportunity.endDate) < DateTime.now(); +} +``` + +And, an example Offer Constraint: + +```js +/** +* @type {OfferConstraint} +*/ +function mustNotBeOpenBookingInAdvanceUnavailable(offer) { + return offer.openBookingInAdvance !== 'https://openactive.io/Unavailable'; +} +``` + ## Developing ### TypeScript