Skip to content

Writing Functional Tests

Jump to bottom
Bruno Rocha edited this page Oct 19, 2021 · 8 revisions

Functional Tests

What is a functional test?

Functional tests as known as end-to-end tests or integration tests are a way to automate the usage and integration workflows of the application.

Functional tests are written in the same way as unit tests, but they are written to test the whole feature workflow in the perspective of the client and not just one isolated unit of internal functionality. (in other words: functional tests do not have access to the internal application code, it tests the client-side behavior).

The functional test goal is to reproduce the client experience when using the application, so they are focused on interacting only with the components that are exposed to the client via web APIs, web UI, or command-line interfaces.

Those tests are commonly based on a test plan which can be created from the feature user stories or from test plans developed to find bugs in the application.

Examples

User Story

As an admin user, via REST API, I want to create a new namespace.

Test Plans

  • positive test case: expects success when performing the action
  • negative test case: expects failure when performing the action

Positive test case

  1. Generate an access token using admin credentials
  2. Generate valid namespace data
  3. Perform a POST request to create a new namespace
  4. Check that the response of the request is successful
  5. Perform a GET request to retrieve the namespace
  6. Check that the response of the request is successful

Negative test case

  1. Generate an access token using admin credentials
  2. Generate invalid namespace data (bad strings, repeated names, etc.)
  3. Perform a POST request to create a new namespace
  4. Check that the response of the request is unsuccessful

Preparing the testing environment

NOTE This guide assumes that the container based dev environment is in use, most of the instructions here might work on vagrant environment and some are already configured.

Prerequisites to run the functional tests locally

NOTE: If you just want to write functional tests to run on Github Actions CI skip this section, and go to Writing functional tests at the end of this page.

Functional tests on galaxy_ng use the same set of tools used by Pulp, those tools are:

  • Python Unittest framework - Python built-in framework to create test cases in a class based manner
  • Pulp-Smash - A set of tools to configure and execute functional tests on Pulp
  • Pulp3-Bindings - A Python client that points to the Pulp and Galaxy APIs generated from OpenAPIspec.
  • A running Automation HUB server - The hub where the functional tests will be executed. (for this guide we assume there is a container based environment running and galaxy API spec is reachable at http://0.0.0.0:5001/api/automation-hub/v3/openapi.json)

NOTE: Ensure you can access http://0.0.0.0:5001/api/automation-hub/v3/openapi.json before continuing. (the 0.0.0.0 can point to any reachable IP address)

Installing

NOTE: On the Github Actions CI that runs on the repository for every pull request those requirements are already installed.

On your local host machine (or a container/vm if you prefer), you can install the tools by running the following commands:

Python testing packages

cd galaxy_ng  # the root of the repository
# ACTIVATE THE VIRTUALENV YOU USE FOR YOUR LOCAL GALAXY DEVELOPMENT
# ex: `source venv/bin/activate` or `workon galaxy_ng`
pip install -r unittest_requirements.txt
pip install -r functest_requirements.txt

Installing Pulp Bindings

NOTE: httpie is needed (dnf install httpie | yay -S httpie | etc...) for the next steps

Config for httpie

echo "machine 0.0.0.0
login admin
password admin
" >> ~/.netrc

chmod og-rw ~/.netrc

You also need to install the pulp bindings, this is a set of Python objects that points to the Pulp and Galaxy APIs and have to be regenerated every time the OpenAPI spec is changed.

Export some environment variables to install the correct versions matching the running system.

export PULP_URL=http://0.0.0.0:5001  # reachable API address
export GALAXY_VERSION=$(http $PULP_URL/pulp/api/v3/status/ | jq --arg plugin galaxy --arg legacy_plugin galaxy_ng -r '.versions[] | select(.component == $plugin or .component == $legacy_plugin) | .version')
export CORE_VERSION=$(http $PULP_URL/pulp/api/v3/status/ | jq --arg plugin core --arg legacy_plugin core -r '.versions[] | select(.component == $plugin or .component == $legacy_plugin) | .version')
export ANSIBLE_VERSION=$(http $PULP_URL/pulp/api/v3/status/ | jq --arg plugin ansible --arg legacy_plugin ansible -r '.versions[] | select(.component == $plugin or .component == $legacy_plugin) | .version')
export CONTAINER_VERSION=$(http $PULP_URL/pulp/api/v3/status/ | jq --arg plugin container --arg legacy_plugin container -r '.versions[] | select(.component == $plugin or .component == $legacy_plugin) | .version')
echo $CORE_VERSION $ANSIBLE_VERSION $CONTAINER_VERSION $GALAXY_VERSION

The above will output the exported version then we can proceed with the installation.

Install binds for Pulp

pip install pulpcore-client==$CORE_VERSION
pip install pulp-ansible-client==$ANSIBLE_VERSION
pip install pulp-container-client==$CONTAINER_VERSION

NOTE: If pip install fails, you can try to generate each the pulp bindings manually from the source code, that can happen if there is an unreleased version of the Pulp API (ex: when you are running on a main branch)

Generate and install bindings for Galaxy

# On the same folder where galaxy_ng, pulpcore, pulp_ansible, pulp_container is located
# skip if you already have the pulp-openpi-generator repo (ensure the repo is up to date)

git clone --depth=1 https://github.com/pulp/pulp-openapi-generator.git

then

cd pulp-openapi-generator
rm -rf galaxy_ng-client
./generate.sh galaxy_ng python $GALAXY_VERSION
cd galaxy_ng-client
python setup.py sdist bdist_wheel --python-tag py3
find . -name "*.whl" -exec pip install {} \;
cd ../../galaxy_ng

NOTE: Above bindings will need to be regenerated every time the OpenAPI spec is changed.

Configuring the PulpSmash

Save this content on ~/.config/pulp_smash/settings.json

{
  "pulp": {
    "auth": [
      "admin",
      "admin"
    ],
    "selinux enabled": false,
    "version": "3"
  },
  "hosts": [
    {
      "hostname": "0.0.0.0",
      "roles": {
        "api": {
          "port": 5001,
          "scheme": "http",
          "service": "nginx"
        },
        "content": {
          "port": 24816,
          "scheme": "http",
          "service": "pulp_content_app"
        },
        "pulp resource manager": {},
        "pulp workers": {},
        "redis": {},
        "shell": {
          "transport": "docker"
        }
      }
    }
  ]
}

And variables needed by Unittest base class

export PULP_GALAXY_API_PATH_PREFIX=/api/automation-hub/

Executing the functional tests

python -m pytest -v -r sx --color=yes --pyargs galaxy_ng.tests.functional

Writing functional tests

Tests are located at galaxy_ng/tests/functional/cli any file prefixed with test_ will be executed as a test.

The test file must include either function prefixed with test_ or subclass of TestCaseUsingBindings base class.

examples:

As a function (pytest style)

For this kind of test only the general Python utilities are available.

  • os.subprocess.run - allows to execute any command line command
  • requests - allows to perform http request to any URL
def test_something():
    assert 1 == 1

As a class (unittest style) Recommended

As this inherits from TestCaseUsingBindings, it will have access to the bindings and the pre configured clients, so more utilities will be available.

  • from pulp_smash.pulp3.bindings import delete_orphans - allows to delete orphans
  • cls.namespace_api - allows to access the namespace API to perform create and list operations
  • cls.collections_api - allows to access the collections API to perform CRUD operations
  • cls.sync_config_api - allows to access the sync config API to perform sync and remote config
  • cls.smash_client - An http client pre-configured with credentials and token.
  • cls.get_token - A method to request token for the current user
  • cls.update_ansible_cfg - A method to update ansible.cfg with the given config
  • cls.sync_repo - A method to sync a repository from a given requirement_file

That base class can be extended to include more utilities https://github.dev/ansible/galaxy_ng/blob/7119816b00151916ba83d898e4d6b1d02e82ddfe/galaxy_ng/tests/functional/utils.py#L162-L162

More info about the methods available on pulp bindings -> https://hackmd.io/@pulp/bindings

class TestSomething(TestCaseUsingBindings):
    def test_something(self):
        self.assertEqual(1, 1)

Example of a functional test

Test plan

  1. Generate an access token using admin credentials
  2. Generate valid namespace data
  3. Perform a POST request to create a new namespace
  4. Check that the response of the request is successful
  5. Perform a GET request to retrieve the namespace
  6. Check that the response of the request is successful

Functional test

Save this file on: galaxy_ng/tests/functional/cli/test_namespace_create.py

import string
import random

from galaxy_ng.tests.functional.utils import TestCaseUsingBindings
from galaxy_ng.tests.functional.utils import set_up_module as setUpModule  # noqa:F401


class CreateNamespaceTestCase(TestCaseUsingBindings):
    """Test whether a namespace can be created."""

    def test_create_namespace(self):
        # generate name formed by 10 random ascii lowercase letters
        random_name = ''.join(random.choices(string.ascii_lowercase, k=10))
        namespace_data = {"name": random_name, "groups": []}

        # create namespace
        namespace = self.namespace_api.create(namespace=namespace_data)
        self.assertEqual(namespace.name, random_name)

        # ensure namespace is available
        namespaces = self.namespace_api.list(limit=100)
        self.assertIn(namespace.name, [item.name for item in namespaces.data])

        # delete namespace
        # namespace_api does not support delete, so we can use the smash_client directly
        response = self.smash_client.delete(
            f"{self.galaxy_api_prefix}/v3/namespaces/{namespace.name}"
        )
        self.assertEqual(response.status_code, 204)

        # ensure namespace is NO MORE available
        namespaces = self.namespace_api.list(limit=100)
        self.assertNotIn(namespace.name, [item.name for item in namespaces.data])

Execute the test

$ python -m pytest -sv -r sx --color=yes --pyargs galaxy_ng.tests.functional.cli.test_namespace_create
============= test session starts =============
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
collected 1 item 
test_namespace_create.py::CreateNamespaceTestCase::test_create_namespace PASSED

================ 1 passed in 1.30s ===============

NOTE: Tests will perform on the running system and may not be 100% reproducible again if not on a fresh install.

Sponsored by Red Hat, Inc.

Clone this wiki locally