Skip to content

Commit

Permalink
[documentation]: update Pytests README (#4157)
Browse files Browse the repository at this point in the history 
* [docs]: update README

Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com>

* [chore]: reposition `.env` in `.gitignore`

Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Ekaterina Mekhnetsova <mekkatya@gmail.com>
Signed-off-by: 0x009922 <43530070+0x009922@users.noreply.github.com>

* Readme: reviewed, polished for clarity

Signed-off-by: yamkovoy <b.yamkovoy@gmail.com>

* Added Custom Env instructions

Signed-off-by: yamkovoy <b.yamkovoy@gmail.com>

* [docs]: update pytests README

Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com>

* README: Minor wording and formatting changes

Signed-off-by: yamkovoy <b.yamkovoy@gmail.com>

---------

Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com>
Signed-off-by: 0x009922 <43530070+0x009922@users.noreply.github.com>
Signed-off-by: yamkovoy <b.yamkovoy@gmail.com>
Co-authored-by: Ekaterina Mekhnetsova <mekkatya@gmail.com>
Co-authored-by: yamkovoy <b.yamkovoy@gmail.com>
  • Loading branch information
@0x009922 @outoftardis @yamkovoy
3 people authored Feb 5, 2024
1 parent 925b556 commit 8d3d1f0
Show file tree
Hide file tree
Showing 2 changed files with 153 additions and 53 deletions.
3 changes: 2 additions & 1 deletion client_cli/pytests/.gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ __pycache__/
# Virtual environment
venv/
*.venv
.env

# Pytest cache
.test_cache/
Expand All @@ -34,3 +33,5 @@ venv/
ehthumbs.db
Thumbs.db
/allure-results/

.env
203 changes: 151 additions & 52 deletions client_cli/pytests/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,92 +1,191 @@
# Overview

This directory contains the pytest framework with test suites for Iroha 2's Client CLI.
This directory contains the `pytest` framework with test suites for the Iroha 2 Client CLI.

For quick access to a topic that interests you, select one of the following:

- [Framework Structure](#framework-structure)
- [Iroha 2 Test Model](#iroha-2-test-model)
- [Using Test Suites](#using-test-suites)
- [Custom Test Environment with Docker Compose](#custom-test-environment-with-docker-compose)
- [Poetry Configuration](#poetry-configuration)
- [Tests Configuration](#tests-configuration)
- [Running Tests](#running-tests)
- [Viewing Test Reports](#viewing-test-reports)

## Framework Structure

The framework is organized into the following directories:

- `common`: Contains common constants and helpers used throughout the framework.
- `models`: Contains the data model classes for accounts, assets, and domains.
- `src`: Contains the source code for the Iroha 2 Client CLI tests, including the `client_cli.py` and related utilities.
- `test`: Contains the test suite for the framework, organized into subdirectories for different test categories (`accounts`, `assets`, `atomicity`, `domains`, and `roles`).

The framework also includes the following configuration files in its root directory:

- `poetry.lock` and `pyproject.toml` — configuration files for [Poetry](https://python-poetry.org/), the dependency management and virtual environment tool used in this test framework.

All tests are written with [Allure Report](https://allurereport.org/) in mind, and therefore require certain configuration prior to being executed.\
For details, see [Running Tests](#running-tests) and [Viewing Test Reports](#viewing-test-reports).

## Iroha 2 Test Model
The Iroha 2 Test Model consists of several test categories that cover different aspects of the Iroha 2 blockchain platform. The structure of the test model is as follows:

- **Configurations:** Test configurations for the Iroha 2 platform.
The Iroha 2 Test Model consists of several test categories that cover different aspects of the Iroha 2 blockchain platform.\
The test model has the following structure:

- **Accounts:** Test cases for account-related operations, such as account registration, key management, and metadata manipulation.
- **Accounts**: Test cases for account-related operations.
- **Assets**: Test cases for asset-related operations.
- **Atomicity**: Test cases for transaction atomicity.
- **Domains**: Test cases for domain-related operations.
- **Roles**: Test cases for roles management.

- **Assets:** Test cases for asset-related operations, including asset creation, minting, burning, transferring, and managing asset definitions and metadata.
<!-- TODO: Add once implemented: - **Configurations**: Test configurations for the Iroha 2 platform. -->

- **Domains:** Test cases for domain-related operations, such as registering and unregistering domains.
## Using Test Suites

- **Roles:** Test cases for roles management.
> [!NOTE]
> The following instructions assume that you're using the `test_env.py` script that is being provided for the default test environment.
> However, it is possible to run the tests in a custom environment, e.g., with Docker Compose.
> For instructions on how to do so, see [Custom Test Environment with Docker Compose](#custom-test-environment-with-docker-compose).
- **Atomicity:** Test cases for transaction atomicity, including multiple instructions within a single transaction, paired instructions, and invalid instructions.
1. Set up a test environment using the [`test_env.py`](../../scripts/test_env.py) script:

## How to use
At first, you need to installed and running [Iroha 2](https://hyperledger.github.io/iroha-2-docs/guide/install.html), and also need to have built [Client CLI](https://hyperledger.github.io/iroha-2-docs/guide/build.html)
```shell
# Must be executed from the repo root:
./scripts/test_env.py setup
```

## Configuration
By default, this builds `iroha`, `iroha_client_cli`, and `kagami` binaries, and runs four peers with their API exposed through the `8080`-`8083` ports.\
This behavior can be reconfigured. You can run `./scripts/test_env.py --help` to see the list of available commands and options.

To configure the application, you can use a `.env` file in the `client_cli/pytest` directory. The `.env` file should contain the following variables:
2. Install and configure [Poetry](https://python-poetry.org/).\
For details, see [Poetry Configuration](#poetry-configuration) below.
3. Configure the tests by creating the following `.env` file in _this_ (`<repo root>/client_cli/pytests/`) directory:

```
CLIENT_CLI_DIR=/path/to/iroha_client_cli/with/config.json/dir/
TORII_API_PORT_MIN=8080
TORII_API_PORT_MAX=8083
```
Replace `/path/to/iroha_client_cli/dir` with the actual paths to the respective files on your system.
```shell
CLIENT_CLI_DIR=/path/to/iroha_client_cli/with/config.json/dir/
TORII_API_PORT_MIN=8080
TORII_API_PORT_MAX=8083
```

If the `.env` file is not present or these variables are not defined in it
For details, see [Tests Configuration](#tests-configuration) below.
4. Run the tests:

## Poetry Configuration
```shell
poetry run pytest
```

This test framework uses [Poetry](https://python-poetry.org/) for dependency management and virtual environment setup. To get started with Poetry, follow these steps:
5. Once you are done, clean up the test environment:

1. Install Poetry by following the [official installation guide](https://python-poetry.org/docs/#installation).
```shell
# Must be executed from the repo root:
./scripts/test_env.py cleanup
```

2. Navigate to the `client_cli/pytests` directory in your terminal.
### Custom Test Environment with Docker Compose

By default, we provide the [`test_env.py`](../../scripts/test_env.py) script to set up a test environment. This environment is composed of a running network of Iroha peers and an `iroha_client_cli` configuration to interact with it.

However, if for any reason this approach is inconvenient, it is possible to set up a custom network of Iroha peers using the provided Docker Compose configurations.

To do so, perform the following steps:

1. Have a local or remote server that has a custom Docker Compose development environment already setup:

```bash
docker-compose -f docker-compose.dev.yml up
```

2. Build the `iroha_client_cli` binary:

```bash
cargo build --bin iroha_client_cli
```

3. Create a new directory, then copy the `iroha_client_cli` binary and its `config.json` configuration file into it:

```shell
# Create a new directory:
mkdir test_client
# Copy the files:
cp configs/client/config.json test_client
cp target/debug/iroha_client_cli test_client
```

4. Proceed with _Step 2_ of the [Using Test Suites](#using-test-suites) instructions.

> [!NOTE]
> Don't forget to specify the path to the directory created for the `iroha_client_cli` binary and its `config.json` configuration file (see Step 3) in the `CLIENT_CLI_DIR` variable of the `.env` file.
> For details, see [Tests Configuration](#tests-configuration) below.
### Poetry Configuration

This test framework uses [Poetry](https://python-poetry.org/) for dependency management and virtual environment setup.

To get started with Poetry, follow these steps:

1. Install Poetry by following the [official installation guide](https://python-poetry.org/docs/#installation).
2. Navigate to the `client_cli/pytests` directory in your terminal.
3. Install the dependencies and set up a virtual environment using Poetry:
```bash
poetry install
```

```bash
poetry install
```

4. Activate the virtual environment:
```bash
poetry shell
```
Now, you should be in the virtual environment with all the required dependencies installed. All the subsequent commands (e.g., pytest, allure) should be executed within this virtual environment.
5. When you're done working in the virtual environment, deactivate it by running:
```bash
exit
```

## Run tests
```bash
poetry shell
```

Now you are in the virtual environment with all the required dependencies installed. All the subsequent commands (e.g., `pytest`, `allure`) must be executed within this virtual environment.

To run tests and generate a report in the allure-results folder, execute the following command:
Once you're done working with the virtual environment, deactivate it:
```bash
pytest -k "not xfail" --alluredir allure-results
exit
```
The `--alluredir` option specifies the directory where the report should be stored.
### Tests Configuration
## View the report
Tests are configured via environment variables. These variables can be optionally defined in a `.env` file that must be created in _this_ (`<repo root>/client_cli/pytests/`) directory.
To launch a web server that serves the Allure report generated, run:
The variables:
```bash
allure serve allure-results
- `CLIENT_CLI_DIR` — Specifies a path to a directory containing the `iroha_client_cli` binary and its `config.json` configuration file.\
Set to `/client_cli`, by default.
- `TORII_API_PORT_MIN`/`TORII_API_PORT_MAX` — This pair specifies the range of local ports through which the Iroha 2 peers are deployed. A randomly selected port from the specified range is used for each test.\
Set to `8080` and `8083` respectively, by default.
**Example**:
```shell
CLIENT_CLI_DIR=/path/to/iroha_client_cli/with/config.json/dir/
TORII_API_PORT_MIN=8080
TORII_API_PORT_MAX=8083
```
The `allure-results` argument specifies the directory where the report is stored. After running this command, you should be able to view the report in your web browser by navigating to `http://localhost:port`, where port is the port number displayed in the console output.
## Running Tests
## Structure
The framework is organized into the following directories:
To run tests and generate an [Allure](https://allurereport.org/) report in to the `allure-results` folder, execute the following command:
```bash
pytest -k "not xfail" --alluredir allure-results
```
`common`: Contains common constants and helpers used throughout the framework.
The `-k` option specifies tests which contain names that match the given string expression (case-insensitive), which can include Python operators that use filenames, class names and function names as variables.\
The `"not xfail"` value specifies that only tests that are _not_ signed with the [`xfail`](https://docs.pytest.org/en/6.2.x/skipping.html#xfail-mark-test-functions-as-expected-to-fail) marking will be conducted.\
This is due to the fact that tests with the `xfail` marking are currently Work-in-Progress and expected to fail.
`models`: Contains the data model classes for accounts, assets, and domains.
The `--alluredir` option specifies the directory where the report is stored.
`src`: Contains the source code for the Iroha 2 Client CLI tests, including the client CLI and related utilities.
## Viewing Test Reports
`test`: Contains the test suite for the framework, organized into subdirectories for different test categories (accounts, assets, atomicity, domains, and permissions).
To launch a web server that serves the generated [Allure](https://allurereport.org/) report, execute the following command:
The framework also includes configuration files:
```bash
allure serve allure-results
```
`poetry.lock` and `pyproject.toml`: Configuration files for Poetry, the dependency management and virtual environment tool used in this framework.
`pytest.ini`: Configuration file for pytest, the testing framework used in this framework.
The `allure-results` argument specifies the directory where the report is stored. After running this command, you will be able to view the report in your web browser by navigating to `http://localhost:port`, where `port` is the port number displayed in the terminal output.

0 comments on commit 8d3d1f0

Please sign in to comment.