-
Notifications
You must be signed in to change notification settings - Fork 279
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[documentation]: update Pytests README (#4157)
* [docs]: update README Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com> * [chore]: reposition `.env` in `.gitignore` Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Ekaterina Mekhnetsova <mekkatya@gmail.com> Signed-off-by: 0x009922 <43530070+0x009922@users.noreply.github.com> * Readme: reviewed, polished for clarity Signed-off-by: yamkovoy <b.yamkovoy@gmail.com> * Added Custom Env instructions Signed-off-by: yamkovoy <b.yamkovoy@gmail.com> * [docs]: update pytests README Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com> * README: Minor wording and formatting changes Signed-off-by: yamkovoy <b.yamkovoy@gmail.com> --------- Signed-off-by: Dmitry Balashov <43530070+0x009922@users.noreply.github.com> Signed-off-by: 0x009922 <43530070+0x009922@users.noreply.github.com> Signed-off-by: yamkovoy <b.yamkovoy@gmail.com> Co-authored-by: Ekaterina Mekhnetsova <mekkatya@gmail.com> Co-authored-by: yamkovoy <b.yamkovoy@gmail.com>
- Loading branch information
1 parent
925b556
commit 8d3d1f0
Showing
2 changed files
with
153 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,92 +1,191 @@ | ||
# Overview | ||
|
||
This directory contains the pytest framework with test suites for Iroha 2's Client CLI. | ||
This directory contains the `pytest` framework with test suites for the Iroha 2 Client CLI. | ||
|
||
For quick access to a topic that interests you, select one of the following: | ||
|
||
- [Framework Structure](#framework-structure) | ||
- [Iroha 2 Test Model](#iroha-2-test-model) | ||
- [Using Test Suites](#using-test-suites) | ||
- [Custom Test Environment with Docker Compose](#custom-test-environment-with-docker-compose) | ||
- [Poetry Configuration](#poetry-configuration) | ||
- [Tests Configuration](#tests-configuration) | ||
- [Running Tests](#running-tests) | ||
- [Viewing Test Reports](#viewing-test-reports) | ||
|
||
## Framework Structure | ||
|
||
The framework is organized into the following directories: | ||
|
||
- `common`: Contains common constants and helpers used throughout the framework. | ||
- `models`: Contains the data model classes for accounts, assets, and domains. | ||
- `src`: Contains the source code for the Iroha 2 Client CLI tests, including the `client_cli.py` and related utilities. | ||
- `test`: Contains the test suite for the framework, organized into subdirectories for different test categories (`accounts`, `assets`, `atomicity`, `domains`, and `roles`). | ||
|
||
The framework also includes the following configuration files in its root directory: | ||
|
||
- `poetry.lock` and `pyproject.toml` — configuration files for [Poetry](https://python-poetry.org/), the dependency management and virtual environment tool used in this test framework. | ||
|
||
All tests are written with [Allure Report](https://allurereport.org/) in mind, and therefore require certain configuration prior to being executed.\ | ||
For details, see [Running Tests](#running-tests) and [Viewing Test Reports](#viewing-test-reports). | ||
|
||
## Iroha 2 Test Model | ||
The Iroha 2 Test Model consists of several test categories that cover different aspects of the Iroha 2 blockchain platform. The structure of the test model is as follows: | ||
|
||
- **Configurations:** Test configurations for the Iroha 2 platform. | ||
The Iroha 2 Test Model consists of several test categories that cover different aspects of the Iroha 2 blockchain platform.\ | ||
The test model has the following structure: | ||
|
||
- **Accounts:** Test cases for account-related operations, such as account registration, key management, and metadata manipulation. | ||
- **Accounts**: Test cases for account-related operations. | ||
- **Assets**: Test cases for asset-related operations. | ||
- **Atomicity**: Test cases for transaction atomicity. | ||
- **Domains**: Test cases for domain-related operations. | ||
- **Roles**: Test cases for roles management. | ||
|
||
- **Assets:** Test cases for asset-related operations, including asset creation, minting, burning, transferring, and managing asset definitions and metadata. | ||
<!-- TODO: Add once implemented: - **Configurations**: Test configurations for the Iroha 2 platform. --> | ||
|
||
- **Domains:** Test cases for domain-related operations, such as registering and unregistering domains. | ||
## Using Test Suites | ||
|
||
- **Roles:** Test cases for roles management. | ||
> [!NOTE] | ||
> The following instructions assume that you're using the `test_env.py` script that is being provided for the default test environment. | ||
> However, it is possible to run the tests in a custom environment, e.g., with Docker Compose. | ||
> For instructions on how to do so, see [Custom Test Environment with Docker Compose](#custom-test-environment-with-docker-compose). | ||
- **Atomicity:** Test cases for transaction atomicity, including multiple instructions within a single transaction, paired instructions, and invalid instructions. | ||
1. Set up a test environment using the [`test_env.py`](../../scripts/test_env.py) script: | ||
|
||
## How to use | ||
At first, you need to installed and running [Iroha 2](https://hyperledger.github.io/iroha-2-docs/guide/install.html), and also need to have built [Client CLI](https://hyperledger.github.io/iroha-2-docs/guide/build.html) | ||
```shell | ||
# Must be executed from the repo root: | ||
./scripts/test_env.py setup | ||
``` | ||
|
||
## Configuration | ||
By default, this builds `iroha`, `iroha_client_cli`, and `kagami` binaries, and runs four peers with their API exposed through the `8080`-`8083` ports.\ | ||
This behavior can be reconfigured. You can run `./scripts/test_env.py --help` to see the list of available commands and options. | ||
|
||
To configure the application, you can use a `.env` file in the `client_cli/pytest` directory. The `.env` file should contain the following variables: | ||
2. Install and configure [Poetry](https://python-poetry.org/).\ | ||
For details, see [Poetry Configuration](#poetry-configuration) below. | ||
3. Configure the tests by creating the following `.env` file in _this_ (`<repo root>/client_cli/pytests/`) directory: | ||
|
||
``` | ||
CLIENT_CLI_DIR=/path/to/iroha_client_cli/with/config.json/dir/ | ||
TORII_API_PORT_MIN=8080 | ||
TORII_API_PORT_MAX=8083 | ||
``` | ||
Replace `/path/to/iroha_client_cli/dir` with the actual paths to the respective files on your system. | ||
```shell | ||
CLIENT_CLI_DIR=/path/to/iroha_client_cli/with/config.json/dir/ | ||
TORII_API_PORT_MIN=8080 | ||
TORII_API_PORT_MAX=8083 | ||
``` | ||
|
||
If the `.env` file is not present or these variables are not defined in it | ||
For details, see [Tests Configuration](#tests-configuration) below. | ||
4. Run the tests: | ||
|
||
## Poetry Configuration | ||
```shell | ||
poetry run pytest | ||
``` | ||
|
||
This test framework uses [Poetry](https://python-poetry.org/) for dependency management and virtual environment setup. To get started with Poetry, follow these steps: | ||
5. Once you are done, clean up the test environment: | ||
|
||
1. Install Poetry by following the [official installation guide](https://python-poetry.org/docs/#installation). | ||
```shell | ||
# Must be executed from the repo root: | ||
./scripts/test_env.py cleanup | ||
``` | ||
|
||
2. Navigate to the `client_cli/pytests` directory in your terminal. | ||
### Custom Test Environment with Docker Compose | ||
|
||
By default, we provide the [`test_env.py`](../../scripts/test_env.py) script to set up a test environment. This environment is composed of a running network of Iroha peers and an `iroha_client_cli` configuration to interact with it. | ||
|
||
However, if for any reason this approach is inconvenient, it is possible to set up a custom network of Iroha peers using the provided Docker Compose configurations. | ||
|
||
To do so, perform the following steps: | ||
|
||
1. Have a local or remote server that has a custom Docker Compose development environment already setup: | ||
|
||
```bash | ||
docker-compose -f docker-compose.dev.yml up | ||
``` | ||
|
||
2. Build the `iroha_client_cli` binary: | ||
|
||
```bash | ||
cargo build --bin iroha_client_cli | ||
``` | ||
|
||
3. Create a new directory, then copy the `iroha_client_cli` binary and its `config.json` configuration file into it: | ||
|
||
```shell | ||
# Create a new directory: | ||
mkdir test_client | ||
# Copy the files: | ||
cp configs/client/config.json test_client | ||
cp target/debug/iroha_client_cli test_client | ||
``` | ||
|
||
4. Proceed with _Step 2_ of the [Using Test Suites](#using-test-suites) instructions. | ||
|
||
> [!NOTE] | ||
> Don't forget to specify the path to the directory created for the `iroha_client_cli` binary and its `config.json` configuration file (see Step 3) in the `CLIENT_CLI_DIR` variable of the `.env` file. | ||
> For details, see [Tests Configuration](#tests-configuration) below. | ||
### Poetry Configuration | ||
|
||
This test framework uses [Poetry](https://python-poetry.org/) for dependency management and virtual environment setup. | ||
|
||
To get started with Poetry, follow these steps: | ||
|
||
1. Install Poetry by following the [official installation guide](https://python-poetry.org/docs/#installation). | ||
2. Navigate to the `client_cli/pytests` directory in your terminal. | ||
3. Install the dependencies and set up a virtual environment using Poetry: | ||
```bash | ||
poetry install | ||
``` | ||
|
||
```bash | ||
poetry install | ||
``` | ||
|
||
4. Activate the virtual environment: | ||
```bash | ||
poetry shell | ||
``` | ||
Now, you should be in the virtual environment with all the required dependencies installed. All the subsequent commands (e.g., pytest, allure) should be executed within this virtual environment. | ||
5. When you're done working in the virtual environment, deactivate it by running: | ||
```bash | ||
exit | ||
``` | ||
|
||
## Run tests | ||
```bash | ||
poetry shell | ||
``` | ||
|
||
Now you are in the virtual environment with all the required dependencies installed. All the subsequent commands (e.g., `pytest`, `allure`) must be executed within this virtual environment. | ||
|
||
To run tests and generate a report in the allure-results folder, execute the following command: | ||
Once you're done working with the virtual environment, deactivate it: | ||
```bash | ||
pytest -k "not xfail" --alluredir allure-results | ||
exit | ||
``` | ||
The `--alluredir` option specifies the directory where the report should be stored. | ||
### Tests Configuration | ||
## View the report | ||
Tests are configured via environment variables. These variables can be optionally defined in a `.env` file that must be created in _this_ (`<repo root>/client_cli/pytests/`) directory. | ||
To launch a web server that serves the Allure report generated, run: | ||
The variables: | ||
```bash | ||
allure serve allure-results | ||
- `CLIENT_CLI_DIR` — Specifies a path to a directory containing the `iroha_client_cli` binary and its `config.json` configuration file.\ | ||
Set to `/client_cli`, by default. | ||
- `TORII_API_PORT_MIN`/`TORII_API_PORT_MAX` — This pair specifies the range of local ports through which the Iroha 2 peers are deployed. A randomly selected port from the specified range is used for each test.\ | ||
Set to `8080` and `8083` respectively, by default. | ||
**Example**: | ||
```shell | ||
CLIENT_CLI_DIR=/path/to/iroha_client_cli/with/config.json/dir/ | ||
TORII_API_PORT_MIN=8080 | ||
TORII_API_PORT_MAX=8083 | ||
``` | ||
The `allure-results` argument specifies the directory where the report is stored. After running this command, you should be able to view the report in your web browser by navigating to `http://localhost:port`, where port is the port number displayed in the console output. | ||
## Running Tests | ||
## Structure | ||
The framework is organized into the following directories: | ||
To run tests and generate an [Allure](https://allurereport.org/) report in to the `allure-results` folder, execute the following command: | ||
```bash | ||
pytest -k "not xfail" --alluredir allure-results | ||
``` | ||
`common`: Contains common constants and helpers used throughout the framework. | ||
The `-k` option specifies tests which contain names that match the given string expression (case-insensitive), which can include Python operators that use filenames, class names and function names as variables.\ | ||
The `"not xfail"` value specifies that only tests that are _not_ signed with the [`xfail`](https://docs.pytest.org/en/6.2.x/skipping.html#xfail-mark-test-functions-as-expected-to-fail) marking will be conducted.\ | ||
This is due to the fact that tests with the `xfail` marking are currently Work-in-Progress and expected to fail. | ||
`models`: Contains the data model classes for accounts, assets, and domains. | ||
The `--alluredir` option specifies the directory where the report is stored. | ||
`src`: Contains the source code for the Iroha 2 Client CLI tests, including the client CLI and related utilities. | ||
## Viewing Test Reports | ||
`test`: Contains the test suite for the framework, organized into subdirectories for different test categories (accounts, assets, atomicity, domains, and permissions). | ||
To launch a web server that serves the generated [Allure](https://allurereport.org/) report, execute the following command: | ||
The framework also includes configuration files: | ||
```bash | ||
allure serve allure-results | ||
``` | ||
`poetry.lock` and `pyproject.toml`: Configuration files for Poetry, the dependency management and virtual environment tool used in this framework. | ||
`pytest.ini`: Configuration file for pytest, the testing framework used in this framework. | ||
The `allure-results` argument specifies the directory where the report is stored. After running this command, you will be able to view the report in your web browser by navigating to `http://localhost:port`, where `port` is the port number displayed in the terminal output. |