Add end-to-end behavioural tests for the python CLI

[parthosa]

Fixes #970.

This PR includes end-to-end behavioural test for the spark_rapids CLI. A high level directory structure and more details on usage and debugging are provided at docs.

Setup

From the <repo_root>/user_tools directory, run the following command to install the required dependencies:

pip install behave
# or
pip install .[tests]

Running Tests

Tests can be run using 'behave' cmd or using 'tox' cmd.

Basic Usage:

behave <options>
# or
tox -e behave -- <options>

Run All Tests:

behave
# or
tox -e behave

Skip building the Tools jar during setup.

behave -D build_jar=false   # Skip building the Tools jar during setup (default: true)
# or
tox -e behave -- -D build_jar=false

More details are provided in the docs.

Notes

1. Most of the steps are straight forward. Below are the implementation details for some key steps:
   1. "<cli>" is not installed
      • Create a mock CLI cmd and prepend to PATH
   2. Qualification tool JAR crashes
      • Mimic crashing by starting a thread that watches for 'QualificationMain' and kills it
   3. HDFS is "running"
      • Download hadoop binaries and setup a single node HDFS
2. Impact on Github Actions runtime:
   • Previously runtime: ~10min.
   • After including these tests: ~12min
3. Behave Official Docs: https://behave.readthedocs.io/en/stable/tutorial.html

[amahussein]

Thanks @parthosa !

I get a failing test error while running on python 3.8 test_id_ELP_0002

It was expecting Qualification. Raised an error in phase [Execution], but instead the log did not have it.

      2024-08-26 11:00:37,116 INFO rapids.tools.qualification.stats: Merging dataframes to get stats...
      2024-08-26 11:00:37,116 INFO rapids.tools.qualification.stats: Preprocessing dataframes...
      2024-08-26 11:00:37,116 ERROR rapids.tools.qualification: Failed to generate the statistics report: Can only use .str accessor with string values!

[tgravescs]

so I haven't looked at this all yet but I saw we are adding more event logs. I really don't like the idea of checking in more static event logs. I guess if we don't have much options to run a few locally, I'm ok with a couple but I don't want that to grow. The test_event_log_1 and test_event_log_2 should be more descriptive. Also all should be zstd compressed unless we are explicilty testing without it.

[parthosa]

Also all should be zstd compressed unless we are explicilty testing without it

Thanks @tgravescs. Compressed all event logs to zstd format.

The test_event_log_1 and test_event_log_2 should be more descriptive

Renamed to valid_eventlog_1.zstd

[amahussein]

Thanks @parthosa !

The first checkpoint LGTME.
We will discuss offline the steps that follow this PR.

[cindyyuanjiang]

Thanks @parthosa for this feature!

[nartal1]

Thanks @parthosa !

[tgravescs]

so valid_event_log_1 and 2 aren't exactly what I meant. what is in that event log? I saw:

| platform | eventlog              |
| onprem   | valid_eventlog_1.zstd |
| dataproc | valid_eventlog_1.zstd |
      

So why not put onprem, or yarn eventlog, dataproc event log... what code was run in that to generate it?
If I need to generate a new version of these eventlogs how do I do it?

[parthosa]

@tgravescs

So why not put onprem, or yarn eventlog, dataproc event log... what code was run in that to generate it?

• Renamed the event log to join_agg_on_yarn_eventlog.zstd
• It is a simple query that reads/creates 3 DFs of 100k rows with random values and does a join -> aggregation -> join.

[tgravescs]

a few comments but overall looking very good

[amahussein]

Thanks @parthosa
This is going to be a very important feature for the tools.

[nartal1]

Thanks @parthosa!