Qualification & Profiling tool: Add automated tests to run inside CSPs.

[viadea]

We need to add automated tests to run Qualification & Profiling tool inside CSPs -- Databricks/EMR/Dataproc, and also CDP at least once per week.
We can rotate the CSPs/CDP envs each time.

The goal is to make sure the tool can be submitted inside those envs using their spark jars.

[zhanga5]

as @tgravescs refined both tools won't require using Spark and SQL any more, is this issue still valid?

[tgravescs]

We still require the Spark jars to be present since we use classes from them so I think this would still be good to do.

[zhanga5]

I see. Any idea about what kind of event logs and where to keep it for these testings?

[tgravescs]

I think for this if we are running inside the CSP we just run some jobs to generate the logs and then read them there. That way no storing of event logs and we regenerate everytime and make sure CSP didn't change anything in formatting that would break us.

As far as what to run we could either run some of the integration tests and then run on the event logs generated from that, let me look into that a bit more.

[tgravescs]

cdp part of this split out into #3360

[tgravescs]

We want to run on event logs generated by running jobs on those. One thought is to start out with just running them on the integration test event logs. Then I think we would want to add in a couple of more specific tests, which perhaps we can add into just the tools test suite area.

[zhanga5]

it's completed for Databricks environment while still in progress for Dataproc. It would be completed soon.

[zhanga5]

test from Dataproc environment is completed