-
Notifications
You must be signed in to change notification settings - Fork 117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add pytest-based acceptance and style tests #58
Conversation
@shivaram and @broxtronix - I'm curious if from your experience with spark-ec2 and spark-gce you have any thoughts on how to test Flintrock. |
Is there a high level description of what these tests are doing ? I never found a good way to test spark-ec2, but I do see some mock implementation of boto (like https://github.com/spulec/moto) that might be useful to get a realistic test -- @nchammas Have you had any experience using any of these ? |
@shivaram - At a high level these tests just invoke Flintrock's various commands like a user would and confirm that the return codes and other output are as expected. Each test's name describes what it's testing for. For example: def test_try_launching_duplicate_cluster(running_cluster):
p = subprocess.run([
'./flintrock', 'launch', running_cluster],
stderr=subprocess.PIPE)
assert p.returncode == 1
assert p.stderr.startswith(b"Cluster already exists: ") Given an already running cluster, this tests that Flintrock exits with a code of 1 and a specific message if it tries to launch a new cluster with the same name. Other tests confirm that HDFS and Spark are functioning, or that files can be uploaded to the cluster, and so on.
I've heard of Moto and similar libraries which mock out AWS, but I'm skeptical of their utility with Flintrock. As far as I can tell, they have 2 main problems:
|
Using a real EC2 instance sounds good to me. There are scenarios where say you want to test if SSDs are coming up right or if all of the disks are getting formatted that'll be hard to do with t2.small. But to capture the most essentially features with t2.small sounds like a good idea. |
I've been using m3.xlarge and sometimes even d2.xlarge instances to test the storage work in #51, but yeah for most things t2.small does the job. Perhaps I can add storage tests that fail if a cluster with ephemeral storage is launched (e.g. d2.xlarge instances) but the storage doesn't come up correctly. With instances that have no ephemeral storage the tests always pass. I'll look into that. |
3b273ba
to
6bdcef4
Compare
I've converted the lint tests from shell to Python, revamped the structure of the pytest-based tests, and added documentation. A few items remain:
|
I think getting these tests to run in parallel in a sensible way via pytest and its xdist plugin will depend on some resolution to this issue. Right now, using |
I've created a proposal to improve how xdist distributes tests to workers. If implemented, it would give us a sane way to run Flintrock's tests in parallel using pytest. That'll be a while in the making, though, so I'm gonna merge this in since everything else is working. |
4c935a5
to
abc5ad5
Compare
Add pytest-based acceptance and style tests
This is a work in progress.Run
py.test
from Flintrock's root directory and you'll kick off a full set of acceptance tests. This will replaceslint.sh
andtest-integration.sh
.I gather this is not a typical way to test software (e.g. the acceptance tests launch real clusters), but I think it's what makes the most sense for Flintrock.
Any input here would be great.
Fixes #49.