Cavy vs Detox?

[joshuapinter]

Hi Guys,

It would be great if we could get a comparison between Cavy and Detox, maybe even on the README.md, that would highlight Cavy's strengths over Detox and the reason to get behind it.

We're currently using Appium because neither Detox or Cavy were supporting Android when we had to build our tests and now we're looking to change to something more React Native centric to simplify things, but are trying to decide between these two libraries.

Thanks!

Joshua

[AbigailMcP]

Hi @joshuapinter - no problem, I'll look into getting a quick Cavy vs Detox comparison for you.

Thanks for your interest in using Cavy!

[joshuapinter]

Awesome. Thanks for the response!

[AbigailMcP]

Hi @joshuapinter - I've pulled together some thoughts re Cavy vs Detox . It's a bit of a brain dump at the moment, but thought I'd share with you before it goes into the README (or a COMPARISON.md).

If you have any specific points you'd like a comparison for just let me know :)

	Cavy	Detox	Conclusion
Installation and set up	This is a major plus for Cavy - installation and set up is identical for both iOS and Android. The only set up you need to do is wrap your app in Cavy's Tester component.	Installation and set up is different for both iOS and Android - both involve a fair amount of configuration. You also need to choose and set up a third party test runner e.g. Jest. In order to run tests you need to add app-specific configuration to your package.json, specifying by name which devices you want to test on (either simulators or real devices).	Cavy is really lightweight, and is much easier to set up and get running quickly. There is no custom configuration. It 'just works'.
Hooking up components	Hook up components by setting a test reference as its ref. If you're using refs for another purpose already, you can still pass these in as a second argument.	Hook up components by setting a test reference as its testID prop. Not all React components support this prop, so you may need to manually propagate it through to the correct native component.	About the same.
Testing functionality	Cavy is JavaScript only, so doesn't support testing functionality that needs access to native code, for example tapping at an x, y coordinate.	Detox does touch native code and therefore offers a wider range of testing functionality.	Detox has more in-built functionality.

[joshuapinter]

This is great, @AbigailMcP! Our app at work is a mix of native Android and React Native so we might have to go with Detox to get some of those native tests. But I think we'll try each of them and see how they both stack up.

If it's good with you, I'll add any feedback from those run throughs here so it can be added to your chart.

[AbigailMcP]

Thanks @joshuapinter - that's awesome! I'm really interested to hear what snags you come up against for either of them.

[joshuapinter]

If it helps, we just spent the day trying to get Detox working and it was a complete dumpster fire. 🚒

Now we're moving onto Cavy. Hopefully, we have better success.

We're on Android and React Native 0.57.8. Do you foresee any issues we should know about?

Thanks!

[joshuapinter]

So, first issue. I'm a little confused. How do you actually run the spec suite?

Your README says:

Your tests will run automatically when you run your app.

But why would we want it to run everytime we run our app. For example, if we start our app to do development or in production we don't want this running.

Do we need cavy-cli to run the specs?

[joshuapinter]

When I run cavy-cli run-android I get the following error:

cavy: running `react-native run-android`...
events.js:183
      throw er; // Unhandled 'error' event
      ^

Seems similar to pixielabs/cavy-cli#3.

[AbigailMcP]

Yep cavy-cli is the recommended way to run the tests - sorry to hear you hit an issue with this!

We're looking into possible reasons today, but we're finding it tough to replicate on our machines where it's working fine.

Are you running in a windows environment by any chance?

[joshuapinter]

Hey Abigail. Running on macOS 10.13.

Does Cavy launch the emulator and/or application? Or should the emulator and app be running before running cavy-cli?

[AbigailMcP]

Running cavy run-android should launch your app in an emulator (or connected device). There's no need to have either of these running beforehand.

[AbigailMcP]

Are you able to run react-native run-android successfully? Also, what version of node.js are you using?

[joshuapinter]

No, I'm getting:

$ react-native run-android --appFolder cntral
Android project not found. Maybe run react-native android first?

[joshuapinter]

Tell you what. I'll close this issue because it's likely the result of our weird setup. Not your fault that we're not starting from a fresh react native install. If we discover anything, I will try and remember to post the results back here.

Thanks for writing up the comparison table. It's certainly valuable and worthy of adding to a README or at least a Wiki.

And thanks for all your help.

Josh

[AbigailMcP]

No problem! Under the hood cavy-cli uses the react-native-cli, so if you can get react-native run-android to work, then you should be on your way!

[simoreira]

Hi guys,

so, as far as I know, Detox does not support testing on real devices yet
(https://github.com/wix/Detox/blob/master/docs/More.Roadmap.md#ios-physical-device-support,
wix/Detox#95,
https://github.com/wix/Detox/blob/master/docs/More.AndroidSupportStatus.md#emulator-control), so I think that should be fixed on Detox's first column.

Does Cavy offer support for real devices?

[AbigailMcP]

Ah thanks @simoreira ! I'll fix this in the FAQs on cavy.app where we have the same table.

And yes, you can run Cavy tests on a real device - it's just the same as running react-native run-ios (or run-android) and specifying a connected device.