Proposal: Support for Cross Browser Testing

[brian-mann]

UPDATES: February 7th, 2020

📦 We have added support for Firefox and MS Edge in Cypress v4.0.0.

• 📣 Read Announcement
• 📘 Read Cross Browser Testing Guide
• 📋 Changelog
• 🛠 Migration Guide

UPDATES: Oct 20th, 2018

I gave more updates on the work we've been doing for cross browser support and other features on a webinar we gave.

https://youtu.be/FfqD1ExUGlw?t=2079

UPDATES: Sept 8th, 2018

Cross browser testing is now a front and center major focus here at Cypress. We've been experimenting over the last few months and have a working implementation to add native event support for Chrome, Firefox, Edge, and IE11. We also believe adding Mobile Safari and Android should also be straightforward - with Safari actually being the most difficult to get.

The architecture by which we are doing cross browser testing is a best-of-breed design where we tap into the native API's for Chrome and Firefox (avoiding the use of webdriver) and for Edge and IE11 we only use a minimal set of Webdriver API's - none of which are the traditional sendKeys or value endpoints, which avoids the network latency problem entirely. We plan to eventually publish this architecture so people understand how Cypress is different than Webdriver. None of the other core components of the Cypress architecture are affected by this change, it is simply how to enable native events in a consistent manner across all browsers without introducing the latency / flaky / non-determinism of Webdriver.

In additional to all these things, we are also adding Sauce Labs support (at a minimum) and will also have a proof of concept prepared for that.

As we add support for native events we are subsequently adding support for the browsers at the same time to ensure all of our algorithms are normalized correctly. However, you'll likely first see us release native event support in Chrome + Firefox, followed by IE11 sometime after that.

...below is the original post from 2016...

Does Cypress support cross browser testing?

Currently the answer is: No

Right now, we only support Chrome* variants such as Chrome, Chromium, and Canary.

Will you ever support other browsers?

Yes.

We will be able to support other Desktop Browsers and Mobile Browsers (using Webviews).

We have built Cypress from day 1 with cross browser testing in mind, and are very conscientious to ensure the API's we write today will work in the future.

The Technical Details

Why do you only support Chrome today?

• Cypress taps into the underlying browser Automation API's of Chrome to do things like modify cookies, take screenshots, focusing the browser window, etc.
• We also launch the browser using very specific command line flags to turn off things that get in the way of automated testing.
• We have synchronized these Automation API's when Cypress runs headlessly (in Electron). This means running headlessly behaves identically to a real headed Chrome browser.

What do you mean by "Automation API's"?

• Automation API's describe the functions that a browser exposes to do things outside of the Javascript Sandbox.
• Currently Cypress comes with its own universal Driver which implements most of the cy commands using Javascript - in the same run loop as your application. This is superior in the sense that Cypress can understand, know, wait for and modify its behavior based on how your application reacts to its commands in real time.
• The downside is its impossible to fully automate a browser entirely in Javascript. Javascript is a security sandbox which restricts what you can do.
• To fully automate a browser you must extend beyond the sandbox and tap into the underlying Automation API's which do not have these restrictions.

What will it take to support cross browsers?

• We have to port commands which need to extend outside of the sandbox to each other browser we are attempting to support.

How will you port these Automation API's?

• Either manually by hand
• Or by using the existing browser drivers

What do you mean by "existing browser drivers"?

• Each browser has built its own driver to the Webdriver Spec.
• Here are some examples of drivers:
• Google Chrome Driver (ChromeDriver)
• Micosoft Edge Driver
• SafariDriver
• Mozilla GeckoDriver (Firefox)
• Internet Explorer Driver
• Appium

Why would you port these by hand?

• Because using the existing drivers comes with a host of problems.

What's wrong with the existing browser drivers?

• They are built for and thus limited to the Webdriver spec.

What is the Webdriver spec?

• It's a HTTP API spec which defines a common interface to drive all browsers consistently.
• It enables developers to abstract away the details of how each different browser implements its Automation API's

What is the problem with Webdriver?

• It's slow
• It's async
• It's stateless
• It's exposed over an HTTP server
• It's a lowest common denominator
• It's limited in scope and features
• There are small inconsistencies
• There are various driver bugs

What do you mean by "lowest common denominator"?

• The spec is essentially limited to only including "all the things that every browser can do".
• This leaves a tremendous amount of power and capability off the table that more modern browsers provide.

Can you give me some examples?

• In Webdriver you cannot clear all cookies for all domains - you can only clear cookies for the current domain context under test.
• There is no way to tell the browser to clear its cache.
• Chromedriver cannot run tests with Dev Tools open.
• Oftentimes the drivers may miss elements you're trying to interact with.
• You cannot easily control the application under test.
• You don't have native access to window, you're limited to stringifying objects over the wire.

Does Cypress currently use Webdriver?

• No. We wrote our own universal driver.

What would be the benefit of using Webdriver?

• We could possibly use the existing drivers to port over various commands without having to write the manual integrations by hand.
• This would allow us to support cross browsers much faster.

If you use Webdriver won't Cypress also inherit these limitations?

• In most cases - no.
• In some cases - possibly.

How can you avoid these limitations?

• By using our universal driver to do as much automation as possible and only go outside of the browser for a limited set of commands.
• The whole point of Cypress is to exceed the boundaries of the current testing ecosystem.
• Because we automate the browser with a completely different architecture we are in a position to extend beyond the Webdriver Spec and also normalize its behavior consistently across all browsers.

Can you give me an example of this?

• Cypress can clear all cookies of all domains instead of just the current domain under test.
• We perform work outside of the browser but then synchronize it seamlessly like with cy.request.
• We have many commands which work outside of the browser which don't need to be ported.

But won't you still be limited by Webdrivers flakiness?

• No. Even if we were to issue native events through Webdriver, because your application is always tested through Cypress, that means Cypress can respond synchronously to events that take place, and modify its behavior accordingly.
• The vast majority of commands do not ever have to leave the browser anyway.

Can you give me an example of what you're talking about?

• Imagine this scenario: you have a modal opening which animates in, and you want to click the "Submit" button.
• In Webdriver, because its an HTTP API which often involves multiple network hops, there is usually a significant delay.
• Let's assume you're attempting to issue a native "click" to the submit button.
• When you issue the "click" command, the browser driver calculates the center coordinates of the button at say: {clientX: 50, clientY: 100}.
• It then issues the native click event with those coordinates.
• By the time your application "receives" this event, it could possibly "miss" the button altogether.
• Your test fails because your button was never clicked, and the modal was never dismissed.
• What happened?
• Because the modal was animating it, its coordinates were changing rapidly. Webdriver has no idea or concept of this, it only takes a coordinate snapshot at the moment in time and because it asynchronously dispatches the native event, it misses.

Why isn't this a problem with Cypress?

• Because Cypress is in the same run loop as your application, its impossible for it to "miss".
• It can synchrously run calculations, and it already automatically figures out if an element is animating beyond a certain threshold - and therefore wait until it stops.
• If it did issue a native event, it could detect that this event has yet to happen, and if the element moved from the coordinate, immediately attempt to recover by either canceling the click event or issuing another click at new coordinates.

When do you think you'll support other browsers?

• We really don't know, its not a priority for us right now.

Why isn't it a priority? This is super important!

• We disagree, cross browser testing is not as important as you think it is.

...

The reasoning and examples behind this are a WIP. We'll be updating this doc when they are ready.

[glambert]

Hey @brian-mann!

First, amazing job on Cypress. I've been researching the best tool out there for JavaScript e2e testing and Cypress is up there with TestCafé for me. The only thing really that prevents me to go full on with Cypress is the cross-browser testing capabilities.

Looking at the above comment, I understand that Cypress does not consider cross-browser testing to be important. Is there anything anywhere that can convince me that I should be thinking the same way? Various OS/Browser combinations can yield different results, and I still think the value of automated cross-browser e2e testing is valuable.

In any case, this is really the only thing that prevents me from choosing Cypress as my weapon of choice for e2e testing, and I wanted to have you updated opinion on this since this issue was opened about 6 months ago.

Is there a way we can talk about this through DM? I'd really love to discuss this with you or any Cypress team member.

Thanks in advance and keep up the amazing work!

[glambert]

For those interested in knowing how the Cypress team/devs feel about this, head over to there Gitter chat and search for "cross-browser". @brian-mann took the time to give me a lot of details on this, and he also told me he would update this issue to provide more context / examples / more compelling argument.

[shcallaway]

@brian-mann

anyway we will be adding other browsers
firefox and edge

Could you provide any updates on support for these browsers? Is it in development? Also, a few people mentioned support in CI for other browsers via Sauce Labs and/or Browserstack. What's the status there?

Love Cypress! 😃

[tonypee]

Seconded, an update on this would be great! thanks

[joegaffey]

Cypress looks fantastic but without cross browser support I'll have to stick with Selenium.

Firefox then Edge would be my preferred priority.

Any possibility for iOS Safari support?

[vvo]

We disagree, cross browser testing is not as important as you think it is.

I am also interested in having more details about this. I don't think unit testing for cross browsers is important. But for e2e I always thought that yes this was important.

[brian-mann]

We are planning on writing up a full blog post on what we think e2e testing should look like, and how cross browser fits into that. It's going to require some pretty graphics ;-)

[danieldram]

This is an amazing solution, but without IE, FF and Safari support, I can't get buy in from my office to use this :(

[joelgrimberg]

great stuff you guys are doing with Cypress!
is connectivity with SauceLabs/CrossBrowserTesting/Browserstack also considered?

[nrxsmith]

Do you guys have an update on your progress with this? Any ETA?

Thanks!

[jennifer-shehane]

Support for Firefox browser proposal: #1096

[green-arrow]

@brian-mann - I’m also very interested in a blog post about the rationale behind cross-browser testing not being as important. I read a few comments in the gitter channel about how modern browsers largely resolve a lot of cross-browser bugs, but I just don’t buy it. There are still huge inconsistencies between browsers with modern features like flex box. Chrome, Firefox, and Safari each have their own quirks with just this one feature that our current cross-browser test suite catch. Even more seemingly benign issues like z-index creating stacking contexts differently in every browser can cause the application to break for a user.

A blog post that addresses some of this would definitely be welcome.

[brian-mann]

Yeah unfortunately, we never wrote it. I haven't necessarily changed my position, which is that repeating the same identical test suite across a variety of browsers is not the way to go - but regardless we are adding cross browser support right now. For users wanting utter and complete coverage, you'll have that ability.

There is a law of diminishing returns here where each successive browser only marginally increases your confidence - and requires substantially more infrastructure and cost. It may be valuable in some cases to some users, but overall we run millions of tests, and only a small fraction of those failures occur in other browsers which would not have been caught by automated functional tests in chrome alone. I believe that's the key here - functional tests will only ever catch a specific class of bug that is mostly logic related, and oftentimes not necessarily visual. We have actually seen situations where most, if not all tests pass even when all of the stylesheets fail to load. One one hand, that's the power and flexibility of Cypress, on the other hand, functional tests give you limited confidence in visual artifacts.

The bigger solution and really the only one that really solves cross browser issues for good (while providing a good balance of adding confidence without diminishing returns) is visual regression testing (screenshot diffing). We are also adding support for that as well. With that feature, I would then recommend full testing in chrome, but only marginal happy path testing in the rest of the browsers with screenshot diffing. This balances what is oftentimes the primary root cause of cross browser testing (visual issues), while still covering the functional aspects, but needing primarily only in one browser.

It might go something like this assuming you have 300 tests.

Browser	# Tests	Screenshot Diffing	Responsive Viewport Testing
Chrome	300	Yes	Yes
Firefox	30	Yes	No
Edge	30	Yes	No
Safari	30	Yes	No

Then when regressions or edge cases arise (meaning bugs found in production) in other various that the 30 happy path tests didn't fix - you analyze those and expand the test suite in those other browsers that may be affected. In fact - even without support for cross browsers today you can still do this by recreating the environment by which it happened. This is also what we see our users do and it works quite well. Once you understand the root cause of a problem you simply generate a contrived test to ensure it works.

[NoNameProvided]

Hi @brian-mann!

No rush or anything, but do you have maybe a date to share? We are starting a new project soon and would love to use Cypress for testing, but support for Firefox/Safari is a requirement for us.

[toshovski]

Thank you @jennifer-shehane for sharing the link. It kind of answered my questions. Unfortunately there is no timeline about releasing the multi browser support, but it looks like you have a lot to do before the release.

[serkanyersen]

@jennifer-shehane Are you going to wait for all browsers to be ready or will you release as soon as one browser is ready and keep adding browsers on the way?

[jennifer-shehane]

@serkanyersen There is some minimal viable work that has to be done to account for all browsers first, but no, when we feel another browser is ready, we will release that.

[adamchenwei]

We disagree, cross browser testing is not as important as you think it is.

I am also interested in having more details about this. I don't think unit testing for cross browsers is important. But for e2e I always thought that yes this was important.

I am super curious about this justification as well, good job so far though, Cypress! Looking forward for the final hockey stick on adoption once this ticket is closed! And ... before that, I am just like anyone else, eye balling on what ppl think about this topic out there...
https://www.quora.com/unanswered/Why-cross-browser-testing-is-not-important

[willischu]

We disagree, cross browser testing is not as important as you think it is.

I am also interested in having more details about this. I don't think unit testing for cross browsers is important. But for e2e I always thought that yes this was important.

Agreed that unit testing for cross browsers is unnecessary! Having a small smoke test suite specifically for testing across different browsers would be ideal.

[irfancharania]

@brian-mann: wrt the comment around "WebDriver API lowest common denominator".

The spec is essentially limited to only including "all the things that every browser can do".

How does Cypress get around this as you add support for other browsers?
If a browser doesn't have a specific feature, how do you address that?

Would you end up in a scenario like iMacros where each command lists browsers it will work with?

[jpike88]

IE11 differs hugely from other browsers and occasionally breaks in weird ways that we can't forsee. It would be definitely worth supporting it.

[irfancharania]

IE11 differs hugely from other browsers and occasionally breaks in weird ways that we can't forsee. It would be definitely worth supporting it.

They demo'd a working proof of concept here

[kunwardeep]

Another ray of hope https://www.windowscentral.com/microsoft-building-chromium-powered-web-browser-windows-10

[UncleGus]

YAY! Microsoft are abandoning Edge!

But BOO, they're building YET ANOTHER BROWSER.

Is there any way we can make them just stop?

[stramel]

Is there anyway we could leverage the Puppeteer API (https://github.com/GoogleChrome/puppeteer) to handle the cross-browser portion?

They currently support Chrome and Firefox. With Edge moving to a Chromium base, I imagine when it lands it will work, nearly out of the box.

[divyaarjnani]

Does Cypress supports firefox and IE, got confused by reading the comments.

[highvoltag3]

@divyaarjnani it does not currently, it is planned for the near-ish future (there's no set date or ETA). This issue has all the updates at the top the comments are made by anyone and has opinions, questions, suggestions and whatever so refer to cypress docs for supported functionality or follow this if you'd like to keep an eye on how things develop/evolve.

[nguyenj]

Any plans on integrating it with saucelabs/browserstack/or a similar service? I know this has been asked before; but I can't seem to have any responses that directly answers this question.

Or is it considered that the cypress service is a direct competitor to those services (saucelabs/browserstack)–if so then I understand why the question has been avoided.

[kellyprankin]

Can anyone elaborate on this statement:

Why isn't it a priority? This is super important!

We disagree, cross browser testing is not as important as you think it is.

If I want to convince people to adopt this tool, I feel like they are going to scoff at that statement. Can you give any ammo?

[sbasinger]

I'm in the same boat as @kellyprankin

We have a large IE user base, and that's where we have the most rendering issues.

[jennifer-shehane]

@kellyprankin @sbasinger The comment on the importance of cross browser testing is an older comment. Our new stance:

Cross browser testing is now a front and center major focus here at Cypress.

We're working on support. It is a major undertaking and dependent on implementing Native Events first, which is making progress every day.

Also, see the list of tasks required for cross browser support here including PRs: #3207

[adamchenwei]

Still having a reallllly hard time to sell Cypress to my team and company. The giantic piece of ugly hole is missing support of other browsers, I mean, if its IE, we can work with it, but not even firefox and safari is a bit tough... really looking forward to a fully browser supported version ... and when that did come, would be nice to come with a selenium test to cypress test convertor ha.

[Gpx]

Hi everyone, sorry for spamming. I follow this issue because I want to know when there are updates on cross-browser support. I believe many others do the same.

Instead, I receive notifications about people saying that they need Cypress to work on this or that browser or asking when will support land.

Can I please ask the maintainers to lock comments for outside members so that only relevant information gets shared? If that's not possible it's ok; I'll find another way to keep up-to-date with Cypress.

Sorry again for the noise and thanks to the maintainers for their hard work on Cypress!

[jennifer-shehane]

We are actively working on releasing Firefox support. Currently the release is blocked by a garbage collection issue that causes Firefox to crash during some tests during cypress run. We're working with the Firefox team and also trying several creative workarounds to prevent the crashing. #1359

We are also working on releasing support for Microsoft Edge as part of this PR. #6181

They are currently targeted for the same release.

[amirrustam]

📦 We have released support for Firefox and MS Edge in Cypress v4.0.0:

• 📣 Read Announcement
• 📘 Read Cross Browser Testing Guide
• 📋 Changelog
• 🛠 Migration Guide

[amirrustam]

Now that Cypress supports Firefox in addition to Chrome-family browsers (including Edge and Electron), this issue is being closed in favor of dedicated proposal issues for each browser under consideration.

• macOS Safari Support: Safari (WebKit) Support #6422
• IE 11 Support: Proposal: IE 11 Support #6423

Please direct further commentary and feedback within these dedicated issues.