Install newest Chrome browser before running tests #6115

clarmso · 2020-01-08T01:15:50Z

I'm trying to investigate some Chrome 79+ related issues from #6023 by running the existing tests using the latest Chrome browser. We've been lagging in using the latest stable version of Chrome in testing Cypress itself.

I've some thoughts on how to use the latest stable version of Chrome in the testing. This work would enable the Cypress developers to address the compatibilities between Cypress and Chrome in a timely manner.

If it all goes well, we may close #5236. 👍

Closes Use newest Chrome (stable) to run tests #5236

User facing changelog

Add the ability to test the changes against the latest stable version of the Google Chrome browser.

Additional details

The latest stable version of the Chrome is installed prior to running the test runs that uses Chrome. This way, we don't need to update the Docker image indicated in this line right away when a new version of Chrome becomes available.

cypress/circle.yml

Line 36 in 639ba32

- image: cypress/browsers:node12.8.1-chrome78-ff70

Since this Docker image has the repository of Chrome configured, we can just run apt to retrieve the latest version of Chrome.

How has the user experience changed?

There should be no change in the user's experience. The developers can automatically run the tests against the latest stable version of Google Chrome instead of an older one.

PR Tasks

N/A

cypress-bot · 2020-01-08T01:15:53Z

Thanks for the contribution! Below are some guidelines Cypress uses when doing PR reviews.

Please write [WIP] in the title of your Pull Request if your PR is not ready for review - someone will review your PR as soon as the [WIP] is removed.
Please familiarize yourself with the PR Review Checklist and feel free to make updates on your PR based on these guidelines.

PR Review Checklist

If any of the following requirements can't be met, leave a comment in the review selecting 'Request changes', otherwise 'Approve'.

User Experience

The feature/bugfix is self-documenting from within the product.
The change provides the end user with a way to fix their problem (no dead ends).

Functionality

The code works and performs its intended function with the correct logic.
Performance has been factored in (for example, the code cleans up after itself to not cause memory leaks).
The code guards against edge cases and invalid input and has tests to cover it.

Maintainability

The code is readable (too many nested 'if's are a bad sign).
Names used for variables, methods, etc, clearly describe their function.
The code is easy to understood and there are relevant comments explaining.
New algorithms are documented in the code with link(s) to external docs (flowcharts, w3c, chrome, firefox).
There are comments containing link(s) to the addressed issue (in tests and code).

Quality

The change does not reimplement code.
There's not a module from the ecosystem that should be used instead.
There is no redundant or duplicate code.
There are no irrelevant comments left in the code.
Tests are testing the code’s intended functionality in the best way possible.

Internal

The original issue has been tagged with a release in ZenHub.

clarmso · 2020-01-08T06:51:42Z

I am not authorized to run the server-performance-tests and all server-e2e-tests-* on CircleCI. I cannot test these items on my end. Could someone from Cypress help out?

https://app.circleci.com/github/cypress-io/cypress/pipelines/fa6c0831-9abb-43fb-b6b6-ef56a8322ee5/workflows/b2bed0f6-5601-4953-95b0-ec03ae4dcc19

driver-integration-tests-chrome installs and uses Chrome 79 successfully.
https://app.circleci.com/jobs/github/cypress-io/cypress/227689

…ways defined.

clarmso · 2020-01-27T06:01:58Z

Hm...Let me install the latest Chrome browser when it is needed. The installation should occur in the server-e2e-tests-chrome-* and the driver-integration-tests-chrome.

I'm not 100% with the solution because Chrome is (re)installed in every server-e2e-tests-chrome-*. I cannot just cache the .deb file for the Chrome installer because the browser may have other dependencies in the future. "apt-install" is the sure way to install Chrome. In addition, the Chrome installer is fetched from an external server. We really need the connection between CircleCI and the location of the Chrome debian package and its dependencies' to work consistently.

If there's a more efficient solution, please let me or the maintainers know!

clarmso · 2020-01-28T15:55:07Z

circle.yml

@@ -66,6 +74,10 @@ commands:
    steps:
      - attach_workspace:
          at: ~/
+      - when:


I intended to have the newest Chrome not to be installed for the server-e2e-tests-electron-*. install-latest-chrome is still installed for the electron tests. :( I could add a boolean flag to dictate whether the browser is installed or not. Is there a better way?

I wonder if you could update the install-latest-chrome command to exit if the supplied browser is not chrome, maybe something like:

if [[ << parameters.browser >> = 'chrome' ]] then echo 'Running Chrome tests, installing latest version...' else echo 'Running in << parameters.browser >>, not updating Chrome...' exit 0 fi

flotwig · 2020-01-28T18:23:40Z

circle.yml

+    steps:
+      - run:
+          command: |
+            apt update ;


apt-get is usually recommended for scripts, but this is probably fine too since we aren't depending on the output

flotwig · 2020-01-28T18:24:05Z

circle.yml

+            apt update ;
+            apt install google-chrome-stable -y ;
+            which google-chrome ;
+            google-chrome --version        


maybe add an echo 'Google Chrome version:' above this to help break it out

…n enum.

…rome79

clarmso · 2020-01-30T16:00:45Z

I've addressed the issues and the suggestions above. The tests driver-integration-tests-chrome installs the newest stable Chrome successfully, but some tests failed (https://app.circleci.com/jobs/github/cypress-io/cypress/239271). Let me investigate if they're Chrome 79 specific. Those failures should not be related to this change in circleci.yml.

~~The electron case as seen in server-e2e-tests-electron-* is not tested yet.~~ server-e2e-tests-electron-* looks good to me: Chrome isn't (re)installed.

clarmso · 2020-01-31T04:42:53Z

The test failure is from packages/driver/integration/commands/actions/type_spec.js with the following error message:

  1) src/cy/commands/actions/type user experience can print table of keys on click:
     TypeError: Cannot convert undefined or null to object
      at Function.keys (<anonymous>)
      at findReactInstance (cypress:///./cypress/support/utils.js:18:20)
      at withMutableReporterState (cypress:///./cypress/support/utils.js:46:24)
      at Context.eval (cypress:///./cypress/integration/commands/actions/type_spec.js:4381:16)

I haven't been able to reproduce this error locally either by running the single test in isolation or all tests from the file. I've Chrome 79.0.3945.130 installed on my computer as well. I don't like intermittent test failures, but let me kick off the tests again through rebasing.

- Add a descriptive name to the install step, so the script doesn't get printed out twice - Let `browser` be any string - we have more browsers on the way 😄

circle.yml

flotwig

Thanks!

jennifer-shehane · 2020-02-05T07:46:45Z

Opened a new issue to revert / change implementation of this due to newly exhibited problems. #6328

Note: this implementation was effectively removed as part of https://github.com/cypress-io/cypress/pull/6293/files#diff-29944324a3cbf9f4bd0162dfe3975d88 in order to just get our changes in.

Install newest Chrome browser when needed

dd72d20

Create command for installing Chrome

40f8a83

clarmso changed the title ~~[WIP] Install newest Chrome browser when needed~~ Install newest Chrome browser before running tests Jan 8, 2020

clarmso and others added 2 commits January 8, 2020 11:49

Merge branch 'develop' into Chrome79

8fa0cf0

Merge branch 'develop' into Chrome79

38d2a57

clarmso changed the title ~~Install newest Chrome browser before running tests~~ [WIP] Install newest Chrome browser before running tests Jan 23, 2020

clarmso and others added 2 commits January 26, 2020 16:24

Install Chrome without any conditions in run-e2e-tests. Browser is al…

b5ee249

…ways defined.

Merge branch 'develop' into Chrome79

b848ad8

clarmso changed the title ~~[WIP] Install newest Chrome browser before running tests~~ Install newest Chrome browser before running tests Jan 26, 2020

Install newest Chrome in e2e tests only if it is needed.

914a979

jennifer-shehane requested review from bahmutov and flotwig January 28, 2020 08:54

clarmso commented Jan 28, 2020

View reviewed changes

flotwig reviewed Jan 28, 2020

View reviewed changes

clarmso and others added 4 commits January 30, 2020 00:12

Add condition to install the latest chrome only for chrome e2e tests

3443f28

Add description to install-latest-chrome. Browser parameter becomes a…

ef870f5

…n enum.

Merge branch 'develop' into Chrome79

334db6e

Merge branch 'Chrome79' of https://github.com/clarmso/cypress into Ch…

1bd1713

…rome79

jennifer-shehane requested a review from flotwig January 31, 2020 08:45

Update circle.yml

a94bcf0

- Add a descriptive name to the install step, so the script doesn't get printed out twice - Let `browser` be any string - we have more browsers on the way 😄

clarmso commented Jan 31, 2020

View reviewed changes

circle.yml Show resolved Hide resolved

flotwig approved these changes Jan 31, 2020

View reviewed changes

flotwig merged commit 81555fc into cypress-io:develop Jan 31, 2020

cypress-bot bot mentioned this pull request Jan 31, 2020

Use newest Chrome (stable) to run tests #5236

Closed

clarmso deleted the Chrome79 branch January 31, 2020 21:26

jennifer-shehane mentioned this pull request Feb 5, 2020

Test against latest stable browser versions (open new PR to do so) #6328

Closed

jennifer-shehane mentioned this pull request Feb 5, 2020

dont automatically install the latest version of chrome in circle #6329

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install newest Chrome browser before running tests #6115

Install newest Chrome browser before running tests #6115

clarmso commented Jan 8, 2020 •

edited

Loading

cypress-bot bot commented Jan 8, 2020

clarmso commented Jan 8, 2020

clarmso commented Jan 27, 2020

clarmso Jan 28, 2020

flotwig Jan 28, 2020

flotwig Jan 28, 2020

flotwig Jan 28, 2020

clarmso commented Jan 30, 2020 •

edited

Loading

clarmso commented Jan 31, 2020

flotwig left a comment

jennifer-shehane commented Feb 5, 2020 •

edited

Loading

Install newest Chrome browser before running tests #6115

Install newest Chrome browser before running tests #6115

Conversation

clarmso commented Jan 8, 2020 • edited Loading

User facing changelog

Additional details

How has the user experience changed?

PR Tasks

cypress-bot bot commented Jan 8, 2020

PR Review Checklist

User Experience

Functionality

Maintainability

Quality

Internal

clarmso commented Jan 8, 2020

clarmso commented Jan 27, 2020

clarmso Jan 28, 2020

Choose a reason for hiding this comment

flotwig Jan 28, 2020

Choose a reason for hiding this comment

flotwig Jan 28, 2020

Choose a reason for hiding this comment

flotwig Jan 28, 2020

Choose a reason for hiding this comment

clarmso commented Jan 30, 2020 • edited Loading

clarmso commented Jan 31, 2020

flotwig left a comment

Choose a reason for hiding this comment

jennifer-shehane commented Feb 5, 2020 • edited Loading

clarmso commented Jan 8, 2020 •

edited

Loading

clarmso commented Jan 30, 2020 •

edited

Loading

jennifer-shehane commented Feb 5, 2020 •

edited

Loading