Add chromatic visual tests #2505

tmeasday · 2017-12-18T06:22:30Z

Issue: We didn't have visual regression tests for our stories

Note: this is built on #2504

What I did

Added chromatic to our CI

How to test

Make a UI change, check out the build at https://chromaticqa.com

NOTE: this is not currently accessible to members of the storybooks org, just sorting out org permissions.

Anyone in the storybooks org on GH should be able to access the project here: https://www.chromaticqa.com/builds?appId=5a375b97f4b14f0020b0cda3

tmeasday · 2017-12-18T06:22:58Z

@ndelangen do you have the permission to grant access from the storybooks GH org?

Hypnosphi · 2017-12-18T12:07:18Z

Note that there's an ongoing work on adding visual testing functionality to storyshots: #2413

tmeasday · 2017-12-18T12:49:17Z

@Hypnosphi

Note that there's an ongoing work on adding visual testing functionality to storyshots

Yes! I think this is great! I think however for a larger project like this one Chromatic can offer a little more on top of that, namely parallelized / cloud screenshots and an review flow UI. Interested to hear your thoughts once you've given it a go.

igor-dv · 2017-12-18T12:55:10Z

package.json

-    "repo-dirty-check": "node ./scripts/repo-dirty-check"
+    "repo-dirty-check": "node ./scripts/repo-dirty-check",
+    "storybook": "start-storybook -p 6006",
+    "chromatic": "chromatic test --storybook-addon --app-code sm3026v658i"


If it's not a public key, probably better hide it with the CI parameter

Yeah, see the note on the commit ;)

Hypnosphi · 2017-12-18T22:27:30Z

Oh, that's actually pretty cool. I think there's a value in adding this for all the cra-kitchen-sink stories

tmeasday · 2017-12-19T03:57:50Z

I think there's a value in adding this for all the cra-kitchen-sink stories

Yes, I'll get this going once we sort out #2504

ndelangen · 2017-12-22T00:41:17Z

Please rebase, so i can better judge this PR, looks all right.

Different storybook app layers install the `start-storybook` command on top of each other. We won't leave the storybook here, though.

tmeasday · 2017-12-22T01:23:53Z

@ndelangen done.

Hypnosphi · 2017-12-22T08:43:05Z

.circleci/config.yml

+      - run:
+          name: "Visually test storybook"
+          command: |
+            cd examples/official-storybook; yarn chromatic


Please add a root-level shortcut script for that

Maybe it should be even made an option in scripts/test.js

Is this something we want to be able to run locally?

I believe so, at least to update the refs

I was under the impression chromatic provided a review system to decide if a change is OK or NOT_OK ?

If the tests are run by each dev, how does chromatic overcome the rendering differences per system?

AFAIU they actually run in cloud even when launched locally

chromatic provided a review system to decide if a change is OK or NOT_OK

That's cool that they do that! And I feel convenient doing that even before submitting a PR

Yeah, you can run it locally and it does take the snapshots in the cloud still.

The only hiccup with this is the CHROMATIC_APP_CODE environment variable -- it will need to be set for the test to run locally. Any thoughts on how to make that easier for devs (you can find it in the settings page at https://www.chromaticqa.com/settings?appId=5a375b97f4b14f0020b0cda3)

@Hypnosphi -- not sure how to best add it to the scripts/test.js command; it seems the way that is written a command either needs to be a jest project or a npm test command for some subdirectory. I'm not sure we want to make it npm test for examples/official-storybook though.

Should I change the script to support a full custom command? Perhaps in a separate PR?

# Conflicts: # yarn.lock

codecov · 2017-12-26T22:58:27Z

Codecov Report

Merging #2505 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2505   +/-   ##
=======================================
  Coverage   32.68%   32.68%           
=======================================
  Files         398      398           
  Lines        8838     8838           
  Branches      953      949    -4     
=======================================
  Hits         2889     2889           
+ Misses       5313     5294   -19     
- Partials      636      655   +19

Impacted Files	Coverage Δ
...-native/src/preview/components/OnDeviceUI/index.js	`0% <0%> (ø)`	⬆️
lib/ui/src/modules/ui/libs/hierarchy.js	`50.74% <0%> (ø)`	⬆️
addons/actions/src/lib/retrocycle.js	`54.54% <0%> (ø)`	⬆️
addons/actions/src/lib/reviver.js	`58.33% <0%> (ø)`	⬆️
addons/knobs/src/components/PropField.js	`10.63% <0%> (ø)`	⬆️
lib/codemod/src/transforms/update-addon-info.js	`50% <0%> (ø)`	⬆️
addons/a11y/src/components/Report/Rules.js	`0% <0%> (ø)`	⬆️
...codemod/src/transforms/update-organisation-name.js	`40.62% <0%> (ø)`	⬆️
addons/knobs/src/components/types/Boolean.js	`11.62% <0%> (ø)`	⬆️
addons/actions/src/lib/util/getPropertiesList.js	`60% <0%> (ø)`	⬆️
... and 48 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a13f39b...e521ade. Read the comment docs.

tmeasday · 2017-12-27T00:27:04Z

This is ready to merge when the 3.3 release process is complete.

Hypnosphi · 2017-12-27T00:29:06Z

How does it affect release process? It doesn't change any external packages

tmeasday · 2017-12-27T00:32:10Z

It shouldn't, but @shilman asked me to hold off merging into master until the release is out...

ndelangen · 2017-12-27T22:59:46Z

Merge when ready, if you ask me

tmeasday requested review from shilman, ndelangen and Hypnosphi December 18, 2017 06:22

igor-dv reviewed Dec 18, 2017

View reviewed changes

tmeasday force-pushed the tmeasday/chromatic branch 2 times, most recently from f71e104 to cacfcf5 Compare December 20, 2017 04:34

tmeasday mentioned this pull request Dec 21, 2017

Add a storybook for the ui package #2504

Merged

tmeasday added 5 commits December 22, 2017 12:15

Added chromatic for main storybook

b6aadd0

Run chromatic tests from CI

b2a5544

Need to add the job to the build

a390a84

Temp: start storybook directly

a5f34c7

Different storybook app layers install the `start-storybook` command on top of each other. We won't leave the storybook here, though.

Move some things around a little

5f65f50

tmeasday force-pushed the tmeasday/chromatic branch from cacfcf5 to 5f65f50 Compare December 22, 2017 01:17

tmeasday assigned ndelangen Dec 22, 2017

Hypnosphi reviewed Dec 22, 2017

View reviewed changes

ndelangen added 2 commits December 22, 2017 13:58

Merge branch 'release/3.3' into tmeasday/chromatic

b6a0677

Merge branch 'release/3.3' into tmeasday/chromatic

b9f3e69

ndelangen changed the base branch from release/3.3 to master December 23, 2017 22:56

ndelangen approved these changes Dec 24, 2017

View reviewed changes

ndelangen added 2 commits December 24, 2017 13:00

Merge branch 'master' into tmeasday/chromatic

a165e6a

# Conflicts: # yarn.lock

UPDATE lockfile

44510cb

storybookjs deleted a comment from codecov bot Dec 24, 2017

Merge remote-tracking branch 'origin/master' into tmeasday/chromatic

19b419b

tmeasday and others added 3 commits December 27, 2017 09:59

Added top-level chromatic command

deb0517

Don't fail the test if there are changes

376527f

Merge branch 'master' into tmeasday/chromatic

dda6905

tmeasday added the ready label Dec 27, 2017

Merge branch 'master' into tmeasday/chromatic

0230c86

Merge branch 'master' into tmeasday/chromatic

98eba63

Hypnosphi added the maintenance User-facing maintenance tasks label Dec 27, 2017

Merge branch 'master' into tmeasday/chromatic

5c04892

Merge branch 'master' into tmeasday/chromatic

e521ade

Hypnosphi merged commit 7e4e6b8 into master Dec 27, 2017

Hypnosphi deleted the tmeasday/chromatic branch December 27, 2017 23:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add chromatic visual tests #2505

Add chromatic visual tests #2505

tmeasday commented Dec 18, 2017 •

edited

Loading

tmeasday commented Dec 18, 2017

Hypnosphi commented Dec 18, 2017

tmeasday commented Dec 18, 2017 •

edited

Loading

igor-dv Dec 18, 2017

tmeasday Dec 18, 2017

igor-dv Dec 18, 2017

Hypnosphi commented Dec 18, 2017

tmeasday commented Dec 19, 2017

ndelangen commented Dec 22, 2017

tmeasday commented Dec 22, 2017

Hypnosphi Dec 22, 2017

Hypnosphi Dec 22, 2017

ndelangen Dec 22, 2017

Hypnosphi Dec 22, 2017

ndelangen Dec 22, 2017

ndelangen Dec 22, 2017

Hypnosphi Dec 22, 2017

Hypnosphi Dec 22, 2017

tmeasday Dec 22, 2017

tmeasday Dec 26, 2017

codecov bot commented Dec 26, 2017 •

edited

Loading

tmeasday commented Dec 27, 2017

Hypnosphi commented Dec 27, 2017 •

edited

Loading

tmeasday commented Dec 27, 2017

ndelangen commented Dec 27, 2017

Add chromatic visual tests #2505

Add chromatic visual tests #2505

Conversation

tmeasday commented Dec 18, 2017 • edited Loading

What I did

How to test

tmeasday commented Dec 18, 2017

Hypnosphi commented Dec 18, 2017

tmeasday commented Dec 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hypnosphi commented Dec 18, 2017

tmeasday commented Dec 19, 2017

ndelangen commented Dec 22, 2017

tmeasday commented Dec 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 26, 2017 • edited Loading

Codecov Report

tmeasday commented Dec 27, 2017

Hypnosphi commented Dec 27, 2017 • edited Loading

tmeasday commented Dec 27, 2017

ndelangen commented Dec 27, 2017

tmeasday commented Dec 18, 2017 •

edited

Loading

tmeasday commented Dec 18, 2017 •

edited

Loading

codecov bot commented Dec 26, 2017 •

edited

Loading

Hypnosphi commented Dec 27, 2017 •

edited

Loading