Discuss relevant details for main purpose of a snapshot #2197
Comments
|
Oh this is great timing. I'm literally working on Jest's plan for the first half of 2017 at Facebook right now and we are very curious about what improvements we can make to snapshot testing. Is this something that you are specifically interested in exploring and potentially working on? Two of the things I'd like to see done are:
|
|
Yeah, putting the ball in play seemed worthwhile. What I can offer now while people are exploring is a viewpoint what seems realistic for an ordinary working dev to set up, maintain, and review. This came up while I am adapting a subset of Whinepad from Stoyan’s book for a potential meetup presentation about Refactoring confidently with checks and tests (Flow and Jest). Let’s keep it as an open possibility that I might learn enough to work with y’all on it in some way. My mascot is a turtle, if you need a mental picture :) |
|
👋 I thought I'd chime in with responses to some of the comments above and other feedback from my experience with snapshot tests.
Thanks! |
|
@davidtheclark Thank you for feedback from your experience. That is what we want.
✋ mee too
Important question: if I gave impression that size itself is the danger, then I miscommunicated.
For some snapshots to include all details as a baseline makes sense to me: I can expect and welcome the frequent diff to review. As someone said, green-red-green is like TDD inside out. But before snapshots, most tests made claims about specific details and intentionally overlooked changes to irrelevant details. If too many snapshots require people to verify relevant changes and at the same time disregard irrelevant changes, then it seems like a half step forward and a step backward. Less traversal logic to write initially, but now people implement it repeatedly. Please don’t mistake me: I am a big fan of snapshots! But see a following comment about cognitive overload. Too many false positive changes, also known as crying wolf, is another way to describe the problem that concerns me. Again, sorry for not making the clearer in the original comment.
Yes, these are wise constraints on potential solutions. Less is more, for sure. If you are already using Enzyme to select what you |
I didn't mean "large" in terms of filesize, but large in terms of the quantity of data in the snapshot. So my question "What is the danger of having snapshots that are too large?" was supposed to be essentially the same as the question "What is the motivation for excluding parts?" Sorry if that was unclear.
If you are making specific assertions about objects instead of checking the whole thing; if you only care about one piece, not the whole — why use snapshots? Might that be time to reset to plain old assertions? |
|
In addition to exploring qualitatively, another leg to walk forward is reasoning quantitatively. Some initial searching found two concepts: Quality inspection task load which refers to an inverted-U-shaped relationship between task demand and performance level, formally known as Yerkes–Dodson law. For the following quote from Quality Inspection Task in Modern Manufacturing analogies are from “noise” to irrelevant changes and from “quality of stored representation of standard” to the mental traversal and filtering to interpret a complete snapshot if it has a limited purpose:
Often associated with learning, cognitive load is explained in a way that applies to this issue in https://www.nngroup.com/articles/minimize-cognitive-load/
Here are the 3 tips from the article:
Depending on the timing, applying research from cognitive psychology and UX design to “diff decision fatigue” to determine a reasonable number of changes to evaluate and a tolerable proportion of false negatives/positives could give a target to achieve, or evaluate the reduction to load, or both. Maybe a win-win for a bright intern to study snapshot testing at Facebook as an A/B test of product teams with and without attention to this issue, as a way to earn an academic credit and have an interesting lightning talk, while providing useful info and having time left over to lend a hand with code :) |
|
I have been using snapshot testing for a few months and I'd like to share my humble opinion. I tend to write stateless functional components and snapshot testing is perfect for it. |
|
Just for reference, it isn't Jest that is missing shallow rendering, it is the |
|
@cpojer I'd love to contribute back to the project, I just wouldn't know where to start for something like this. |
|
See facebook/react#7148 – I'm sure the React team can help you get started. |
|
@raspo Thank you for bringing For people who use Enzyme for its traversal API, how much improvement if Enzyme package could include import { shallow } from 'enzyme';
it('renders the right title', () => {
const wrapper = shallow(<MyComponent className="strong-class"/>);
expect(wrapper.find('h3').toJSON()).toMatchSnapshot();
});Instead of what you write now: import { shallow } from 'enzyme';
import { shallowToJson } from 'enzyme-to-json';
it('renders the right title', () => {
const wrapper = shallow(<MyComponent className="strong-class"/>);
expect(shallowToJson(wrapper.find('h3'))).toMatchSnapshot();
}); |
|
@pedrottimark I personally don't make much use of the traversal API (I can cover most of what I care for with snapshots). With that said, I think that would be a great idea! |
|
Update to #2197 (comment) about shallow snapshots:
|
|
Of the use cases described above, only diff seems highest priority to explore. Given expect(diffSnapshot(prev, next)).toMatchSnapshot();Here is an experiment based on exports[`CheckboxWithLabel changes the text after click (only diff) 1`] = `
"
- checked={false}
+ checked={true}
- Off
+ On
"
`;If you pretend that the red and green GitHub convention is green and red Jest convention, can you interpret the result as a possible mistake related to internationalization of the component? "
- checked={false}
+ checked={true}
-- Off
-+ On
+- éteint
++ allumé
"Or is -- -+ +- ++ from snapshot diff and state change too confusing? I will dig deeper into various data types and render methods. This exploration does not presume that a |
|
Here is another experiment. Does < and > instead of - and + help or hurt your understanding? exports[`CheckboxWithLabel changes the text after click (only diff) 1`] = `
< checked={false}
> checked={true}
< Off
> On
`;If you pretend that the red and green GitHub convention is green and red Jest convention, can you interpret the result as a possible mistake related to internationalization of the component? < checked={false}
> checked={true}
-< Off
-> On
+< éteint
+> alluméP.S. Also experimenting with a snapshot serializer for changes from
|
|
Negative nit: closing bracket is an irrelevant change that can happen once in a while even with differences. For example, the relevant changed -< prop1="value1-prev" />
-> prop1="value1-next" />
+< prop1="value1-prev"
+> prop1="value1-next"Major misunderstanding to avoid: if there is a change in a sequence, for example, of todos: expect(diffSnapshot(prev, next)).toMatchSnapshot();exports[`todo marks a todo as completed and diffs the object 1`] = `
< "completed": false,
> "completed": true,
`;What if the right change occurred in the wrong element? Here is a more specific claim to get the same snapshot: expect(diffSnapshot(prev[index], next[index])).toMatchSnapshot();The same principle applies to adding a todo. It was added, but where? |
|
What strikes me as the main problem with diff-only snapshots, regardless of the annotation style, is that the message lacks context. This is why I would prefer to use entire diffs or focused assertions. For example, let's say you have a component that uses the class |
|
Yes, context! So a diff snapshot displays hints about what changes are expected? exports[`CheckboxWithLabel changes attribute and text after click (diff) 1`] = `
<label>
<input
< checked={false}
> checked={true}
onChange={[Function]}
type="checkbox" />
< Off
> On
</label>
`;To explore that theory, here are 4 examples of test results for relevant versus irrelevant changes: If you change how If you change how <label>
<input
-< checked={false}
-> checked={true}
+ checked={false}
onChange={[Function]}
type="checkbox" />
-< Off
-> On
+ Off
</label>If you change the markup that - <label>
+ <div>
<input
< checked={false}
> checked={true}
+ id="id"
onChange={[Function]}
type="checkbox" />
-< Off
-> On
- </label>
+ <label
+ for="id">
+< Off
+> On
+ </label>
+ </div>If you change the markup that - <label>
+ <div>
<input
-< checked={false}
-> checked={true}
+ checked={false}
onChange={[Function]}
type="checkbox" />
-< Off
-> On
- </label>
+ <label
+ for="id">
+ Off
+ </label>
+ </div>Or is the last example such a mess it’s evidence that focused assertions are more effective? @davidtheclark Thank you for hanging in there for this conversation :) Building step-by-step while responding to comments isn’t stubbornness, but applies advice from Gerald Weinberg to consider at least 3 possibilities before deciding on a change. If this issue reaches a point of diminishing returns and becomes one of the im-possibilities, we have more confidence in whatever we do instead to snapshots. Maybe explaining guidelines and describing patterns more clearly how to use what Jest already has is more helpful than adding features. |
|
To tie up loose ends for the first two use cases: For a better solution to only text as implied by comments, Enzyme has a describe('TodoList', () => {
it('renders text', () => {
const state = [todo0, todo1, todo2];
const onTodoClick = jest.fn();
const wrapper = mount(
<TodoList
onTodoClick={onTodoClick}
todos={state}
/>
);
expect(wrapper.text()).toEqual(state.map(({text}) => text).join(''));
});
});For a better solution to without functions see #2202 (comment) |
|
To tie up the second loose end with a double knot, I just learned to use Enzyme import renderer from 'react-test-renderer';
import {mount, shallow} from 'enzyme';
import enzymeToJSON from 'enzyme-to-json';
test('render with functions', () => {
const onClick = jest.fn();
const { text } = todo0;
const completed = false;
expect(renderer.create(
<Todo onClick={onClick} text={text} completed={completed} />
).toJSON()).toMatchSnapshot();
expect(enzymeToJSON(mount(
<Todo onClick={onClick} text={text} completed={completed} />
))).toMatchSnapshot();
expect(enzymeToJSON(shallow(
<Todo onClick={onClick} text={text} completed={completed} />
))).toMatchSnapshot();
});
// In React element, `style` prop has object value:
exports[`render with functions 1, 2, 3`] = `
<li
onClick={[Function]}
style={
Object {
"textDecoration": "none",
}
}>
Install Jest
</li>
`;import {mount, shallow} from 'enzyme';
import enzymeToJSON from 'enzyme-to-json';
test('render without functions', () => {
const onClick = jest.fn();
const { text } = todo0;
const completed = false;
expect(enzymeToJSON(mount(
<Todo onClick={onClick} text={text} completed={completed} />
).render())).toMatchSnapshot();
expect(enzymeToJSON(shallow(
<Todo onClick={onClick} text={text} completed={completed} />
).render())).toMatchSnapshot();
});
// In HTML element, `style` attribute has string value:
exports[`render without functions 1, 2`] = `
<li
style="text-decoration: none;">
Install Jest
</li>
`;For more information, see http://airbnb.io/enzyme/docs/api/render.html You can omit "snapshotSerializers": ["enzyme-to-json/serializer"]Or if your Jest 17 configuration has: "snapshotSerializers": ["<rootDir>/node_modules/enzyme-to-json/serializer"]For more information, see https://github.com/adriantoine/enzyme-to-json#serializer |
|
Example of diff snapshots in JSX versus HTML markup to test update to table cell. Some details omitted, of course. EDIT: My bad, the describe('Table', () => {
it('updates a number field: Update of CRUD (JSX/HTML diff)', () => {
const recordIndex = 1; // any non-zero index
const fieldIndex = 0; // `when` field is a number
const field = fieldsReceived[fieldIndex];
invariant(field.type === 'number', 'testing a number field');
const store = createStore(reducer);
store.dispatch(receiveData(fieldsReceived, recordsReceived));
const wrapper = mount(
<Provider store={store}>
<Table />
</Provider>
);
const wrapperCell = wrapper.find('tbody tr').at(recordIndex).find('td').at(fieldIndex);
const initialJSX = enzymeToJSON(wrapperCell);
const initialHTML = enzymeToJSON(wrapperCell.render());
wrapperCell.simulate('doubleClick');
const updatingJSX = enzymeToJSON(wrapperCell);
const updatingHTML = enzymeToJSON(wrapperCell.render());
// increment the value in the cell
const updatedJSX = enzymeToJSON(wrapperCell);
const updatedHTML = enzymeToJSON(wrapperCell.render());
expect(diffSnapshot(initial, updating)).toMatchSnapshot(); // either JSX or HTML
expect(diffSnapshot(updating, updated)).toMatchSnapshot(); // either JSX or HTML
});
});exports[`Table updates a number field: Update of CRUD (JSX diff) 1`] = `
<td
data-field-key="when"
data-record-id={1}>
2016
> <form
> onSubmit={[Function]}>
> <input
> autoFocus={true}
> defaultValue={2016}
> type="number" />
> </form>
</td>
`;
exports[`Table updates a number field: Update of CRUD (JSX diff) 2`] = `
<td
data-field-key="when"
data-record-id={1}>
< 2016
< <form
< onSubmit={[Function]}>
< <input
< autoFocus={true}
< defaultValue={2016}
< type="number" />
< </form>
> 2017
</td>
`;exports[`Table updates a number field: Update of CRUD (HTML diff) 1`] = `
<td
data-field-key="when"
data-record-id="1">
2016
> <form>
> <input
> type="number"
> value="2016" />
> </form>
</td>
`;
exports[`Table updates a number field: Update of CRUD (HTML diff) 2`] = `
<td
data-field-key="when"
data-record-id="1">
< 2016
< <form>
< <input
< type="number"
< value="2016" />
< </form>
> 2017
</td>
`; |
|
is it just me or wouldn't a diffing app solve a lot of the problems here? The biggest problem seems to be you're scrolling your terminal. That is, unless you want to commit the updated snapshots, and then compare using standard git diffing tools. The downside of that being that most of the time you're going to want to rollback your repository to the point before you abused it to compare snapshots. Maybe I didn't get the memo of how you're supposed to do this before you commit code?? If there currently is no solution, the path forward seems to be to use Basically a simple jest-specific snapshot diffing application seems like a must for us all. However, if we go down this route, something quite amazing seems to be waiting for us: If we can reach this point--which ultimately isn't too hard--we can build tools into this snapshot comparison application to filter by component just like the React devtools for chrome. And here's the best part: if you're following best practices and managing to build your app almost entirely out of stateless components in combination with Redux, then all you need to do is to snapshot your ENTIRE app in all the states your app will ever be in. The result is you use the diffing application to filter and drill down to components by name, perhaps filtering by only "red" changed components, etc. The need to stub out child components and constantly concern yourself with how deeply/shallow you render goes away. You're no longer concerned about tests passing or failing--you're able to concern yourself with what you see as "red" and "green" in the diffing editor, even if that means you're always essentially "failing." Another key ingredient is that you can see the redux state and actions used to create that component tree right next to your components. Ideally, the system is able to determine and display the relevant state to the component you're currently focused on, as well as the reducer code itself, colored in green and red. And to the right of those reducers, is the action creator code in green and red. The net result is you have a breadcrumb trail leading you back from the components to your reducers to your action creators. For example, you may have found a "red" component in the far left column. Then next to it is its relevant reducer (also "red"), and to the right of that is the relevant action creator's code that triggered that specific state change. However it is green, which means the culprit is the reducer. If however, the reducer was also green, you would know it was changes to the component that triggered the change. Next, you can generate all this with completely automated tests. I.e. You don't need to write any code Here's what you do: you create all the states you need by dispatching all your action creators in all their relevant combinations, and of course you do that in the redux devools and simply export the list of actions. That sets in motion the chain of effects that will result in your entire component tree being rendered in all the states your concerned with. Reducer snapshots are automatically generated along the way, and actions recorded. HERE'S WHERE IT GETS JUICY: From there I see you being able to expand your state tree, and examine your array of actions (i.e. the actions returned from NEXT: we add another notion of time: code commits. So you can drag a slider to navigate through actions dispatched just as you currently can with the redux devtools, but you also can based on commits and see how your component code looks like at a different commit, side by side with the snapshot of the runtime result. From that commit, of course you can "time travel" to actions in its "universe," i.e. all the actions snapshotted in that commit. You're essentially moving in 3D--or whatever you wanna call it--across the history of your snapshots, with lenses for state, actions, component code, reducer code, action creator code, and the ability to filter to just the relevant slices of each as you navigate between associated lenses. Now, take the example above where you simply made changes to one of your components without involving reducers and action creators. Your goal is to determine where the change came from. Keep in mind: you are no longer wasting time diligently unit testing each component shallowly in isolation, so you don't have the benefit of having a specific test point you in its direction. We can compensate for that with the commit time traveling. First you pinpoint a potentially large "red" area in your component tree, and then you switch to git time traveling mode. The snapshot of the component tree will change as you go back in commits, and the associated component code will as well. Basically the big block of red in your snapshot component tree will be reduced to a like 1 or 2 lines of red in code changes. That's what you'll be looking out for, and you'll be able to scan back a few commits and see exactly where the red was first introduced. In general, your workflow goes from what you're doing now, which is looking at pass/fails of tests you spent a lot of time optimizing to give you a relevant but not too noisy perspective of your code to quickly analyzing your snapshots in this "3D" time-traveling differ app. You're now a detective armed with all the tools you need to get to core of the problem--and you do so continually as part of your workflow--rather than perhaps over-confidently relying whether your tests pass. Using snapshots, your tests are going to fail every time. We need to overcome that with a solution built with the new rules of the game in mind. We can do that through a completely automated testing experience that shifts the "work" to an exploratory after the fact experience within this "multi-lens time traveling diffing browser." Lastly, it's worth mentioning that it's redux and a component tree of almost completely pure functions that makes this all possible. And that's the way you should be developing. You don't even need to 100% accomplish it. I still have a few escape hatches and side-effects for various browser issues--I could still very much rely on this system, and independently write tests for side-effects as needed. CONCLUSION: USAGE: SUPER QUICK USAGE using So the conclusion is we're doing too much work, and we do the above for now, but instead of thinking about how to best optimize our "zoom" level of snapshots, we need to take a step back and build a snapshot browser built from the ground-up with these problems in mind. Jest is so good. It's changed the game. These options wouldn't be available to us if it wasn't for Jest, React and Redux. I think we have no choice but to double down and see the whole new world that is available to us and build for that, rather than build for the micro problems of yesterday that will all seize to exist under one fell of something like this. Where do we start? How do we get the electron diffing app up asap? We open source that and get it going, and then we all contribute the various pieces as part of a plugin system. Lets go! |
|
A DISTILLATION OF THE CORE CONCEPTS:
Basically to picture this in your head first imagine the react devtools. Then imagine that instead of (or in addition to) props, in the right column you see the relevant redux state for the focused component. Then imagine redux devtools time traveling built right into that interface, and imagine the snapshots, mapped code, etc, changing as you time travel. Then imagine you can time travel your commits in a similar way. All the while associated lenses are changing to reflect the combination of the 2 forms of time traveling + component filtering. And of course everything is color-coded in green and red to indicate what changes your attention should be drawn to. With all of this you should be armed with all the information you need to find culprit code and fix regressions. |
|
I finally had time to read through this entire issue. Thanks for discussing so feverishly. @faceyspacey: what you are bringing up is very interesting. We specifically didn't do any of this visual integration because we thought that it would be too slow and too annoying for engineers. However, building such a thing on top of Jest would be awesome and I'm sure many people would find it valuable. @pedrottimark: I'm trying to figure out how we can take everything in this issue and turn it into an action plan to improve snapshot testing and make it more useful. I apologize for not fully digesting this issue when I closed your PR recently. So far the thing that sticks out to me the most is that snapshots can often get too big and that the value of the individual snapshots goes down. One thing that @rogeliog came up with; which I guess is more about the narrative than how it actually works, is "snapshots over time". (See https://github.com/facebook/jest/blob/master/packages/jest-cli/src/__tests__/__snapshots__/watch-pattern-mode-test.js.snap ). The idea is that we want to snapshot how state evolves over time. @faceyspacey has discovered this, as per his comment above as well as @xixixao who proposed diffs for action changes over time at FB. It seems like everyone is independently coming to the conclusion that we need some form of diffs not just within one snapshot but also snapshots between two or more snapshots. The next question I'd like to find an answer to is whether this has to be a technical/behavior change in how snapshots work and whether we need new APIs or whether this is mostly a visual change in how Jest is being used – basically whether this is a change in how tests are run or how tests are written or both. Once we have an answer to that, we can figure out how to solve this problem. |
|
This is a very interesting discussion and I thought I'd share a solution I recently came up with for my team. I've added a custom matcher to our jest environment that will snapshot test the shallow-rendered output from a react tree or a component instance. // setupTestFrameworkScriptFile.js
import { Component } from 'react';
import { createRenderer } from 'react-addons-test-utils';
jasmine.addMatchers({
toMatchShallowRenderSnapshot: () => ({
compare(expected) {
let tree;
if (expected instanceof Component) {
tree = expected.render();
} else {
const renderer = createRenderer();
renderer.render(expected);
tree = renderer.getRenderOutput();
}
expect(tree).toMatchSnapshot();
return { pass: true };
},
}),
});An example of what our tests look like: // MyComponent.test.js
import React from 'react';
import { mount } from 'enzyme';
import MyComponent from './MyComponent';
it('should render with class name', () => {
expect(<MyComponent className="test" />).toMatchShallowRenderSnapshot();
});
it('should toggle on', () => {
const wrap = mount(<MyComponent />);
wrap.find('button').first().simulate('click');
expect(wrap.instance()).toMatchShallowRenderSnapshot();
});
it('should toggle off', () => {
const wrap = mount(<MyComponent />);
wrap.find('button').first().simulate('click');
wrap.find('button').first().simulate('click');
expect(wrap.instance()).toMatchShallowRenderSnapshot();
}); |
|
@cpojer I just wanted to point out and let the Jest community know that WallabyJS in the meantime has become the perfect snapshot browser, especially since its creator @ArtemGovorov added a feature I requested to show snapshot diffs inline as you browse your tests: wallabyjs/public#936 (comment) https://twitter.com/wallabyjs/status/819121381002489856
Obviously the plan I laid out above is the basis for a company and a massive undertaking, but with Wallaby today you can achieve the foundation of the strategy. Wallaby is the perfect match for Jest. It's a match made in heaven. The combination has take my workflow to a whole new level. I think the next step for Wallaby is to add an "UPDATE SNAPSHOT" button on all failing snapshot diffs/tests so you can easily update one snapshot at a time (which is a current pain point just with Jest since granular test-matching is time-consuming and since at the very least you almost always end up updating all snapshots in matched files) AND the ability to view snapshots even if they aren't failing. You can already view your snapshots separately by adding them to the list of files Wallaby tracks, but you want to be able to view snapshots on one click of the corresponding test (again, even if it's not failing). If it's failing, you will see the diff on one click, which is the feature added above. TL;DR: if you're using Jest and doing a lot of snapshotting, there's currently no better way to analyze/review/compare them than with Wallaby. |
|
Closing this discussion with thanks to all commenters. So we can keep open issues < 200 :) |
Jest has a challenge and an opportunity with snapshot testing:
https://twitter.com/cpojer/status/774427994077048832
To decide which ways to move forward… (for example, more traversal API: facebook/react#7148 (comment)) …let’s begin with end goals for effective snapshot testing.
Improve developer experience while testing and reviewing code changes:
Hypothesis to discuss: accurate decision = clear purpose + definite expectation + relevant details
Here are 3 use cases for y’all to challenge or confirm.
Examples of snapshots are for the following components:
Todohttp://redux.js.org/docs/basics/ExampleTodoList.html#componentstodojsTodoListhttp://redux.js.org/docs/basics/ExampleTodoList.html#componentstodolistjsLinkhttp://facebook.github.io/jest/docs/tutorial-react.html#snapshot-testingContent: only text
A basic component specification is the text that people see or hear for a realistic variety of props and state. Developers and reviewers can usually expect whether or not a change to code intends to cause a change to text.
Could we balance snapshots that make claims about everything…
…with some snapshots that make claims about only text content?
What do you think about an array of text nodes? Does description tell you which details are included?
Structure: without functions
Depending how a component specifies style, changes to markup can affect how people see content.
If the purpose of a snapshot is to make a claim about structure, then it has irrelevant changes when event handlers
onMouseEnterandonMouseLeaveare added to improve interaction.What do you think about a snapshot without functions? Does description tell you which details are excluded?
Bonus thought: accessibility baseline = content + certain details about structure?
Interaction: only diff
A way to test that a component “works the way it should” is take snapshots for states during an interaction. But the relevant differences are needles in a haystack of irrelevant details. Especially if presence of irrelevant change hides absence of relevant change. For example, a developer might incorrectly update a snapshot after refactoring when a class name didn’t change, but should have.
What do you think about snapshots of only differences between rendered states? Does description tell you which details are included?
This first draft format needs improvement: when there is a change, a diff of a diff doesn’t seem clear.
What next?
Of course, if this issue isn’t helpful, you can close it with no hard feelings :)
Otherwise, let’s imagine, discuss, and then describe an improved developer experience.
Moar use cases are welcome!
The text was updated successfully, but these errors were encountered: