Addon Test: Add telemetry on discrepancy results #29329

yannbf · 2024-10-11T06:28:00Z

What I did

This PR implements a telemetry event for scenarios where there is a discrepancy between test results, with the following payload:

type Event = {
  type: 'test-discrepancy';
  payload: {
    storyId: StoryId;
    browserStatus: 'PASS' | 'FAIL';
    cliStatus: 'FAIL' | 'PASS';
    message: string;
  };
};

Perhaps in the future there would be other events, such as "global-error" or something like that.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

Add or update documentation reflecting your changes
If you are deprecating/removing a feature, make sure to update
MIGRATION.MD

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts
Make sure this PR contains one of the labels below:
Available labels
- bug: Internal changes that fixes incorrect behavior.
- maintenance: User-facing maintenance tasks.
- dependencies: Upgrading (sometimes downgrading) dependencies.
- build: Internal-facing build tooling & test updates. Will not show up in release changelog.
- cleanup: Minor cleanup style change. Will not show up in release changelog.
- documentation: Documentation only changes. Will not show up in release changelog.
- feature request: Introducing a new feature.
- BREAKING CHANGE: Changes that break compatibility in some way with current major version.
- other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	78.7 MB	78.7 MB	0 B	0.69	0%
initSize	147 MB	147 MB	0 B	-1.08	0%
diffSize	68.3 MB	68.3 MB	0 B	-1.08	0%
buildSize	7.1 MB	7.1 MB	335 B	-0.66	0%
buildSbAddonsSize	1.79 MB	1.79 MB	335 B	-0.71	0%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.85 MB	1.85 MB	0 B	-0.73	0%
buildSbPreviewSize	271 kB	271 kB	0 B	-0.76	0%
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	4.1 MB	4.1 MB	335 B	-0.73	0%
buildPreviewSize	3 MB	3 MB	0 B	1.09	0%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	5.9s	15.8s	9.9s	0.34	62.8%
generateTime	22.1s	20.9s	-1s -208ms	-0.29	-5.8%
initTime	14.7s	14.6s	-87ms	-0.48	-0.6%
buildTime	10.7s	9.3s	-1s -410ms	-0.48	-15.1%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	6s	6.1s	84ms	-0.34	1.4%
devManagerResponsive	4s	3.9s	-79ms	-0.54	-2%
devManagerHeaderVisible	589ms	631ms	42ms	0.61	6.7%
devManagerIndexVisible	620ms	658ms	38ms	0.44	5.8%
devStoryVisibleUncached	809ms	946ms	137ms	-0.54	14.5%
devStoryVisible	619ms	656ms	37ms	0.41	5.6%
devAutodocsVisible	568ms	515ms	-53ms	-0.72	-10.3%
devMDXVisible	573ms	563ms	-10ms	0.49	-1.8%
buildManagerHeaderVisible	572ms	519ms	-53ms	-0.88	-10.2%
buildManagerIndexVisible	638ms	593ms	-45ms	0.16	-7.6%
buildStoryVisible	639ms	594ms	-45ms	-0.2	-7.6%
buildAutodocsVisible	542ms	465ms	-77ms	-1.39	🔰-16.6%
buildMDXVisible	690ms	458ms	-232ms	-1.23	-50.7%

Greptile Summary

This PR adds telemetry functionality for tracking discrepancies between browser and CLI test results in the Storybook addon-test.

Added STORYBOOK_ADDON_TEST_CHANNEL constant for telemetry events
Implemented telemetry emission in TestDiscrepancyMessage component
Updated InteractionsPanel to conditionally render TestDiscrepancyMessage
Modified preset.ts to handle telemetry events and include addon version
Added storyId prop to relevant components for accurate event tracking

nx-cloud · 2024-10-11T06:29:52Z

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 8db0daa. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

✅ Successfully ran 1 target

nx run-many -t build --parallel=3

Sent with 💌 from NxCloud.

greptile-apps

_{7 file(s) reviewed, 9 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2024-10-11T06:29:02Z

code/addons/test/src/components/InteractionsPanel.tsx

@@ -1,6 +1,7 @@
 import * as React from 'react';

 import { styled } from 'storybook/internal/theming';
+import type { StoryId } from 'storybook/internal/types';



syntax: Import StoryId from @storybook/csf instead of storybook/internal/types

greptile-apps · 2024-10-11T06:29:41Z

code/addons/test/src/components/TestDiscrepancyMessage.tsx


 import { Link } from 'storybook/internal/components';
 import { useStorybookApi } from 'storybook/internal/manager-api';
 import { styled } from 'storybook/internal/theming';
+import type { StoryId } from 'storybook/internal/types';



syntax: The import for StoryId should be from '@storybook/csf' instead of 'storybook/internal/types'

code/addons/test/src/components/TestDiscrepancyMessage.tsx

greptile-apps · 2024-10-11T06:30:01Z

code/addons/test/src/constants.ts

@@ -1,6 +1,7 @@
 export const ADDON_ID = 'storybook/test';
 export const TEST_PROVIDER_ID = `${ADDON_ID}/test-provider`;
 export const PANEL_ID = `${ADDON_ID}/panel`;
+export const STORYBOOK_ADDON_TEST_CHANNEL = 'STORYBOOK_ADDON_TEST_CHANNEL';


style: Consider using a more specific name for the channel, e.g., 'STORYBOOK_ADDON_TEST_DISCREPANCY_CHANNEL'

code/addons/test/src/preset.ts

greptile-apps · 2024-10-11T06:30:29Z

code/addons/test/src/preset.ts

+      readFileSync(packageJsonPath, { encoding: 'utf-8' })
+    );
+
+    channel.on(STORYBOOK_ADDON_TEST_CHANNEL, (event: Event) => {
+      // @ts-expect-error This telemetry is not a core one, so we don't have official types for it (similar to onboarding addon)
+      telemetry('addon-test', {
+        ...event,
+        addonVersion,
+      });


logic: The event type check ('telemetry') doesn't match the expected type ('test-discrepancy') from the PR description. This might cause the telemetry to be sent for unintended events.

code/addons/test/src/preset.ts

yannbf · 2024-10-11T13:55:47Z

This PR needs some extra work based on feedback from @shilman

shilman · 2024-10-11T15:27:42Z

This looks good to me.

Separately, can we also add the following telemetry:

test watch initiated
- provider
- number of stories requested
test run completed
- provider
- number of stories tested
- number of stories requested
- test run duration
- completion: 'finished' | 'canceled' | 'error'

yannbf · 2024-10-14T10:27:50Z

test watch initiated

provider

number of stories requested

test run completed

provider

number of stories tested

number of stories requested

test run duration

completion: 'finished' | 'canceled' | 'error'

We decided to make this into a separate PR!

yannbf added telemetry ci:normal addon: test labels Oct 11, 2024

yannbf self-assigned this Oct 11, 2024

yannbf force-pushed the yann/discrepancy-detection-telemetry branch from 48aef59 to 893e91b Compare October 11, 2024 06:28

yannbf requested review from ghengeveld, valentinpalkovic and shilman October 11, 2024 06:28

greptile-apps bot reviewed Oct 11, 2024

View reviewed changes

yannbf force-pushed the yann/discrepancy-detection-telemetry branch from 893e91b to 04d8ce0 Compare October 11, 2024 06:57

add telemetry on discrepancy results

cf7ba11

yannbf force-pushed the yann/discrepancy-detection-telemetry branch from 04d8ce0 to cf7ba11 Compare October 11, 2024 08:49

add testRunId to telemetry

8db0daa

valentinpalkovic approved these changes Oct 11, 2024

View reviewed changes

yannbf added the ci: do not merge label Oct 11, 2024

yannbf merged commit 1c671de into unified-ui-testing Oct 14, 2024
58 checks passed

yannbf deleted the yann/discrepancy-detection-telemetry branch October 14, 2024 10:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addon Test: Add telemetry on discrepancy results #29329

Addon Test: Add telemetry on discrepancy results #29329

yannbf commented Oct 11, 2024 •

edited by storybook-pr-benchmarking bot

Loading

nx-cloud bot commented Oct 11, 2024 •

edited

Loading

greptile-apps bot left a comment

greptile-apps bot Oct 11, 2024

greptile-apps bot Oct 11, 2024

greptile-apps bot Oct 11, 2024

greptile-apps bot Oct 11, 2024

yannbf commented Oct 11, 2024

shilman commented Oct 11, 2024

yannbf commented Oct 14, 2024

Addon Test: Add telemetry on discrepancy results #29329

Addon Test: Add telemetry on discrepancy results #29329

Conversation

yannbf commented Oct 11, 2024 • edited by storybook-pr-benchmarking bot Loading

What I did

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

🦋 Canary release

Greptile Summary

nx-cloud bot commented Oct 11, 2024 • edited Loading

☁️ Nx Cloud Report

greptile-apps bot left a comment

Choose a reason for hiding this comment

greptile-apps bot Oct 11, 2024

Choose a reason for hiding this comment

greptile-apps bot Oct 11, 2024

Choose a reason for hiding this comment

greptile-apps bot Oct 11, 2024

Choose a reason for hiding this comment

greptile-apps bot Oct 11, 2024

Choose a reason for hiding this comment

yannbf commented Oct 11, 2024

shilman commented Oct 11, 2024

yannbf commented Oct 14, 2024

yannbf commented Oct 11, 2024 •

edited by storybook-pr-benchmarking bot

Loading

nx-cloud bot commented Oct 11, 2024 •

edited

Loading