core(lib): add error reporting #2420

patrickhulce · 2017-06-01T23:27:31Z

brendankenny · 2017-06-02T01:37:19Z

an overview of your design would be helpful (and be useful as docs for CONTRIBUTING). e.g. difference between expected errors and non-fatal ones, when to capture message vs exception, when things will be automatically collected and when it should be explicitly called (like the stack trace comment)

brendankenny

some hopefully helpful comments.

An overview would definitely be helpful cause it is hard to tell what's on purpose and what's currently just sketched out (e.g. handling all our platforms) :)

brendankenny · 2017-06-02T01:37:45Z

lighthouse-cli/bin.ts

@@ -175,10 +176,22 @@ export async function runLighthouse(
    url: string, flags: Flags, config: Object|null): Promise<{}|void> {
  let launchedChrome: LaunchedChrome|undefined;

+  if (typeof flags.disableErrorReporting === 'undefined') {


should be enableErrorReporting?

that makes opt-out (which is the only usable flag) become --no-enable-error-reporting which feels odd, admittedly it definitely feels odd to check disableErrorReporting in runner but I didn't think we had any other case where we renamed the CLI flag from CLI to core

brendankenny · 2017-06-02T01:38:30Z

lighthouse-cli/bin.ts

-    const results = await lighthouse(url, flags, config);
+    const results = await lighthouse(url, flags, config, {
+      environment: isDev() ? 'development' : 'production',
+      serverName: 'redacted',


they send the serverName by default which is potentially private information we definitely we don't need to collect, this doesn't actually fix it but was trying it out

brendankenny · 2017-06-02T01:41:50Z

lighthouse-core/audits/load-fast-enough-for-pwa.js

@@ -122,10 +123,17 @@ class LoadFastEnough4Pwa extends Audit {
        }

        if (!areLatenciesAll3G) {
+          // eslint-disable-next-line max-len


move not needed?

ya i'll revert

brendankenny · 2017-06-02T01:42:57Z

lighthouse-core/audits/load-fast-enough-for-pwa.js

+          // eslint-disable-next-line max-len
+          const debugString = `First Interactive was found at ${timeToFirstInteractive.toLocaleString()}, however, the network request latencies were not sufficiently realistic, so the performance measurements cannot be trusted.`;
+
+          Sentry.captureMessage('Network request latencies were not realistic', {


what is the level of detail we want to log for this sort of thing? I guess magnitude of the issue is something, and then can cross reference with the site being tested?

yeah saving the diff to 150ms sgtm

brendankenny · 2017-06-02T01:44:36Z

lighthouse-core/gather/driver.js

@@ -287,7 +288,7 @@ class Driver {
            }

            // If both the data and the url are empty strings, the page had no manifest.
-            return reject('No web app manifest found.');
+            return reject(ExpectedError(new Error('No web app manifest found.')));


this seems more like impetus to add a third return possibility (e.g. null)

(and this whole method will hopefully get fixed in #2349 (comment))

brendankenny · 2017-06-02T01:45:36Z

lighthouse-core/gather/gather-runner.js

    return promise.catch(err => {
      if (err.fatal) {
        throw err;
+      } else {
+        Sentry.captureException(err, {tags: {gatherer: gathererName}, level: 'warning'});


if this is just for non-fatal errors, may make more sense in collectArtifacts (or even runner)

why there? the method calling out that just explicitly catches errors and looks at them was the first place that came to my mind or where this logic might live if I was looking for it, so open to much better explanations than that haha :)

IMO it's just for pulling fatal errors out of the stream. Non-fatal errors are handled on down the line (and due to a continuing lack of fatal errors, we should start to consider getting rid of it altogether :)

collectArtifacts is explicitly for collating the results of gatherers, including any errors, and has a slot ready made for the sentry call

brendankenny · 2017-06-02T01:50:16Z

lighthouse-core/gather/gatherers/start-url.js

@@ -77,7 +79,7 @@ class StartUrl extends Gatherer {

  afterPass(options, tracingData) {
    if (!this.startUrl) {
-      return Promise.reject(new Error('No start_url found inside the manifest'));
+      return Promise.reject(ExpectedError(new Error('No start_url found inside the manifest')));


if we go with ExpectedError, it should definitely be new ExpectedError or createExpectedError and should settle on whether to pass a string or an exception in

passing a string with createExpectedError has the problem of sentry showing you the line where the error was created and not the more useful line here, and I thought you didn't like things extending from Error for some reason but I can't remember why? haha

brendankenny · 2017-06-02T01:52:47Z

lighthouse-core/lib/sentry.js

+const log = require('./log');
+
+// Fix the polyfill. See https://github.com/GoogleChrome/lighthouse/issues/73
+self.setImmediate = function (callback) {


prefer rest args, but what did happen to the fix for #73?

its still just waiting for us to bump devtools :)

brendankenny · 2017-06-02T01:53:58Z

lighthouse-core/lib/sentry.js

+};
+
+const noop = () => undefined;
+const sentryDelegate = module.exports = {


this delegate is very hard to follow but you probably already know that :)

brendankenny · 2017-06-02T01:54:35Z

lighthouse-core/runner.js

@@ -33,6 +34,8 @@ class Runner {

    const config = opts.config;

+    Sentry.init(opts.flags.disableErrorReporting === false, opts.environmentData || {});


definitely should be opts.flags.enableErrorReporting === true

is having || {} going to be confusing in the data? runner may want to try to infer more to differentiate different environments (but you might already be planning that)

mentioned above too, but why "definitely"?

brendankenny · 2017-06-02T04:26:43Z

and, just to be clear, the structure of things seems to be what we've been discussing for a while/what I was imaginging, so it's looking good to me :)

The biggest unfortunate thing is the expected/regular/fatal exception split, but we've also been anticipating having to go down that path for a while as well. Maybe we can revisit resolving on a sentinel value instead of rejecting on an "expected" exception, but avoid going back to the -1 world by still having them automatically managed by gather-runner/runner.

patrickhulce · 2017-06-09T20:51:12Z

lighthouse-cli/bin.ts

@@ -151,8 +152,8 @@ function saveResults(results: Results, artifacts: Object, flags: Flags) {
        return innerPromise.then((_: Results) => Printer.write(results, outputType, outputPath));
      }, Promise.resolve(results));
    } else {
-      const outputPath =
-          flags.outputPath || `${resolvedPath}.report.${typeToExtension(flags.output)}`;
+      const outputPath = flags.outputPath ||


this was actually totally breaking vscode-insiders syntax highlighting :) filed an issue about it

paulirish

can you add us to the sentry project so we can see the results?

paulirish · 2017-06-19T18:38:06Z

lighthouse-cli/sentry-prompt.ts

+
+  let timeout: NodeJS.Timer;
+
+  const prompt = inquirer.prompt([


getting an exception in the prompt() method:

hmm strange, I can't repro, it happens every time?

yarn install-all fixes. :)

paulirish · 2017-06-19T18:45:13Z

lighthouse-cli/sentry-prompt.ts

+
+const MESSAGE = [
+  'Lighthouse would like to report back any errors that might occur while auditing. \n  ',
+  'Information such as the URL you are auditing, its subresources, your operating system, Chrome, ',


this message is pretty huge. (See screenshot) can we break it up?
fwiw the message string can be colorized before going into prompt()

Lighthouse requests permission to anonymously report back runtime exceptions
This can include data such as the test URL, its subresources, your OS, Chrome version, and Lighthouse version.
May Lighthouse report this data to aid in improving the tool?

patrickhulce · 2017-06-19T19:12:02Z

can you add us to the sentry project so we can see the results?

@paulirish yeah the free plan only allows one member though, time to apply for OSS account or whip out the gcard?

paulirish · 2017-06-19T19:19:50Z

OSS acct for sho

paulirish · 2017-06-19T21:13:46Z

lighthouse-core/runner.js

+      networkThrottling: !opts.flags.disableNetworkThrottling,
+      cpuThrottling: !opts.flags.disableCpuThrottling,
+    };
+    Sentry.mergeContext({extra: Object.assign({}, environmentData.extra, sentryContext)});


wdyt about moving this context setting stuff into lib/sentry ?

patrickhulce · 2017-06-20T01:12:37Z

A not insignificant portion of this PR is now changing the behavior of the StartUrl gatherer to pass the debugString instead of an error, so might be worth breaking that out into a separate PR at this point

brendankenny · 2017-06-20T01:17:53Z

so might be worth breaking that out into a separate PR at this point

I'd say 👍

brendankenny · 2017-06-20T06:22:42Z

lighthouse-cli/cli-flags.ts

@@ -56,6 +56,7 @@ export function getFlags() {
          ],
          'Configuration:')
      .describe({
+        'enable-error-reporting': 'Enables error reporting (on by default)',


maybe we can come up with a different flag name, as it isn't really the case that it's on by default (instead it prompts by default), and a first glance may alarm some

hence why I had it as disable-error-reporting but someone wanted it flipped :P

brendankenny · 2017-06-20T06:36:05Z

CONTRIBUTING.md

+}
+```
+
+#### If you throw an error in an audit to communicate audit failure instead of using the score property, *use createExpectedError*.


If you throw an error in an audit to communicate audit failure instead of using the score property, use createExpectedError

after the changes to manifest fetching/start_url, are expected errors still needed? It sounds more like we should just discourage throwing errors to indicate an expected negative result

yes I've removed createExpectedError entirely now, will update

patrickhulce · 2017-06-20T19:29:30Z

opened and rebased against #2549

paulirish · 2017-07-07T19:21:15Z

Blocked on #2549

brendankenny

some ideas on not saying opt-out since there's just opting in or not opting in. Take or leave these :)

brendankenny · 2017-10-20T01:09:21Z

docs/error-reporting.md

+* Your Chrome version
+* Your operating system
+
+## How do I opt-in/opt-out?


maybe just have "How do I opt-in?" to be clear it's only opt-in?

brendankenny · 2017-10-20T01:09:51Z

docs/error-reporting.md

+* Your operating system
+
+## How do I opt-in/opt-out?
+The first time you run the CLI you will be prompted with a message asking you if Lighthouse can anonymously report runtime exceptions. You can give a direct response of `yes` or `no` to save your response, and you will not be prompted again. If no response is given within 20 seconds, a `no` response will be assumed, and you will not be prompted again. Non-interactive terminal sessions and invocations with the `CI` environment variable set will automatically be opted out.


"You can give a direct response of yes or no to save your response, and you will not be prompted again." -> "You can give a direct response of yes or no, and you will not be prompted again.", maybe?

"will automatically be opted out." -> "will automatically not be opted in." maybe? (slightly more awkward but clearer there is no state that needs to be opted out of :)

also "y" and "n" work.
and just hitting enter defaults to "no". this should be mentioned as well.

brendankenny · 2017-10-20T01:11:43Z

docs/error-reporting.md

+
+The CLI also has two flags to control error reporting that will override the saved preference. Running Lighthouse with `--enable-error-reporting` will report errors regardless of the saved preference, and running Lighthouse with `--no-enable-error-reporting` will *not* report errors regardless of the saved preferences.
+
+## How do I change my opt-in/opt-out preference?


How do I change my opt-in preference??

brendankenny · 2017-10-20T01:14:19Z

docs/error-reporting.md

+
+## What's going on?
+
+The Lighthouse team is constantly trying to improve the reliability of our tools. We've added functionality to anonymously report runtime exceptions, given your consent, using [Sentry](https://sentry.io/welcome/). We will use this information to detect new bugs and avoid regressions.


maybe
"If you give your consent, we've added functionality to anonymously report runtime exceptions using [Sentry](https://sentry.io/welcome/)."

heh, but the functionality is there if they give consent or not, the phrase is modifying "report" ;)

rephrased

brendankenny · 2017-10-20T01:22:07Z

docs/error-reporting.md

+
+The Lighthouse team is constantly trying to improve the reliability of our tools. We've added functionality to anonymously report runtime exceptions, given your consent, using [Sentry](https://sentry.io/welcome/). We will use this information to detect new bugs and avoid regressions.
+
+Only CLI users are currently impacted. Node module consumers are opted out by default without a prompt, and the DevTools and extension integrations have no automatic error reporting functionality.


maybe just include Node module consumers with Devtools and extension users for now since it won't be happening for them?

yeah was trying to be explicit that the code isn't even there, but that was probably confusing

brendankenny · 2017-10-20T01:22:25Z

docs/error-reporting.md

+## What will happen if I opt-in?
+Runtime exceptions will be reported to the team along with information on your environment such as the URL you tested, your OS, and Chrome version. See [what data gets reported](#what-data-gets-reported).
+
+## What will happen if I opt-out?


"What will happen if I don't opt-in?"?

paulirish

great doc.

paulirish · 2017-10-20T01:26:26Z

docs/error-reporting.md

+* Your operating system
+
+## How do I opt-in/opt-out?
+The first time you run the CLI you will be prompted with a message asking you if Lighthouse can anonymously report runtime exceptions. You can give a direct response of `yes` or `no` to save your response, and you will not be prompted again. If no response is given within 20 seconds, a `no` response will be assumed, and you will not be prompted again. Non-interactive terminal sessions and invocations with the `CI` environment variable set will automatically be opted out.


also "y" and "n" work.
and just hitting enter defaults to "no". this should be mentioned as well.

paulirish · 2017-10-20T22:31:33Z

@patrickhulce change lgtm

heads up: this discussion is unresolved https://github.com/GoogleChrome/lighthouse/pull/2420/files#r145858868

paulirish · 2017-10-21T00:39:51Z

@brendankenny lets do thisssssssssss. @patrickhulce now understood what you were going for in that init() thread. :)

brendankenny

two last things, but otherwise LGTM!

brendankenny · 2017-10-25T20:13:31Z

lighthouse-cli/run.ts

@@ -138,13 +138,13 @@ export function saveResults(results: Results, artifacts: Object, flags: Flags) {
 }

 export async function runLighthouse(
-    url: string, flags: Flags, config: Object|null): Promise<{}|void> {
+    url: string, flags: Flags, config: Object|null, environmentData?: Object): Promise<{}|void> {


not needed anymore?

brendankenny · 2017-10-25T20:16:11Z

lighthouse-cli/bin.ts

+    },
+  });
+
+  return await runLighthouse(url, cliFlags, config);


nit: since the function is async anyway, can drop the await here

paulirish · 2017-10-27T01:23:47Z

epic PR. super excited about this. \o/

* refactor(StartUrl): switch from error to debugString object * feat: add error reporting * cave to peer pressure * lint * merge fixes * reduce yarn churn * limit impact to run.ts * clang format * avoid duplicate errors * lint * disable error reporting on test * feedback * remove enableErrorReporting default * update readme * fix yarn.lock * add error reporting doc * doc feedback * move sentry call * more cleanup * remove unused environmentData

patrickhulce force-pushed the add_sentry branch from 002650b to 2b263cd Compare June 1, 2017 23:27

brendankenny reviewed Jun 2, 2017

View reviewed changes

paulirish force-pushed the add_sentry branch from 2b263cd to d84d4dc Compare June 5, 2017 16:47

patrickhulce commented Jun 9, 2017

View reviewed changes

patrickhulce mentioned this pull request Jun 16, 2017

fix(unused-css-rules): update to support new coverage format #2518

Merged

paulirish requested changes Jun 19, 2017

View reviewed changes

paulirish reviewed Jun 19, 2017

View reviewed changes

brendankenny reviewed Jun 20, 2017

View reviewed changes

refactor(StartUrl): switch from error to debugString object

fbd724e

patrickhulce mentioned this pull request Jun 20, 2017

refactor(StartUrl): switch from error to debugString object #2549

Merged

feat: add error reporting

528286d

patrickhulce force-pushed the add_sentry branch from d98d240 to 528286d Compare June 20, 2017 19:28

patrickhulce added 2 commits June 20, 2017 14:59

cave to peer pressure

062fee6

lint

b7f89f6

paulirish mentioned this pull request Jul 7, 2017

fix: remove recovery logic for fmp #2244

Closed

patrickhulce added 6 commits July 12, 2017 15:06

Merge branch 'master' into add_sentry

94f6f54

merge fixes

5ebc608

reduce yarn churn

80f2040

limit impact to run.ts

140ac9e

clang format

eea3f67

avoid duplicate errors

40c0506

patrickhulce requested a review from vinamratasingal-zz as a code owner October 19, 2017 17:57

patrickhulce added waiting4reviewer and removed waiting4committer labels Oct 19, 2017

brendankenny reviewed Oct 20, 2017

View reviewed changes

paulirish reviewed Oct 20, 2017

View reviewed changes

patrickhulce added waiting4committer and removed waiting4reviewer labels Oct 20, 2017

doc feedback

cf0e640

patrickhulce added waiting4reviewer and removed waiting4committer labels Oct 20, 2017

paulirish added waiting4committer and removed waiting4reviewer labels Oct 20, 2017

patrickhulce added 2 commits October 20, 2017 16:35

Merge branch 'master' into add_sentry

637b124

move sentry call

d77270a

patrickhulce added waiting4reviewer and removed waiting4committer labels Oct 21, 2017

more cleanup

6b4f9ea

paulirish approved these changes Oct 21, 2017

View reviewed changes

brendankenny approved these changes Oct 25, 2017

View reviewed changes

patrickhulce added 2 commits October 26, 2017 10:42

remove unused environmentData

1aac5a2

merge branch 'master' into add_sentry

a4c982d

patrickhulce merged commit de57850 into master Oct 27, 2017

patrickhulce deleted the add_sentry branch October 27, 2017 00:01

dependencies bot mentioned this pull request Dec 17, 2017

Update lighthouse in / from 2.5.0 to 2.7.0 chauncey-garrett/dotfiles#57

Open

paulirish removed the waiting4reviewer label Mar 6, 2018

GoogleChrome deleted a comment from stevinus73 Feb 4, 2022

		@@ -33,6 +34,8 @@ class Runner {

		const config = opts.config;

		Sentry.init(opts.flags.disableErrorReporting === false, opts.environmentData \|\| {});


		The CLI also has two flags to control error reporting that will override the saved preference. Running Lighthouse with `--enable-error-reporting` will report errors regardless of the saved preference, and running Lighthouse with `--no-enable-error-reporting` will not report errors regardless of the saved preferences.

		## How do I change my opt-in/opt-out preference?


		## What's going on?

		The Lighthouse team is constantly trying to improve the reliability of our tools. We've added functionality to anonymously report runtime exceptions, given your consent, using [Sentry](https://sentry.io/welcome/). We will use this information to detect new bugs and avoid regressions.


		The Lighthouse team is constantly trying to improve the reliability of our tools. We've added functionality to anonymously report runtime exceptions, given your consent, using [Sentry](https://sentry.io/welcome/). We will use this information to detect new bugs and avoid regressions.

		Only CLI users are currently impacted. Node module consumers are opted out by default without a prompt, and the DevTools and extension integrations have no automatic error reporting functionality.

core(lib): add error reporting #2420

core(lib): add error reporting #2420

Conversation

patrickhulce commented Jun 1, 2017 • edited Loading

brendankenny commented Jun 2, 2017

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce Jun 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented Jun 2, 2017

Choose a reason for hiding this comment

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Jun 19, 2017

paulirish commented Jun 19, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Jun 20, 2017

brendankenny commented Jun 20, 2017

Choose a reason for hiding this comment

patrickhulce Jun 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Jun 20, 2017 • edited Loading

paulirish commented Jul 7, 2017

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulirish commented Oct 20, 2017

paulirish commented Oct 21, 2017

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulirish commented Oct 27, 2017

patrickhulce commented Jun 1, 2017 •

edited

Loading

patrickhulce Jun 6, 2017 •

edited

Loading

patrickhulce Jun 20, 2017 •

edited

Loading

patrickhulce commented Jun 20, 2017 •

edited

Loading