test: replace console.error() with debuglog calls

[Trott]

Somehow thought I did this in 8905be2
but clearly did not.

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• commit message follows commit guidelines

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/30303/ ✅

[HarshithaKP]

@Trott, there are a 100 places where console.error is present. Do you recommend we should replace this with debug statements?

[Trott]

@Trott, there are a 100 places where console.error is present. Do you recommend we should replace this with debug statements?

I'd prefer not until we have a clear/obvious/documented way to cause those debug statements to show up in CI (Jenkins and GitHub Actions).

We'd also have to be careful not to do this in tests where console.error() itself is being tested.

I'm doing this one because I felt bad because I somehow messed it up when I just landed a change to the test a day ago.

Other people (notably @jasnell) might be considerably more enthusiastic about it than I am, though.

[jasnell]

@HarshithaKP :

there are a 100 places where console.error is present. Do you recommend we should replace this with debug statements?

Incrementally where it makes sense, yes. The idea is to separate output that is considered part of the test itself from informational/debug output. The rationale is to make the intent of the output clear and to avoid cases where the debug output itself may alter the outcome of the test (this is extremely rare but we have encountered it before so the probability is definitely non-zero).

What I would not do is make these changes all at once. If someone is updating a test file for another reason, then it makes sense to also make these changes. Or, if we want to document which ones can be changed, it would make a good first contribution for a new contributor to change one test at a time.

@Trott

...until we have a clear/obvious/documented way to cause those debug statements to show up in CI (Jenkins and GitHub Actions).

@nodejs/build can enable that by optionally adding NODE_DEBUG=test to the environment of CI jobs. Note, however, that this output has rarely been helpful in diagnosing issues reported in CI, and when a test fails, it is trivial for someone investigating the issue to enable NODE_DEBUG=test in their local environment, so I would not consider this to be a blocking concern.

[Trott]

@nodejs/build can enable that by optionally adding NODE_DEBUG=test to the environment of CI jobs.

Counterpoints:

1. We shouldn't have to involve Build WG to make that happen. (Counterpoint to my counterpoint: There are other ways to do this already. They just need to be documented and/or made a bit more obvious.)
2. That covers Jenkins only, but not GitHub Actions. (Counterpoint to my counterpoint: Jenkins is the more important one, at least at this time.)

it is trivial for someone investigating the issue to enable NODE_DEBUG=test in their local environment,

Yeah, true, that works when you can make the test fail reliably in the local environment. However, that is not the case I am concerned about. I am concerned about relatively infrequent failures ("flaky" tests) that happen on platforms that maybe the person debugging doesn't have easy access to. (This is a situation I have found myself in more often than I like, and I know I'm not the only one.)

so I would not consider this to be a blocking concern.

I would, but I don't expect everyone to agree with me on that, so it's 👍 that you don't. In any event, it's definitely a solvable problem. Someone just needs to spend a little time doing it. (On my list!)

[rvagg]

you could enable it via the test-ci Makefile target, then you wouldn't need to involve Build.

[addaleax]

@Trott Technically, this is ready to land … if you think we should, feel free to go ahead.

(As you know, I don’t personally have a strong opinion here.)

[Trott]

you could enable it via the test-ci Makefile target, then you wouldn't need to involve Build.

@rvagg I hadn't considered that viable because if we do it that way, it will be on all the time in CI rather than only when requested. Considering it more now, I suppose that could be considered desirable, although it could also be a reason to not do these kinds of changes at all.
¯\(ツ)/¯ I had imagined it as an option one turns on when needed only.

[rvagg]

Well, I'm not super keen on more verbose logs having to be stored by Jenkins and parsed by users, they're pretty big as they are. I'm just saying this is an option if it's needed, I don't consider myself in a position to judge whether it is or not!

[Trott]

Landed in 83ebd77