Artillery + Playwright: Handling playwright errors

[staramani-ut]

Hey there,

I hope you're doing well! I'm running some tests using Artillery + Playwright, and I've hit a snag with some errors related to Playwright.

I'm a bit stuck, and I was hoping someone could lend a hand. I am also new to both Artillery and Playwright, so please pardon if I am asking something trivial. Specifically, I'm looking for advice on handling playwright errors and marking an artillery test as failed when these errors arise.

I see errors like Error: pening a page: Ge [Error]: Timed out 40000ms waiting for expect(locator).toBeVisible() three times.

At the end, I see artillery results like this (I would like to see 3 for vuusers.failed):
vusers.completed: .............................................................. 50
vusers.created: ................................................................ 50
vusers.created_by_name.run surveys: ............................................ 50
vusers.failed: ................................................................. 0

I believe that makes sense since the failure happened within Playwright. But I would like to fail the artillery test if something like this happens. I am trying an afterResponse function, but that doesn't seem to help. I am probably missing something simple, and I would appreciate any help!

On a side note, I was curious if the failures were because of my laptop and tried them in GitHub Actions, and I encounter them there as well.

If you have any experience with this or any tips to share, I would be incredibly grateful. Thank you so much for your time and assistance!

p.s - It has been absolutely amazing writing performance tests that use Playwright code (yay 🥳 )! Thank you to the Artillery team!

[staramani-ut]

I have it working with the following catch in my main artillery function. But I am not sure if this is the best way to handle what I am trying to do.

catch (error) {
events.emit('counter', 'vusers.failed', 1);
events.emit('counter', 'vusers.completed', -1);
}

[up201306506]

If you want the vUser to count as a failure, you can always just fail to catch the error. Or re-throw it after you do. That's kind of the point of expect() statements.

By catching it, you're essentially telling the test that what happened is not enough to stop the test at that point and you wish to keep going.

[hassy]

hi @sthavamani-ut 👋 I'll need to double check - errors like the one you shared should result in a failed VU in the reports, so it might be an issue in our integration

p.s - It has been absolutely amazing writing performance tests that use Playwright code (yay 🥳 )! Thank you to the Artillery team!

Thank you, that's really nice to hear! :)

[staramani-ut]

Thank you, Hassy!

For now I think my hack does the trick. I appreciate your help.

[staramani-ut]

@hassy do you know if this is an issue still? I am still catching errors and emitting events.

[staramani-ut]

Okay, I was able to check node_modules/artillery-engine-playwright/index.js to see how error handling is done and I was able to remove my event.emits. 🎉