How to aggregate Playwright metrics by test.step instead of page

[stefan123t]

I do appreciate the artillery-engine-playwright integration to re-use DOM-model based playwright scripts for load-testing and I have been able to come up with a simple scenario, as below.

This works fine but as the app-context does not change all the events / interactive actions are aggregated with the same app-context.
Is there a way to allow aggregation based on playwright test.step's ?

I tried to require / import test from @playwright/test but it still gives me as in #1325:

| Error: test.step() can only be called from a test
    at TestTypeImpl._step (...\node_modules\playwright\lib\common\testType.js:191:26)
    at Function.step (...\node_modules\playwright\lib\transform\transform.js:236:12)
    at moveSimpson (...\search-person.js:6:16)
    at Array.scenario (...\artillery\node_modules\artillery-engine-playwright\index.js:136:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processImmediate (node:internal/timers:447:9)

Is simply aggregating the metrics using playwright test.step's the right approach or would I rather need to create an artillery-plugin-metrics-by-test-step like in #1196 ?

I read #1831 and #1754 discussion but I am still at loss being fairly new to Artillery.io

---

Details

search-person.js:

// @ts-check
module.exports = { searchPerson };
// Note: this works without the following pwtest.step grouping's, i.e. only using await page....
const { test: pwtest } = require("@playwright/test");

async function searchPerson(page) {
  await pwtest.step('Step 1 - Homepage', async() => {
    await page.goto('https://example.tld/app-context/');
  });
  await pwtest.step('Step 2 - Login', async() => {
    await page.getByLabel('username').click();
    await page.getByLabel('username').fill('username');
    await page.getByLabel('username').press('Tab');
    await page.getByLabel('password').click();
    await page.getByLabel('password').fill('password');
    await page.getByRole('button', { name: 'login' }).click();
  });
  await pwtest.step('Step 3 - Some Button', async() => {
    await page.getByRole('button', { name: 'Yes' }).click();
  });
  await pwtest.step('Step 4 - Start > Search', async() => {
    await page.getByTitle('Search').click();
  });
  await pwtest.step('Step 5 - Lastname: Someone, Surname: Else, Search', async() => {
    await page.getByLabel('lastname').click();
    await page.getByLabel('lastname').fill('Someone');
    await page.getByLabel('surname').click();
    await page.getByLabel('surname').fill('Else');
    await page.getByRole('button', { name: 'Search' }).click();
  });
  // ...
  await pwtest.step('Step 14 - Logout', async() => {
    await page.getByRole('link', { name: 'logout' }).click();
  });
}

search-person.yml

config:
  target: https://example.tld/app-context/
  # Load the Playwright engine:
  phases:
    - name: Warm up phase
      duration: 20
      arrivalRate: 0
      rampTo: 5
    - name: Cool down phase
      duration: 5
      arrivalRate: 5
      rampTo: 0
  engines:
    playwright:
      launchOptions:
        headless: true # false to debug in browser instance
#        aggregateByName: true
  # Path to JavaScript file that defines Playwright test functions
  processor: "./search-person.js"
scenarios:
  - name: searchPerson
    engine: playwright
    testFunction: "searchPerson"

[bernardobridge]

Hi @stefan123t,

I believe the best way to do what you're describing would be to leverage Artillery's ability to emit custom metrics. You might be interested in using the pattern I described here #2147 (comment) to achieve this, which does the following:

1. Creates the following utility wrapper withTransactionTimer which captures timings for arbitrary actions:

const withTransactionTimer = async (transactionName, events, userActions) => {
  const startedTime = Date.now();
  await userActions();
  const difference = Date.now() - startedTime;

  events.emit('histogram', transactionName, difference)
}

2. Instead of wrapping your steps with pwtest.step, use the withTransactionTimer function;
3. You should now see metrics emitted in your Artillery reports with the names given to the first argument of withTransactionTimer, in addition to the usual Playwright metrics. It will give you metrics for the groups of actions you did.

Hope that helps!

[fatkobra]

I tried to use your code as inspiration bt I am getting the following error: @bernardobridge @stefan123t

TypeError: Cannot read properties of undefined (reading 'emit')

[stefan123t]

@fatkobra I believe to have seen this also for a specific version of my script.
However this could be solved ...

...
// async function searchPerson(page) {
async function searchPerson(page, vuContext, events, test) {

I recall that I had to pass the vuContext, events and test to the async function too, in order to have access on these objects. #2147 (comment)

Did you check the below solution by @bernardobridge which sounds very promising ?

[bernardobridge]

Hi @fatkobra , you'll need to provide a sample script (YAML and JS/TS file) for me to be able to see what might be wrong.

As @stefan123t kindly mentioned though, we have added step() helper into Artillery now, so you might not need a custom function for it: https://www.artillery.io/docs/reference/engines/playwright#teststep-argument

[fatkobra]

Hi Stefan and Bernardo,

thank you so much, I got it to work. Appreciate the reply tough.

[stefan123t]

Glad you got it to work @fatkobra ! What was the issue for us others to learn from you ?
See above for the way I have solved it back then with the help from @bernardobridge

[bernardobridge]

For anyone checking this thread, we've released in Artillery v2.0.0-38 a convenience helper similar to the one exemplified above, allowing you to wrap parts of your Playwright code with step() that emit Artillery custom metrics.

More information in: https://www.artillery.io/docs/reference/engines/playwright#teststep-argument

[qwertybutter]

Hi, @bernardobridge !
I wonder if it's possible to use something like sub-steps in Artillery + Playwright tests. What I mean is let's say I do some actions in my browser, so I open 'Discount' page. Then I want to create a new discount and search for a some other discount on my 'Discount' page. So my main step would be "Open Discount page". Its sub steps would be create a new discount and search for a discount. So I see it something like this:
Step1: Open Discount page
Step1-1: Create a new discount
Step1-2: Search for an existing discount

So I need to capture metrics for each step and sub steps. To do this I have implemented the solution you suggested above and the built in test functionality, so my code looks like this:

export async function testWithSteps(page: Page, vuContext, events, test) {
  const { step } = test;
  async function executeSubStep(transactionName, events, userActions) {
    const startedTime = Date.now();
    await userActions();
    const difference = Date.now() - startedTime;
    events.emit('histogram', `${transactionName}`, difference);
  }
  async function timedStep(name: string, action: () => Promise<void>) {
    const start = Date.now();
    await step(name, action);
    const duration = Date.now() - start;
  }
  await timedStep('Step1: Go to Discounts page', async () => {
    await page.getByText('Discounts').click();
    await executeSubStep("Step1-1: Click 'Add discount' button", events, async () => {
      await page.getByRole('button', { name: 'Add Discount' }).click();
    });
    await executeSubStep("Step1-2: Click 'Cancel' button", events, async () => {
      await page.getByRole('button', { name: 'Cancel' }).click();
    });
    await executeSubStep("Step1-3: Search for a discount by name'discount'", events, async () => {
      await page.getByRole('textbox', { name: 'Enter Name' }).fill('discount');
      await page.getByRole('textbox', { name: 'Enter Name' }).press('Enter');
    });
  });
}

Should it work properly ? May be there is another way to implement steps and sub-steps tracking ?

[stefan123t]

@qwertybutter did you try it with the step() helper available in artillery v2.0.0-38 onwards ?
I would assume that you can also enclose multiple of those step() helpers within each other.

@bernardobridge maybe we can enhance the example to reflect this in the documentation ?

async function loginSearchAndLogout(page, vuContext, events, test) {
  //1. simply add this line to your scenario function, or use test.step below instead
  const { step } = test;
  const userid = vuContext.vars.userid;
  const recordid = vuContext.vars.recordid;
 
await step('login_process', async () => {  // <-- added this
 //2. wrap any logic you have in steps (sometimes you might already have something like this done from existing playwright tests)
  await step('landing_page', async () => {
    await page.goto('https://internaltesturl.com/landing');
  });
 
  await step('submit_login', async () => {
    await page.getByLabel('id-label').fill(`${userid}`);
    await page.getByLabel('Password').fill(`${password}`);
    await page.getByRole('button', { name: 'Submit' }).click();
  });
 
await step('example_task', async () => {  // <-- added this
  await step('search_record', async () => {
    await page.getByPlaceholder('enter request id').fill(`${recordid}`);
    await page.getByRole('button', { name: 'Go' }).click();
    await page.locator('css=button.avatar-button').click();
  });
  // additional example action
  await step('do_something', async () => {
    await page.getByPlaceholder('enter order count').fill(`${recordid}`);
    await page.getByRole('button', { name: 'Order' }).click();
    await page.locator('css=button.avatar-button').click();
  });
}); // <-- and that
 
  await step('logout', async () => {
    await page.getByText('Logout').click();
  });
}); // <-- and that
}

[qwertybutter]

@stefan123t it seems to be working)

[stefan123t]

@qwertybutter thanks for the verification.

@bernardobridge I think we could add this to the documentation to improve the example with aggregated steps.