Using Plugin-Expect with CSV files --solo flag

[jinman]

We are trying to use artillery for functional testing as per https://artillery.io/blog/using-artillery-for-your-functional-testing/. However, we have defined all our test cases (requests and responses) in a CSV file (300 rows) which we are reading in sequential order, instead of defining 300+ scenarios. What should be the duration and arrivateRate functional phases? We tried following options:

1. duration: 300, arrivalRate: 1 - this timesout before completing all rows in the CSV
2. duration: 1, arrivalRate: 300 - this creates 300 virtual users but we are not sure whether all the APIs are complete
3. duration: 1, arrivalRate: 1 - this creates 1 virtual user and runs only 1st row in the CSV
4. --solo flag?

Can someone please provide some ideas on what should we provide as config so that all the rows in the CSVs are atleast tested once?

@dennmart, you blog was super helpful, if you can provide some pointers on how to run all the tests from CSV file once, that will be great!

[hassy]

hi @jinman, that's a really neat way to define your tests, love it! (reminds me of the concept of table-driven tests popular in Go)

Artillery does not have a way to provide the contents of the entire CSV file to a virtual user right now (it's done per-row), which would make this easy. There's a slightly more involved way to get this done right now though with:

1. Add a custom function to load the contents of the CSV into an array
2. Use loop-over to iterate over the data and send a request + check assertion

I've put together a working example here:

https://github.com/artilleryio/artillery-examples/tree/main/table-driven-functional-tests

[jinman]

hi @hassy awesome! Thanks for detailed answer.

Our CSV is very simple with following fields:
url,method,requestBody,responseStatus,responseBody

which we try to check for responseStatus and check for expected responseBody.

By theory, if we have 300 apis (300 rows in CSV), with duration: 1 and arrivalRate:300, (sequential order) it will spin up 300 virtual users (thread) in 1 second. But this is also unfortunately not working.

Any idea why?

again, thanks for the awesome library!

[hassy]

duration: 1 and arrivalRate: 300 will create 300 VUs, but each VU only gets one row from the CSV file (picked at random), so you don't get the same guarantee of sequential execution. The example solution I posted should do exactly what you need (and you can extend it to work with the extra fields in your CSV file).

[jinman]

@hassy makes sense. Thanks a lot! It would be great we can use the sequence flag to get things in sequence. But your solution works for us. Thanks a lot.