-
Notifications
You must be signed in to change notification settings - Fork 516
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Artillery runs forever and never exit (v2) #238
Comments
Thanks for the report @endyjasmi. I'm unable to reproduce the issue here. The following command completes fine for me:
What does the normal Artillery output look like on your system? Do you see reported latency metrics for the full 60 seconds? Are there any errors reported? What does |
@hassy From what I understand, this happens when the connection from artillery to the server is high. |
@endyjasmi Can you post a log? (Just the normal output from Artillery as it runs the test). Can you also check the CPU usage of node processes and check that there are two of them still running? |
I see the same problem.
It ran against an environment that never went down. The load generator machine did experience high cpu usage, but then it went down to normal levels. |
@portenez thanks, can you post the full log? Did you ever see metrics being reported while Artillery was running or always just |
We just figured it out. It was an exception being thrown during custom js code run via So, I guess there's still a bug. That an exception in one of the preProcessors will cause the whole test to hang |
Hi folks, I'm seeing this too, but when I'm using a prepared Websocket test. It gets to the end of the test, everything's fine, cluster being tested is fine, and then Artillery just keeps doing this forever:
|
@portenez thanks, that's something to fix. @igorclark does it happen every time or only sometimes? A reproducible test case would be amazing, but until then the only option I have is to review the code and try to spot any potential causes. |
@hassy don't know about every time, but certainly 9 times out of 10. Would love to provide a full test case including back-end, but it's not a public project. Here's the majority of the test plan though - don't know if it'll be much use but hope so :-)
|
@igorclark Thanks for that. Just to confirm - you don't see any error messages printed to the console at any point during a run that ends up spinning forever? If you don't mind, next time that happens, does the following command print anything:
|
Hey @hassy - no, no Artillery error messages. There are sometimes a couple of "ECONNRESET" or "connection closed by other party" when the server is getting really hammered, but that's normal, no? And when it ends up just spinning, it doesn't have any other errors, just the report every 10s. I'll do another run tomorrow and run the |
Thanks @tiaod. Seems like high arrival rates have something to do with it but I was unable to reproduce previously. I'll give it another go though. |
@hassy Oh I forgot to mention that. You are right, I use child_proccess spanw to run Artillery. But on a lower arrivalRate (eg: 1000), it could stop normally. |
Hi @hassy The same thing happens if I don't start the server. Artillery will keep waiting. |
@peara Thanks for the info. Artillery does try to time out connections - with an explicit timeout setting for HTTP and the default value used by the underlying library for Socket.io - something to look into to try to solve this. Cheers |
Unsure if relevant but thought it worth posting. Recently I came across this issue as well and it seemed to be issues with target and the scenarios url. example:
resulted in: the NaN issue as described by @igorclark
but changing the target to
|
We suffered this when using an invalid request URL (literal This would explain why prefixing the path worked for @cwthompson, as any URL which is not prefixed by |
Hey @hassy 👋 I'm experiencing the same symptoms. Artillery: 1.6.0-28 My config looks something like: config:
target: "https://example.com"
phases:
- duration: 60
arrivalRate: 20
defaults:
headers:
x-api-key: 'XXXXXXX'
scenarios:
flow:
- get:
url: "/example"
qs:
foo: "bar" The issue is intermittent (occurring ~2/3rds of the time). Below are two runs, one run immediately after the other with no config changes. Run no. 1 (error)
Run no. 2 (success)
If I get to the bottom of it I'll post here. If you have any ideas about what I might be doing wrong, or if I can provide any more information please let me know! Thanks |
I seem to be seeing the same issue, but have a bit more info that might be useful. First, see this screenshot, showing the issue with the hung test: The log output on the service has stopped, all the received requests have completed successfully. The test runner itself is in the hung state and sending no more requests (the test should have ended at 1 minute). The interesting part is that, when I kill the test runner, it seems like one last small batch of requests is fired off just before artillery exits: Notice the last two requests in the server log, that are received immediately upon killing artillery. You can also see that these requests are then immediately aborted (the My guess would be that artillery is hung because it's waiting for these requests, but the requests were never actually sent. And then killing the process causes them to somehow get flushed out and finally sent. |
This is so annoying issue 😭 I have artillery for stress tests and I want to find a limits of server - and artillery found it, but I am not able to get some reasonable output from artillery due this issue. My setup - I run this 2x parallel in docker containers to avoid problems with CPU overload.
The problem begins when the server begins to have data processing problems and starts to have timeouts and eventually crashes. After that, artillery starts show
Any solution for that? It is so annoying due I send results from artillery to elasticsearch and now, it doesn't send anything. |
@hassy - Can you provide some information about this issue? Without it, it doesn't make any sense to use it 🤷♂️ So please provide some information, if you care about that or this will be still open and we should look for another tool for stress testing. |
It's an extremely difficult issue to reproduce (and to create a reproducible test case for). In your case specifically @vojtech-cerveny, there's nothing unusual about the test script itself, so the issue is somewhere else, but there's no way to try to isolate what it might be unless you can provide more information. |
I can confirm that the issue is consistently reproducible with high arrival rate.
The higher arrival rate, the longer it takes from Artillery to finish and print a report. E.g. with timeout = 30s it succeeds to finish in 30s with arrival rate = 10, but it takes 60s for arrival rate = 500 I'm running it on a pretty beefy machine that has a lot of free CPU, yet I see CPU usage warnings |
@whitebyte If Artillery still completes successfully, that's not an instance of the issue. It would make sense that the test takes a longer time to complete if you're creating more virtual users (50 times more if you're going from 10 to 500/second). CPU warnings-wise - that also still makes sense. Artillery uses one CPU by default, so you're likely maxing a CPU, especially with a high arrival rate (each arrival is a whole new TCP connection by default). |
I'm using only one phase with a constant arrival rate. After some arrival rate threshold Artillery starts working indefinitely, eventually spitting NaNs |
Turns out that elapsed time is ignored during the run and only # of scenarios launched is taken into account. It looks like Artillery calculates the required number of scenarios to be launched before start, and then just sticks to this number, ignoring the actual time passed. Which is OK, but should be documented somewhere. |
Hi, I am running artillery too for load testing.
I am getting the same issue. Did anyone able to resolve this issue? If yes, please share.
|
I encountered the same issue too. My config
|
Which version of Artillery is that @alexanderankin? ( |
@hassy I am using errors to stop my scenario, its using http, version 1.6.1. |
I'm having the same issue. My test runs once per day, in docker container (based on image Here is a log: https://gist.github.com/mirao/780e34d7443a327cbac02306c89767b0. You can see that the test has been already running for more than 6 hours and haven't finished yet. Normally it finishes in a few minutes.
load-test.yml: config:
socketio:
transports: ["websocket"]
phases:
- duration: 120 # Run scenario for x seconds
arrivalCount: 2000 # Create total of x clients
ensure:
maxErrorRate: 0
environments:
dev:
target: "wss://my_socketio_server_in_kubernetes"
scenarios:
- engine: "socketio"
name: "Get teacher's sections"
flow:
- emit:
channel: "authenticate"
data:
{
"strategy": "local",
"accessToken": "{{ $processEnvironment.TEACHER_TOKEN }}",
}
acknowledge:
match:
json: "$.1.authenticated"
value: true
- think: 1
- emit:
channel: "section::find"
data: {}
- think: 20
- emit:
channel: "section::find"
data: {}
- think: 120
- engine: "socketio"
name: "Get student's sections"
flow:
- emit:
channel: "authenticate"
data:
{
"strategy": "local",
"accessToken": "{{ $processEnvironment.STUDENT_TOKEN }}",
}
acknowledge:
match:
json: "$.1.authenticated"
value: true
- think: 1
- emit:
channel: "section::find"
data: {}
- think: 30
- emit:
channel: "section::find"
data: {}
- think: 120
|
------------ Version Info ------------
|
For me just using an |
@jasperblues is your https://www.artillery.io/docs/guides/guides/http-reference#afterresponse-hooks |
Ooops! Thanks @hassy . . . i guess that's a common n00b error. |
Hello, I am seeing a very similar issue, when I intentionally put a bad path in the config for the processor artillery never exits.
VERSION INFO:
|
I'm seeing the same behavior on Artillery Core: 2.0.0-23, test with an invalid processor path errors out but never exits |
Drop the latest link that will resolve the issue to help people to find it easier as this is the first link on search! |
When the command
artillery quick -d 60 -r 256 https://www.company.com/
is executed, artillery will run fine for the first few request and later keep on looping with the following progress report and never exit.Report for the previous 10s @ 2017-01-11T09:08:56.069Z Scenarios launched: 0 Scenarios completed: 0 Requests completed: 0 RPS sent: NaN Request latency: min: NaN max: NaN median: NaN p95: NaN p99: NaN Scenario duration: min: NaN max: NaN median: NaN p95: NaN p99: NaN
Following is the environment it runs in:
Following is the log file generated by running
DEBUG=* artillery quick -d 60 -r 256 https://www.bloomon.nl/ 2>&1 | tee debug.log
:https://gist.github.com/endyjasmi/a597adc6a0fc5e1c874d7abbe9a93262
Thanks in advance for the helps.
The text was updated successfully, but these errors were encountered: