[Fargate]: Workers running out of memory or out of space

[DanSmithSC]

While running some recent load tests on Artillery version 2.0.15 - I'm noticing we're starting to get early terminations and exits with EXIT CODE 6

In researching the logs i'm starting to see errors for:
worker error, id: 1 Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory

Additional Logged Error of maybe some importance:
/artillery/loadgen-worker: line 277: echo: write error: No space left on device

However the fargate containers were only run with a count of 2, and the memory allocation for those were set to 20gigs of memory

--count 2 --launch-config '{\"cpu\": 4096, \"memory\": 20480}'

Any idea why the containers may be running out of space or hitting these memory problems?

Let me know if you need any details - Thanks in advance!

[bernardobridge]

Hey @DanSmithSC 👋

Would you be able to share some details about what you were running? If you are able to share the test and config, that would be helpful, so we can try to recreate it.

Also, was this same test working in older versions of Artillery and suddenly you're getting this issue (with no test changes), or you've only tried on 2.0.15?

Thank you!

[DanSmithSC]

I believe this was working before or at least on version 2.0.8. We had not seen these out of memory or spaces errors previously.

Any hints would be helpful.

config:
  ensure:
    p95: 5000
    maxErrorRate: 10
  variables:
    search_term:
      - tucker
      - test
      - west
      - savannah
  includeFiles: 
    - "../config/perf/events.csv"
    - "../config/perf/games.csv"
    - "../config/perf/schools.csv"
    - "../config/perf/users.csv"    
  payload:
    -
      path: "../config/{{ $environment }}/events.csv"
      delimiter: "\n"
      fields:
        - "eventKey"
    -
      path: "../config/{{ $environment }}/games.csv"
      delimiter: "\n"
      fields:
        - "gameKey"
    -
      path: "../config/{{ $environment }}/schools.csv"
      delimiter: "\n"
      fields:
        - "schoolSlug"
    -
      path: "../config/{{ $environment }}/users.csv"
      delimiter: "\n"
      fields:
        - "username"
    -
      path: "../config/sports.csv"
      delimiter: "\n"
      fields:
        - "sport"
  environments:
    perf: 
      target: "<ENV URL>"
      variables:
        memberService: "<ENV URL>"
      phases:
        - name: Starting
          duration: 1800
          arrivalRate: 1
          rampTo: 75
        - name: Warm up
          duration: 900
          arrivalRate: 75
          rampTo: 150
        - name: Ramp up load spike 1
          duration: 1800
          arrivalRate: 150
          rampTo: 250
        - name: Sustained load 
          duration: 900
          arrivalRate: 250
          rampTo: 300
        - name: Ramp up load spike 2
          duration: 900
          arrivalRate: 300
          rampTo: 600
        - name: Ramp down load
          duration: 4500
          arrivalRate: 600
          rampTo: 1
scenarios:
  - name: "Home page"
    weight: 2.5
    flow:
      - get:
          url: "/"
      - think: 5
  - name: "Login flow"
    weight: 1
    flow:
      - get:
          url: "/"
      - think: 5
      - get:
          url: "/users/sign_in"
      - think: 5
      - post:
          url: "{{ memberService }}/oauth/token"
          json:
            grant_type: "password"
            username: "{{ username }}"
            password: "{{ account_password }}"
            redir: "/"
          capture:
            - json: "$.access_token"
              as: "accessToken"
      - think: 5
      - get:
          url: "/member/account"
      - think: 5
  - name: "Watch events page"
    weight: 2
    flow:
      - get:
          url: "/watch-events"
      - think: 5
  - name: "Game page"
    weight: 2
    flow:
      - get:
          url: "/events/null/{{ gameKey }}"
      - think: 5
  - name: "School page"
    weight: 1.5
    flow:
      - get:
          url: "/schools/{{ schoolSlug }}"
      - think: 5
  - name: "School sport page"
    weight: 1
    flow:
      - get:
          url: "/schools/{{ schoolSlug }}/{{ sport }}"
      - think: 5
  - name: "Search query"
    weight: 2
    flow:
      - get:
          url: "/search?query={{ search_term }}"
      - think: 5
  - name: "Subscribe flow"
    weight: 1
    flow:
      - get:
          url: "/"
      - think: 5
      - get:
          url: "/subscribe/retail"
      - think: 5

[bernardobridge]

Thanks for sharing that @DanSmithSC . A couple more questions:

• how big are those CSV files?
• it seems the test runs for around 3 hours. How far along into it do you start seeing errors?

I will try to see when I have a chance if I notice any performance degradation on long running tests. However, we typically recommend scaling horizontally, so it might be worth you trying to split the load across more containers. For example, it would be better to run 10-20 containers with 5-10x less load per container.

[DanSmithSC]

The CSVs are very small... maybe about 6 lines each at most. and approximately 40-44 "bytes" in file size.

We are using it for a longer running "Soak" test. Also to test some of our auto-scaling. Typically the Errors would show approximately anywhere from 80-120 minutes in to like 150 minutes into a test if not earlier. Or either almost to the 2 hour or just after the 2-hour mark.

I'll try going more Horizontal in a future run. Reducing the arrival rates, but increasing the counts.

Thanks for that tip. I'll try and report back, but yes it seems the containers tend to EXIT out after a while or are killed :

Finished with code 137
June 18, 2024 at 15:02 (UTC-4:00) | /artillery/loadgen-worker: line 277: 132 Killed artillery "${cli_args[@]}"