Test cases run out of memory

[sr258]

The test cases in CI run out of memory regularly. This might be because of the "integration" tests that use all the hub examples. If these tests are run in parallel, it will use a lot of memory. We should either increase the resources of the test runners or think of a way of limiting the parallelism of tests.

[JPSchellenberg]

As we don't pay for CircleCI we can not increase resources. We could enforce a queue via the workflows. So certain tests have to pass first before other tests are run. It would increase the running-time, though, but that is fine for me.

[JPSchellenberg]

On second thought. I think that every test runs in its own virtual machine. Limiting the parallelism of the tests would not increase resources for one specific test, as every virtual machine has the same amount of resources. Maybe we have to improve our tests.

[sr258]

Ok, I‘ll watch which tests are affected and report back here when I‘ve identified the culprit.

[sr258]

I've moved the integration tests into a different directory and they are now run at a different IC stage with limited workers.

[sr258]

The memory problems have returned, so I'm reopening this issue...

I've played around with this and using npx jest test/JsonStorage.test.ts --detectLeaks (requires weak package to be installed) shows memory leaks in 13 tests. Running npx jest --runInBand --logHeapUsage shows that the heap usage increases with every test, which is an indication of a memory leak. :(

However, I've been unsuccessful so far in finding the cause of this. Even a very simple test like JsonStorage.test.ts seems to have a memory leak, which is really weird, as not much is done there. I've reduced the test to just reading the JSON file from the disk and this still produces a memory leak!

Because of this, I suspect that the leak is (not necessarily) in our code but caused by Jest. A very similar issue is discussed (here)[https://github.com/jestjs/jest/issues/7874] but it seems unresolved. I would still not rule it out that there might also be a memory leak in our code, though.

Temporarily, this problem might be adressed by restricted the parallelism of the tests further (see PR)