The Cache used by ArchUnitRunner should allow some option to be cleaned after the test run

[codecholeric]

At the moment imported classes for a specific set of URLs will stay in memory forever. While this might be useful, if multiple tests in fact want to import the same set of classes, it might be necessary in some scenarios, to empty the cache after each test class is finished.
Proposal: It should be possible to declare

@AnalyzeClasses(cache = CacheMode.FOR_CLASS)

where the default is the current behavior CacheMode.FOREVER.

[codecholeric]

I've reconsidered this, because I think the desired behavior should be pretty much the same for everyone: "If you have the memory, just cache the classes, if you run out of memory, clear them".

So I think, it's better to just make the cache smarter, which should be achievable by using the correct references:

• The cache by class can use strong references, if the runner clears them after the test run. This makes sense, since the scope is clearly defined, while the test class is running, we want to keep the classes, once the test run is over, we want to discard them, with respect to the class as key.
• The cache by locations should use soft references, i.e. hold those references as long as possible, but drop them, once the heap limit is reached. This will enable multiple ArchTests to reuse the imported classes for the same locations, but still prevent successive tests from failing due to low memory.

I did some experiments and with soft references (as opposed to weak references, where the references are dropped at will, even if there is sufficient memory, until the same classes have been imported several times) the behavior seems correct. The Cache might trigger an import for every test, if the heap is small, but only the minimally necessary time, if the heap is big enough. I don't know, how the behavior would be, if the heap is already pretty filled with other dangling objects (i.e. what will be cleared first), but this should be a fair compromise between usability and performance.

Also as far as I can see (happy to hear objections 😉), this would also resolve #44 .

[thmuch]

I can confirm the cache is cleared if there is not enough heap available for the tests that run after ArchUnit!

But: We still see a noticeably slowdown in our build, because the soft references don't seem to be cleared quick enough.

I'm not sure if ArchUnit should be adapted to our special use case any further as the soft ref cache is working. We're back to ArchUnit 0.4 and the old cache clear workaround for the time being. Maybe I can use importClasspath() in BeforeClass (instead of AnalyzeClasses) and run the tests in JUnit methods (instead of using ArchTest fields).

[codecholeric]

Oh, I'm sorry to hear that, I hoped, the issue would be gone with the soft references. I don't think, this is desirable behavior of the cache though, to noticeably slow down consecutive test executions 😉
I'll reopen this issue and look into it, as soon as I have time.
If you have only one test, you could of course import the classes manually in @BeforeClass and throw them away in @AfterClass. It might be better to importPackages(..) though, instead of the whole importClassPath() (since that brings a lot of 'useless' classes, from test infrastructure to the whole JDK classes)
I'll still try to find a solution, in the worst case it's back to an additional configuration to override the caching behavior, for example to completely skip the caching by locations and only cache for the test run (that would be a trivial change and IMHO a valid use case).

[thmuch]

To put this into context: We're running ArchUnit first in a suite of ca. 7500 tests. Ca. 500 of these tests use HSQL, and the first few of these HSQL based tests are slowed down (the slowdown is roughly 1 minute in a total test time of 8 minutes). After the soft refs have been cleared, all tests run with their usual speed.
So, I think your cache is working for most scenarios. After all, the soft refs get cleared and all tests are run successfully.
We've got plenty of options to use the current release of ArchUnit (use importPackages, tune our environment or project setup etc.) It's just we can't use AnalyzeClasses in our setup, but I'm happy with the alternatives.

[codecholeric]

You did try @AnalyzeClasses(packages = ...) though, right? 😉 (since you make it sound, like it's @AnalyzeClasses OR importPackages)
Also there is @AnalyzeClasses(importOptions = ...) where you can fine-tune the URLs to import (you can even implement your own ImportOption, where you can decide exactly about every single URL, if you want to import the class)
But one way or the other, if there are serious drawbacks from the cache, like a noticeable slowdown for consecutive tests, I'll add an option to cache classes only for the test run, if I can't find a better solution.
I would guess, that the GC keeps the soft references up to the last moment, and then suddenly has to perform so much clean up in a single shot, that it has a serious performance impact ☹️

[athkalia]

Hi @codecholeric , I am experiencing some out of memory issues myself when running ArchUnit Tests. Do you have any best practices to suggest overall?

[codecholeric]

@athkalia it's hard to tell without any background. Since you replied to this issue, do you think it's cache related? You should not run out of memory though, since the cache uses soft references.
If you have a lot of classes you might have to increase the heap, there is no way around it. I've had cases where importing 50000 classes would need around 6GB of heap to run arch rules.
So maybe to better help you:

• how many classes are you trying to import?
• how much heap did you configure?
• is there a difference between using @ArchTest style and a plain classFileImporter.import...(..)?
• are you running out of memory during the import or while checking rules?

[athkalia]

HI @codecholeric, I think your answer answers my question. It's a very big project and after adding ArchUnit tests I had to increase the heap size from about 400MB to 2GB and that seemed like a lot. But if you have experienced that and even more then I rest my case. I am importing around 15,000 classes, and that is with classFileImporter.import...(..).