OutOfMemoryError when indexing AOSP

[kneep]

Hello.
I got OutOfMemoryError on a computer with 20GB RAM and Ubuntu 17.10.

07:15:27 SEVERE: 90417 successes (21.8%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at org.opensolaris.opengrok.index.IndexDatabase.indexParallel(IndexDatabase.java:1092)
        at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:474)
        at org.opensolaris.opengrok.index.IndexDatabase$1.run(IndexDatabase.java:204)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

OS: Ubuntu 17.10.
Opengrok: 1.1-rc23
Java: OpenJDK 8
Tomcat: 8

BTW, is there any guide for indexing large projects like AOSP?

[tarzanek]

I did index aosp in past with minor modifications, using JAVA_OPTS you can always up this
https://github.com/oracle/opengrok/blob/master/OpenGrok#L379

look at tuning guide:
https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases
I think what is in tuning guide is the lowest

I also had to setup some ignore patterns, alas I don't have them anymore and they are obsolete by now anyways ...

give it a shot using tuning guide, it will work

[vladak]

I would still consider this as a bug. The indexing should scale no matter the code size. Maybe the only exception would be number of threads w.r.t. heap size. Dne čt 19. 4. 2018 8:43 uživatel Lubos Kosco <notifications@github.com> napsal:

…

I did index aosp in past with minor modifications, using JAVA_OPTS you can always up this https://github.com/oracle/opengrok/blob/master/OpenGrok#L379 look at tuning guide: https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases I think what is in tuning guide is the lowest I also had to setup some ignore patterns, alas I don't have them anymore and they are obsolete by now anyways ... give it a shot using tuning guide, it will work — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2074 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACzGDHXo969YsmjnAlrLmfW7bSO4yr8jks5tqDIAgaJpZM4TbArn> .

[vladak]

Maybe you can try to reduce the parallelism (via the -T/--threads option passed to opengrok.jar) and see it that helps. Also, wondering if OpenJDK might behave differently than Oracle JDK.

[tarzanek]

well you're right Vlada,
the out of memory points out to a bug (be it in paralelism or leaking analyser)
AOSP is a good testing data set ...

[idodeclare]

Well I’m not clear from the report if @kneep has raised the Java heap max from the default 2GB.

E.g. I have a 32GB test machine, but I cannot index the set of repos I care about without raising the heap to 10GB.

[tarzanek]

well Idea starts at 756MB of ram and I was puzzled it could index big (java,scala) repos with ASTs, huge jar dependencies etc.
obbviously when I got it to 3GB it behaves much more snappier, but it did work with 756M too (although slow when I hit a certain spot)
so OpenGrok should be able to index with 2GB too (even with smaller) - lucene flushes after certain amounts of data(tunable), so that OOM really points to some leak (though I hoped we fixed all of analyzer leaks)

@kneep would you be able to figure out what files are open/indexed when the indexer crashes? (from logs?)

[amckenna-pinterest]

I'm getting these with -Xmx12g on some repos.

[amckenna-pinterest]

One of the repos has a 519M .json file, could that be related?

[vladak]

Could well be. Try to put the file into ignored list and see what will happen.