limit addFile() processing time

[vladak]

When indexing bunch of projects, the indexer (based on 1.13.4) had a trailing worker thread that was spinning on the CPU for inordinate amount of time (minutes) while all the other files were finished. The stack trace looked like this (comes from my branch when experimenting with the fix for #4549 so thread name and line numbers do not match):

"OpenGrok-index-worker-48" #48 prio=5 os_prio=0 cpu=773733,68ms elapsed=1140,14s tid=0x00007f3648023000 nid=0x2134a7 runnable  [0x00007f364e5fd000]
   java.lang.Thread.State: RUNNABLE
	at org.opengrok.indexer.analysis.javascript.JavaScriptSymbolTokenizer.yylex(JavaScriptSymbolTokenizer.java:1645)
	at org.opengrok.indexer.analysis.JFlexTokenizer.incrementToken(JFlexTokenizer.java:106)
	at org.apache.lucene.index.IndexingChain$PerField.invertTokenStream(IndexingChain.java:1167)
	at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1145)
	at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:697)
	at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:576)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:243)
	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:421)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1551)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1836)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1476)
	at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:1278)
	at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$9(IndexDatabase.java:1920)
	at org.opengrok.indexer.index.IndexDatabase$$Lambda$370/0x00000008401e7c40.call(Unknown Source)

Specifically, the file in question came from the https://hg.mozilla.org/mozilla-central , it was testing/modules/sinon-7.2.7.js which contains long lists of numbers. Such files are known to cause problems. Ideally the analyzers should skip/truncate the files with huge number of terms.

The indexer eventually finished, however there should be some time limit for addFile() or writer.addDocument() processing. There is already timeout enforced for xref production via the XrefWork class.

[vladak]

Rather than trying to terminate and cleanup the writer.addDocument() call it would be easier to check the term count in certain fields and skip the document altogether if the count is above given threshold.

Another idea is to trim the number if terms.

[saurav-max]

Review the code of the JavaScriptSymbolTokenizer class to understand what it's doing and where the error might be occurring. Look for any potential bugs, errors, or exceptions that could be causing the issue.