Indexing failed #4303

jeevavm · 2021-03-16T13:55:03Z

jeevavm
Mar 16, 2021

We have installed Opengrok 1.6.4, OpenJDK 15.02 and apache tomcat 10.0.2

While indexing multiple projects in the source folder we are getting below errors and the indexing skipped for many of the files in some projects and those projects are not listed in Opengrok search.

Mar 15, 2021 3:00:05 PM org.opensolaris.opengrok.index.IndexDatabase addFile
INFO: Skipped file '/logger.d.ts' because the analyzer didn't understand it.
Mar 15, 2021 3:00:05 PM org.opensolaris.opengrok.index.IndexDatabase indexDown
INFO: Progress: 25,441 (68.238%)
Mar 15, 2021 3:00:05 PM org.opensolaris.opengrok.index.IndexDatabase addFile
INFO: Skipped file '/ResourceManager.cs' because the analyzer didn't understand it.
Mar 15, 2021 3:00:05 PM org.opensolaris.opengrok.index.IndexDatabase addFile
INFO: Skipped file '/CaseTestData.sql' because the analyzer didn't understand it.

vladak · 2021-03-16T14:04:59Z

vladak
Mar 16, 2021
Maintainer

I don't think skipped files lead to failure of the indexer as a whole. After all, these exceptions are swallowed:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexDatabase.java

Lines 776 to 786 in 4271459

    
           } catch (Exception e) { 
        
               LOGGER.log(Level.INFO, 
        
                       "Skipped file ''{0}'' because the analyzer didn''t " 
        
                       + "understand it.", 
        
                       path); 
        
               if (LOGGER.isLoggable(Level.FINE)) { 
        
                   LOGGER.log(Level.FINE, "Exception from analyzer " + 
        
                       fa.getClass().getName(), e); 
        
               } 
        
               cleanupResources(doc); 
        
               return;

and the impact is on single document, not whole index.

Could you run the indexer with the FINE logging level ? This should shed more light on these errors.

0 replies

jeevavm · 2021-03-16T14:27:45Z

jeevavm
Mar 16, 2021
Author

We are getting below error now,

Mar 16, 2021 2:25:58 PM org.opensolaris.opengrok.index.IndexDatabase$1 run
SEVERE: Problem updating lucene index database:
org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource: ChecksumIndexInput(NIOFSIndexInput(path="/var/opengrok/data/index/tools/segments_a"))): 10 (needs to be between 0 and 1)
at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:148)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:329)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:404)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:694)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:400)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:746)
at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:330)
at org.opensolaris.opengrok.index.IndexDatabase$1.run(IndexDatabase.java:168)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)

0 replies

vladak · 2021-03-16T16:58:07Z

vladak
Mar 16, 2021
Maintainer

Um, we dropped the org.opensolaris.opengrok prefix long time ago so you must be running some really old indexer. Definitely not 1.6.4.

0 replies

jeevavm · 2021-03-21T17:52:03Z

jeevavm
Mar 21, 2021
Author

Sorry, my bad my script pointing the old opengrok jar before. Now the indexing job getting struck after the below log, Can please help us with how to resolve this.

2021-03-21 14:07:34.745+0000 FINER t1 SystemPropertiesConfigurationModel.getProperties: System properties configuration provider not allowed
2021-03-21 14:07:34.746+0000 FINE t1 ClientExecutorProvidersConfigurator.lookupManagedScheduledExecutorService: null
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.glassfish.jersey.client.ClientExecutorProvidersConfigurator.lookupManagedScheduledExecutorService(ClientExecutorProvidersConfigurator.java:174)
at org.glassfish.jersey.client.ClientExecutorProvidersConfigurator.init(ClientExecutorProvidersConfigurator.java:124)
at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:452)
at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:317)
at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:814)
at org.glassfish.jersey.client.ClientRequest.getClientRuntime(ClientRequest.java:176)
at org.glassfish.jersey.client.ClientRequest.getInjectionManager(ClientRequest.java:567)
at org.glassfish.jersey.client.JerseyWebTarget.onBuilder(JerseyWebTarget.java:364)
at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:192)
at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:36)
at org.opengrok.indexer.configuration.RuntimeEnvironment.writeConfiguration(RuntimeEnvironment.java:1422)
at org.opengrok.indexer.index.Indexer.sendToConfigHost(Indexer.java:1149)
at org.opengrok.indexer.index.Indexer.main(Indexer.java:378)
Caused by: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or in an application resource file: java.naming.factory.initial
at java.naming/javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:691)
at java.naming/javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305)
at java.naming/javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:342)
at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
... 16 more

2021-03-21 14:07:34.746+0000 CONFIG t1 ExtendedLogger.config: Selected ExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.client.ClientAsyncExecutor] annotation.
2021-03-21 14:07:34.747+0000 CONFIG t1 ExtendedLogger.config: Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.client.ClientBackgroundScheduler] annotation.
2021-03-21 14:07:34.751+0000 CONFIG t1 HttpUrlConnector.: Restricted headers are not enabled using [sun.net.http.allowRestrictedHeaders] system property (setting only takes effect on connections created after the property has been set/changed).
2021-03-21 14:07:34.753+0000 FINEST t1 HttpURLConnection.plainConnect0: ProxySelector Request for http://localhost:8080/source/api/v1/configuration?reindex=true
2021-03-21 14:07:34.754+0000 FINEST t1 HttpURLConnection.plainConnect0: Proxy used: DIRECT
2021-03-21 14:07:34.755+0000 FINE t1 HttpURLConnection.writeRequests: sun.net.www.MessageHeader@8dfe9217 pairs: {PUT /source/api/v1/configuration?reindex=true HTTP/1.1: null}{Content-Type: application/xml}{User-Agent: Jersey/3.0.1 (HttpUrlConnection 15.0.2)}{Host: localhost:8080}{Accept: text/html, image/gif, image/jpeg, *; q=.2, /; q=.2}{Connection: keep-alive}{Content-Length: 44036}

Please find the below logs from tomcat,

15-Mar-2021 14:02:13.315 WARNING [Catalina-utility-2] org.opengrok.indexer.framework.PluginFramework.reload Plugin directory not found or not readable: /opengrok/data/../plugins. All requests allowed.
15-Mar-2021 14:02:14.024 WARNING [Catalina-utility-2] org.opengrok.web.WebappListener.check_index Project ui index check failed, marking as not indexed
org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/opengrok/data/index/ui/segments_4"))): 1 (needs to be between 7 and 10). This version of Lucene only supports indexes created with release 6.0 and later.
at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:213)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:309)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:461)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:458)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:672)
at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:463)
at org.opengrok.indexer.index.IndexCheck.checkDir(IndexCheck.java:138)
at org.opengrok.web.WebappListener.check_index(WebappListener.java:127)
at org.opengrok.web.WebappListener.contextInitialized(WebappListener.java:108)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4665)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5126)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:946)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1792)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:757)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:426)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1565)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

The command we are using to index the projects is below,

java -Djava.util.logging.config.file=/opengrok/etc/logging.properties -jar /opengrok/dist/lib/opengrok-1.6.0.jar -i *.js -r on -c /usr/local/bin/ctags -s /opengrok/src -d /opengrok/data -H -P -S -G -W /opengrok/etc/configuration.xml -U http://localhost:8080/source

0 replies

vladak · 2021-03-22T08:38:42Z

vladak
Mar 22, 2021
Maintainer

If the indexer is indeed stuck (i.e. the process is still present however not progressing), could you capture the stack traces of the threads using jstack and attach it ?

The java.naming.factory.initial exception is innocuous. The latter exception IndexFormatTooOldException says that you have too old index for the project called ui. You need to wipe it out and reindex from scratch.

In overall, it would be handful to know what is it you are trying to do. Upgrading from old OpenGrok version to new one ?

0 replies

jeevavm · 2021-03-22T13:37:17Z

jeevavm
Mar 22, 2021
Author

We have migrating opengrok from old version to new version. We have installed new version opengrok in new server and while coping the scripts files new server we have also copied the old lib files. Our scripts pointed the old jar fils instead of new opengrok jar file. We have fixed that issue now.

Please fins the stack traces of the threads below,

"ForkJoinPool-1-worker-55" #411 daemon prio=5 os_prio=0 cpu=575.98ms elapsed=80.55s tid=0x00007fa5a8043810 nid=0x1cc4 waiting on condition [0x00007fa590ce7000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@15.0.2/Native Method)
- parking to wait for <0x000000008992e578> (a java.util.concurrent.ForkJoinPool)
at java.util.concurrent.locks.LockSupport.park(java.base@15.0.2/LockSupport.java:211)
at java.util.concurrent.ForkJoinPool.runWorker(java.base@15.0.2/ForkJoinPool.java:1633)
at java.util.concurrent.ForkJoinWorkerThread.run(java.base@15.0.2/ForkJoinWorkerThread.java:183)