Limiting history indexing to select repositories only

[aerofeev2k]

Hi,

We've recently added a few GIT directories to the already checked out and indexed tree (that used to have only CVS before), and since then producing an index of history of changes went downhill. What used to take an hour is now running for days.

It seems that for every source file OpenGrok is using JGIT to get the history of changes, and that's taking 3-5s per file. I can see in lsof/strace how java is reading huge GIT pack files all the time, I can see in jstack how it's decompressing them and then walking the revision tree (a typical stack is below).

Not passing -H to the indexer helps things tremendously, but we lose the index of CVS history as well, and that used to work just fine before.

Is there a way to prevent OpenGrok from indexing history of all GIT repositories or of select repositories (specified by path) while still preserving index of CVS repositories in one big combined tree?

We've tried a few things, but short of not passing -H, nothing seemed to work. E.g. we've tried using --repository command line option to explicitly list only CVS repositories, or setting historyEnabled to false for GIT repositories in configuration.xml, but OpenGrok still tries indexing history for files from the GIT section of the checked out tree.

java.lang.Thread.State: RUNNABLE
        at org.eclipse.jgit.treewalk.TreeWalk.next(TreeWalk.java:820)
        at org.eclipse.jgit.revwalk.TreeRevFilter.include(TreeRevFilter.java:170)
        at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:108)
        at org.eclipse.jgit.revwalk.BlockRevQueue.<init>(BlockRevQueue.java:40)
        at org.eclipse.jgit.revwalk.FIFORevQueue.<init>(FIFORevQueue.java:37)
        at org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:133)
        at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:591)
        at org.eclipse.jgit.revwalk.RevWalk.nextForIterator(RevWalk.java:1526)
        at org.eclipse.jgit.revwalk.RevWalk.iterator(RevWalk.java:1550)
        at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:520)
        at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:476)
        at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:443)
        at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:438)
        at org.opengrok.indexer.history.FileHistoryCache.get(FileHistoryCache.java:683)
        at org.opengrok.indexer.history.HistoryGuru.getHistoryFromCache(HistoryGuru.java:248)
        at org.opengrok.indexer.history.HistoryGuru.getHistory(HistoryGuru.java:309)
        at org.opengrok.indexer.history.HistoryGuru.getHistory(HistoryGuru.java:215)
        at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:601)
        at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:831)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$4(IndexDatabase.java:1361)
        at org.opengrok.indexer.index.IndexDatabase$$Lambda$324/0x00000008002f3440.apply(Unknown Source)

[vladak]

When creating history cache for Git repositories (or any repository type which supports retrieving history per directory) the history for the top level directory of given repository is taken first and then inverted in memory to arrive to per file history. This is done for all repositories for given project. This happens during the first stage of indexing and avoids the costly per file history retrieval in the second phase, if successful.

Which version do you run ? Since 1.12.21 the failure to generate history cache during the first phase of indexing for such repositories will cause the respective project indexing to fail (before entering the 2nd phase), unless overridden by the --ignoreHistoryCacheFailures indexer option. Before this change if the history cache failed to be generated during first phase, the history was retrieved in the second phase, per file - the behavior you see. Anyhow, the indexer logs should provide some clue for the failure of the history cache generation in the first phase.

As for controlling history cache per project/repository I will have to check.

[aerofeev2k]

This is 1.7.30.

Could the failure to generate index for the top-level directory be related to submodules? The way how the GIT tree is structured is that there's a top-level local GIT repository that has no real files in it, and then, in various subdirectories of that repository, the actual GIT trees are plugged as submodules. I do see that all these submodules have been discovered and recorded by OpenGrok as separate GIT repositories in configuration.xml. And there's this error in the beginning:

WARNING: An error occurred while creating cache for /opengrok/src/tree/git (GitRepository)
java.lang.NullPointerException
        at org.eclipse.jgit.lib.ObjectId.<init>(ObjectId.java:258)
        at org.eclipse.jgit.lib.ObjectIdOwnerMap$Entry.<init>(ObjectIdOwnerMap.java:335)
        at org.eclipse.jgit.revwalk.RevObject.<init>(RevObject.java:31)
        at org.eclipse.jgit.revwalk.RevCommit.<init>(RevCommit.java:120)
        at org.eclipse.jgit.revwalk.RevWalk.createCommit(RevWalk.java:1631)
        at org.eclipse.jgit.revwalk.RevWalk.lookupCommit(RevWalk.java:878)
        at org.opengrok.indexer.history.GitRepository.accept(GitRepository.java:455)
        at org.opengrok.indexer.history.BoundaryChangesets.getBoundaryChangesetIDs(BoundaryChangesets.java:88)
        at org.opengrok.indexer.history.RepositoryWithPerPartesHistory.doCreateCache(RepositoryWithPerPartesHistory.java:85)
        at org.opengrok.indexer.history.Repository.createCache(Repository.java:426)
        at org.opengrok.indexer.history.HistoryGuru.createCache(HistoryGuru.java:593)
        at org.opengrok.indexer.history.HistoryGuru.lambda$createCacheReal$3(HistoryGuru.java:639)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

[vladak]

This is 1.7.30.

Could the failure to generate index for the top-level directory be related to submodules? The way how the GIT tree is structured is that there's a top-level local GIT repository that has no real files in it, and then, in various subdirectories of that repository, the actual GIT trees are plugged as submodules. I do see that all these submodules have been discovered and recorded by OpenGrok as separate GIT repositories in configuration.xml. And there's this error in the beginning:

Could be the #4608 recently fixed in 1.13.13

[aerofeev2k]

No, I'm afraid, it's something different. I've added a dummy file to the top-level repository, checked it in, etc, and the NullPointerException didn't go away.

I then completely removed the /opengrok/src/tree/git directory, gave up on submodules and cloned individual subtrees in separate subdirectories below /opengrok/src/tree/git. Updated config and tried indexing again. The exception went away, but still OpenGrok was reopening and reinflating packs for every file.

[vladak]

As for the per project override, the following read-only configuration overrides the default enabled history:

<?xml version="1.0" encoding="UTF-8"?>
<java version="11.0.4" class="java.beans.XMLDecoder">
 <object class="org.opengrok.indexer.configuration.Configuration" id="Configuration0">

   <void property="projects">
     <void method="put">
      <string>foo</string>
      <object class="org.opengrok.indexer.configuration.Project">
       <void property="handleRenamedFiles">
        <boolean>false</boolean>
       </void>
       <void property="historyEnabled">
        <boolean>false</boolean>
       </void>
       <void property="name">
        <string>foo</string>
       </void>
       <void property="path">
        <string>/foo</string>
       </void>
       <void property="mergeCommitsEnabled">
        <boolean>false</boolean>
       </void>
      </object>
     </void>
   </void>

 </object>
</java>

When the indexer is run e.g. like this:

-P
-S
-H
-R
/var/opengrok/etc/readonly_configuration-per_project_history.xml
-s
/var/opengrok/src.perproj
-d
/var/opengrok/data.perproj
-W
/var/opengrok/etc/configuration.xml

the history cache creation will be skipped for project foo and will be created for the rest of the projects. One can invert it and run the indexer without -H option and override this in positive way per project.

This works for the initial indexing, there might be some caveats when using the written configuration as a read-only for the subsequent indexing.