Remove camel case tokenization from repo indexer

[guillep2k]

This PR is intended to address #5694. Currently, the repository indexer breaks camel cased identifiers into tokens, so thisMethodName is indexed separately as "this", "method" and "name". Therefore, if one searches for "thismethodname", it's never found (and very confusing).

This PR adds the REPO_INDEXER_CAMEL_CASE setting flag to the [Indexer] section. If it's true (default), current behaviour continues. If it's set to false, camel cased identifiers are no longer separated into words.

The only important thing to clarify is that it's imperative to delete any previous indexes from REPO_INDEXER_PATH after changing this setting to force the indexes to be rebuilt (with no gitea instance running). Otherwise the indexes will contain mixed criteria. I was pondering about performing the index drop automatically (by adding a flag to some table in the database to check how was it set previously), but I didn't want to pollute the code too much. Another possibility is to add a new cmd to gitea:
gitea drop-repo-indexes
or something like that. You guys tell me. In any case, there seems to be no standard (API) way to drop indexes from bleve other than delete the files manually.

One more thing: regarding the index size, I haven't noticed much difference. A test repository with C code gave me a 26,337,280 byte index file with the default behaviour, and 26,869,760 without camel casing (about 2% increase). Mileage might vary, though.

[codecov-io]

Codecov Report

Merging #7733 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7733      +/-   ##
==========================================
- Coverage   41.32%   41.31%   -0.01%     
==========================================
  Files         473      473              
  Lines       63828    63828              
==========================================
- Hits        26375    26369       -6     
- Misses      34014    34021       +7     
+ Partials     3439     3438       -1

Impacted Files	Coverage Δ
modules/indexer/repo.go	70.22% <100%> (ø)	⬆️
models/unit.go	62.16% <0%> (-5.41%)	⬇️
routers/repo/view.go	42.23% <0%> (-1.02%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 026696b...2e4479c. Read the comment docs.

[silverwind]

Does this result in a search for thisCameCaseName still finding it? If so, I don't think we even need the option and make it the default behaviour.

gitea drop-repo-indexes

That sounds like a good command to have. I'd make a gitea rebuild-indexes which could drop both issue and repo indexes at once and rebuild them. Could also have that in the admin UI.

[lafriks]

I also think this does not need an option

[guillep2k]

The reasons I pondered when I added the app.ini option were:

• There must be a number of users that actually rely on the current behaviour, or became used to it.
• Changing the tokenizer involve rebuilding the indexes: in large installations this may increase downtime when upgrading, produce a peak in disk I/O, change memory requirements, etc.
• There's a slight risk that insufficient disk space would produce problems when rebuilding; perhaps this should be a user-controlled risk.
• For those who are already used to this behaviour, it could be in fact useful: searching for "monster" will bring up insertMonsterRecords, updateMonsterRecords and deleteMonsterRecords. Sometimes looking for "insert monster records" brought me up insertMonsterRecords, but it was quite unreliable, I don't know why.

Anyway, if the change becomes mandatory, dropping the old indexes could be a migration step. Should I do it that way?

[lafriks]

Rebuilding index would probably be not good idea for migration step, this should probably be done manually

[lafriks]

Otherwise I think we should change as current behavior is quite broken

[guillep2k]

But but.... wouldn't that make upgrading a little more difficult? Perhaps if we add the admin UI option as @silverwind suggested, then the procedure is a little less difficult.

[lafriks]

That could probably be different PR

[zeripath]

I agree with @lafriks and @silverwind CamelCase is so common and the current behaviour just doesn't work nicely that I think we should make this the default.

The trouble with making this an automatic migration step is precisely that this will take too long to run so I think a rebuild the indexes command would be best. I think the repository indexing architecture isn't necessarily correct - I know that @lunny was working on making it more generic - but I wonder if we need to farm at least some of the index querying out to a different gitea process or other process - similar to how we use a db for querying data.

So to summarize:

• Drop the configuration - it doesn't break old indices - and newly indexed things will be better.
• Make a command that can rebuild the indices offline.

[guillep2k]

So, if I get your idea correctly, you propose:

• Simplify this PR, leaving only the change in tokenizer options, fixed to do not camel case.
• A new PR, adding a "rebuild indexes" for issues and repos to the admin UI.

If you confirm, I'll go that way.

[guillep2k]

I agree with @lafriks and @silverwind CamelCase is so common and the current behaviour just doesn't work nicely that I think we should make this the default.

The trouble with making this an automatic migration step is precisely that this will take too long to run so I think a rebuild the indexes command would be best. I think the repository indexing architecture isn't necessarily correct - I know that @lunny was working on making it more generic - but I wonder if we need to farm at least some of the index querying out to a different gitea process or other process - similar to how we use a db for querying data.

So to summarize:

* Drop the configuration - it doesn't break old indices - and newly indexed things will be better.

* Make a command that can rebuild the indices offline.

OK, I'll go for the command, then.

[guillep2k]

I had to retain the following:

import _ "github.com/blevesearch/bleve/analysis/token/camelcase"

Otherwise, if the user doesn't delete the previous indexes, the instance would not start up. That's because the existing indexes were created with the camel case tokenizer and the bleve library was attempting to instantiate it.

Now, the new effect of this PR is the following:

• Existing indexes will continue to break up camel case identifiers into tokens (that's something that's set on creation).
• New repositories will have indexes without camel case tokenization.
• If any indexes are deleted (e.g. for rebuild), new indexes will be created without camel case tokenization.

[guillep2k]

Hold on! I've found an interesting piece of code:

// InitRepoIndexer initialize repo indexer
func InitRepoIndexer(populateIndexer func() error) {
	var err error
	repoIndexer, err = openIndexer(setting.Indexer.RepoPath, repoIndexerLatestVersion)
...
}

func openIndexer(path string, latestVersion int) (bleve.Index, error) {
...
	if metadata.Version < latestVersion {
		// the indexer is using a previous version, so we should delete it and
		// re-populate
		return nil, os.RemoveAll(path)
	}
...
}

repoIndexerLatestVersion is currently set to 1. Setting it to 2 would force all repository indexes to be rebuilt upon restart. This opens two paths for us:

1. We go the way we were going; indexes will be rebuilt by the users when they see fit (I'm working on the rebuild-indexes cmd).
2. We increase repoIndexerLatestVersion to 2 and force automatic rebuild of repo indexes when the user upgrades.

Just so we know all the options.

[lafriks]

We could probably increase version, I would say go for it