Code/repo search #2582
Code/repo search #2582
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2582 +/- ##
==========================================
- Coverage 27.2% 26.86% -0.34%
==========================================
Files 88 89 +1
Lines 17374 17596 +222
==========================================
+ Hits 4726 4727 +1
- Misses 11962 12183 +221
Partials 686 686
Continue to review full report at Codecov.
|
tboerger
added
the
lgtm/need 2
lunny
added
the
type/feature
|
Issue related changed should be a separate PR. |
routers/routes/routes.go
Outdated
| @@ -642,6 +642,7 @@ func RegisterRoutes(m *macaron.Macaron) { | |||
| m.Group("/:username/:reponame", func() { | |||
| m.Get("/stars", repo.Stars) | |||
| m.Get("/watchers", repo.Watchers) | |||
| m.Get("/search", repo.Search) | |||
models/migrations/v44.go
Outdated
|
|
||
| func addLastIndexSyncToRepository(x *xorm.Engine) error { | ||
| type Repository struct { | ||
| LastIndexSync string |
There was a problem hiding this comment.
From name it's not really understandable what is stored here. Time? Commit Sha?
models/models.go
Outdated
| @@ -150,7 +150,9 @@ func LoadConfigs() { | |||
|
|
|||
| sec = setting.Cfg.Section("indexer") | |||
| setting.Indexer.IssuePath = sec.Key("ISSUE_INDEXER_PATH").MustString("indexers/issues.bleve") | |||
| setting.Indexer.RepoPath = sec.Key("REPO_INDEXER_PATH").MustString("indexers/repos.bleve") | |||
There was a problem hiding this comment.
Need to check if path is relative and make it absolute if is.
ethantkoenig
force-pushed
the
repo_search
branch
from
77c5f98
to
c0dc8dc
Compare
ethantkoenig
force-pushed
the
repo_search
branch
2 times, most recently
from
b86a1b4
to
76eecf7
Compare
|
Rebased to resolve merge conflict |
models/models.go
Outdated
| @@ -335,3 +340,14 @@ func DumpDatabase(filePath string, dbType string) error { | |||
| } | |||
| return x.DumpTablesToFile(tbs, filePath) | |||
| } | |||
|
|
|||
| func absolutePath(path string) string { | |||
There was a problem hiding this comment.
How about to move this to util package.
models/repo.go
Outdated
| ForkID int64 `xorm:"INDEX"` | ||
| BaseRepo *Repository `xorm:"-"` | ||
| Size int64 `xorm:"NOT NULL DEFAULT 0"` | ||
| IndexCommitSha string |
There was a problem hiding this comment.
Maybe a standalone table to store RepoID, BranchName, LastCommitSHA, so that we can support search for an extra branch.
|
And maybe put the search input box on |
|
When I click the line number of search result page. it will show |
|
Added line number links, hopefully I will be able to address the rest of your suggestions soon |
|
@ethantkoenig I think only left one from me: Maybe a standalone table to store RepoID, BranchName, LastCommitSHA, so that we can support search for an extra branch. |
|
@lunny searching in every branch could had too much work or maybe (in other PR) we should let the user select the branch with the option of all but by default only the default branch. |
|
@ethantkoenig @sapk only support one branch is also OK. I only want the new struct of the table for flexible design. |
ethantkoenig
force-pushed
the
repo_search
branch
2 times, most recently
from
a58f214
to
df9bccb
Compare
|
@lunny Moved indexer commit shas to a separate table. In the future, the table can be extended to support multiple branches. |
ethantkoenig
force-pushed
the
repo_search
branch
from
df9bccb
to
a7dd218
Compare
|
Edit: never mind, I just forgot to add the new table to |
|
@ethantkoenig for each run I think new DB instance is created. Also I don't think migrations are run for integration tests |
ethantkoenig
force-pushed
the
repo_search
branch
from
a7dd218
to
b010cca
Compare
Indexed search of repository contents (for default branch only)
|
@lafriks I don't think so (this code doesn't directly touch merging), but I'll take a closer look when I get the chance |
|
@ethantkoenig Where is the option setting to enable/disable repo indexer? Only in app.ini? |
|
@webjoel in app.ini file |
|
@ethantkoenig @lafriks In try gitea demo it would be better to come true by default, or not? |
|
@webjoel I'm not sure our demo server disk resources allow that as than we would need to create some kind of automatic cleanup there |
|
Okay, I imagined something like that. Is that the way it is, disabled by default, the functionality was half hidden. |
|
@webjoel For my test repositories that total in 121 MB index size was 735 MB |
|
@lafriks In a production environment is have to be very careful. It's important, but the use of this feature should be well evaluated. can be done as you said, deleting the index every day midnight for example. Currently has how to know the space used in the demo with repositories? |
|
This is a bit later but I'm experiencing somewhat of a memory leak when initializing the indexer. |
|
In case someone else comes here from the 1.3.0-rc1 release notes. I've been running to my |
|
@HenrikBengtsson it was intentionally left disabled by default as code search index database takes a lot of disk space - about 6 times more than repository size. So use with caution. |
|
@lafriks, yes I think I understood that. I'm my case, since I was coming from an older version, I didn't even have a About large search DB: I'll use with care. Have you had any discussion on being able to enable/disable search per user/org or repository? PS. I'm a first-timer with Gitea so I haven't been around for other x.y.z -> x.y+1.0 updates. Does Gitea make it easy to do such updates? For instance, will Gitea have to be "reinstalled" such that an up-to-date |
|
@HenrikBengtsson such discussion would be better in discord chat but in short yes, update is just replacing binary and restarting gitea. No need to change app.ini, missing entries will just have default values. So it's not always disabled if entry is missing, just in this case. Per org/repo search option it's something we will need but noone is working on this currently |
|
assume i have a linux kernel source tree and some other large git trees, indexing might be too heavy(for disk and cpu), can I designate certain git repos for indexing instead of indexing all of the git repos? it will be nice if I can choose what to index, or even better, when to index(e.g. indexing at mid-night via cron). |
|
@laoshaw That's what I think next step to do.
|
|
One downside is that, the search doesn't show all found results in single file. For example, if a file includes Can this be improved? Should I open another issue for that? |
|
@xak2000 GitHub only shows the first couple of results, not all. |
|
@techknowlogick Yes, you are right. I'm not sure how it determine how much, though. For example this search shows "Showing the top five matches" for one of the files: But on other files it shows two or three. Maybe it is percent of total matches in the file or something. But anyway the one is not enought I think. |
|
Please open a separate issue, though, otherwise if the problem is left here then we won't be able to track it properly :) (edit: yes, I realise I'm a bit late on this, sorry about that >_>) |
|
Hello! How much load disk subsytem on server, if I enable indexer and set path for store his files ? |
|
The code index will spend 3x of the codes disk according to the usage experiences. |
|
How can I search for code across all repositories in an organization? |
|
This is codes search, but not repositories search. The only place you can use it is in |
yes im enable Thank you, sorry there is no organization filter. But it suits me |
Indexed search of repository contents (for default branch only). Implements #1102.
Details/features:
Current Screenshots:
Old Screenshots (for comparison):