Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index gets stale #17

Open
Bubu opened this issue Mar 2, 2021 · 4 comments
Open

Index gets stale #17

Bubu opened this issue Mar 2, 2021 · 4 comments

Comments

@Bubu
Copy link
Contributor

Bubu commented Mar 2, 2021

I'm not exactly sure what I'm doing wrong but I keep running in situations where newer mails seem not searchable unless I explicitly call

doveadm fts rescan -u user@example.com
doveadm index -u user@domain -q '*'

for that user.

Here's my config:

mail_plugins = $mail_plugins fts fts_elastic

plugin {
  fts = elastic
  fts_elastic = url=http://localhost:9200/m/ bulk_size=5000000 refresh=fts

# no indexes new emails when user make search
# yes indexes every email when delivered
fts_autoindex = no
fts_autoindex_exclude = \Junk
fts_autoindex_exclude2 = \Trash
}

Any ideas on how to debug this?

@filiphanes
Copy link
Owner

Elastic you can set refresh.interval https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-refresh-interval-setting
in that time new documents are not searchable, if you have this time set long, it may appear that new email are not indexed

Since you have fts_autoindex = no emails are indexed on imap search command and on this first search they may not be visible, because they are queried right after indexing, you can try to wait and search again after refresh interval

You can try setting:

  • fts_autoindex = yes this will index each email right after delivery
  • fts_elastic = refresh=index this will refresh elastic right after indexing so new emails should be immediately searchable, hmm, but you have refresh=fts, so it should refresh elastic on each search, maybe it does not.

If new emails are not searchable even after several minutes after delivery or last imap search command then there is probably some bug.

@Bubu
Copy link
Contributor Author

Bubu commented Mar 5, 2021

refresh.interval isn't explicitly set, so I guess that defaults to 1s.

Argh, this is so frustrating to debug, because whenever I want to look at it, it seems to be all working and then a month or so later I'll notice missing emails in my searches again :-/.

I remember changing fts_autoindex = yes to no (together with refresh=fts) in the past because that seemed to work better with actually keeping the index up to date. But this has been a while and I haven't done any recent tests.

(I'll keep the issue updated if I discover anything more related to this. Might be a bit until this reoccurs.)

@filiphanes
Copy link
Owner

Are missing emails only the latest, or there are some "holes"?

@Bubu
Copy link
Contributor Author

Bubu commented Mar 5, 2021

The issues I've noticed to far have always been that emails after a cutoff time weren't indexed. Like "everything newer than 1 month ago" I usually notice if only after a month because K9-mail does a local search in the most recent 200 emails anyway...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants