Flags (relevant, qualityAssured, ...) cannot be removed persistently from entries

[derTimme]

I'm using JabRef version 4.0 on ArchLinux (Linux 4.13.11-1-arch amd64) with Java 1.8.0_144.

Steps to reproduce:

0. Set "Options" - "Preferences" - "Entry table columns" - "Synchronize with keywords"
1. Create a new empty Biblatex library (or use an existing one, doesn't make a difference for me)
2. Add a new entry (of any type, I tested with article)
3. Add a flag like "relevant" via the entry table (either using the right click menu or the corresponding column)
4. Save the library - the flag gets stored in the keyword field of the entry
5. Remove the flag in the entry table and save the library again
6. The flag is still present in the .bib file and shows up in JabRef when opening that library again

The log just says "Opening <filename>".
The same happens when setting multiple flags on a single entry.

[Edit 1]: Problem only occurs when syncing special fields with keywords.

[Siedlerchr]

Could you please test with the latest version from builds.jabref.org

[derTimme]

Yes - I'm out of time for today, will try it tomorrow. Am 14. November 2017 21:46:25 MEZ schrieb Christoph <notifications@github.com>:

…

Could you please test with the latest version from builds.jabref.org -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #3432 (comment)

[derTimme]

I tried it with JabRef 4.1-dev--snapshot--2017-11-13--master--c1d425a31 and got the same results.

[derTimme]

Currently investigating, maybe I can fix it myself.
It seems like these special fields (as they are called in the code) are represented in two different ways:

• In the running instance, they are represented as additional biblatex fields (e.g. relevance={relevant}), these fields are not shown in the biblatex source tab of the entry editor however.
• In the stored .bib file, they are represented as a list of keywords (in the keywords field)

When opening a database, the keywords field is parsed in org.jabref.logic.importer.OpenDatabase#loadDatabase(File, ImportFormatPreferences) and the "runtime" fields are created.

[derTimme]

Investigated a bit further: The problem only occurs when special fields are synced with the keywords field, if special fields are synced as separate bibtex fields, everything works as expected (note however that the current values of special fields are "lost" when changing this setting, see below).
I updated my initial comment accordingly.

Several things came to my mind:

1. Whether special fields are synced via keywords or via separate fields shouldn't be a global setting, but a property of the library.
2. When changing this setting from keywords to "separate fields", the current values are "lost" (they are still present in the .bib file, but they aren't interpreted by JabRef). This can be quite confusing, but I guess there is no "simple" way of changing this:
   • It certainly isn't sufficient to only "convert" the libraries which are currently opened when the setting is changed
   • Looking for both "styles" of special fields when loading a database would require dealing with "merge conflicts" and if the user selected to store special field values in separate fields and uses the keywords field for something else, these "merge conflicts" would be at least confusing (if the user has to resolve them - I can't think of any other way to handle this correctly).
   • An explicit (user triggered) conversion could work, but the user has to a) use that conversion, and b) this conversion has to be very careful with what can be deleted from an entry (e.g. "relevant" in the keywords field might be a "real" keyword, which the user wants to preserve; but on the other hand, another user would expect the keywords field to be empty after switching to the "separate fields" mode)

[derTimme]

It would be nice to have a little discussion with the more experienced devs here (I started looking at the code two days ago...) about what needs to be done.
Sure, I could try to fix the original issue; the problem should be somewhere in the SpecialFieldsUtils, but these are marked as deprecated, so it would be nice to "properly" fix the handling of special fields.

[derTimme]

The problem is in how org.jabref.model.entry.BibEntry#replaceKeywords and org.jabref.logic.specialfields.SpecialFieldsUtils#exportFieldToKeywords work together:
exportFieldToKeywords calls replaceKeywords with an empty Optional when a special field is "removed" (e.g. 'printed' is set to false). However, replaceKeywords does nothing if it gets an empty Optional as newValue.

[derTimme]

I provided a PR which fixes the original issue.
But the other problems (migration of DBs from "keywords" to "separate fields" and vice verca, the question whether this setting should be DB-specific and the apparently inconsistent handling of keywords) still remain.