huge memory usage on big library (> 1000 items)

[ghost]

JabRef version

5.9 (latest release)

Operating system

Windows

Details on version and operating system

Windows 10 22H2

Checked with the latest development build

• I made a backup of my libraries before testing the latest development version.
• I have tested the latest development version and the problem persists

Steps to reproduce the behaviour

When opening a big library (45974 items), the memory usage of JabRef goes up to 1.6 GB RAM .
Is this normal or are there memory problems?

When opening 2 big libraries at the same time (45974 items) and (134352 items), the memory usage is at a whopping 2.4 GB RAM.

JabRef running without any library uses 960 MB RAM on my Windows PC.

• So it goes from 960 MB -> 1.6 GB for a library 45974 items
• it goes from 1.6 GB -> 2.4 GB for another library of 134352 items (opened at the same time)

Also when those 2 big libraries are opened at the same time, JabRef becomes slow when scrolling the entry list of the library. When JabRef is running some minutes on the libraries, the slow scrolling dissapears after some time

Is JabRef not designed for those kind of big libraries or is this a problem in the code I found?

Attached the 2 databases

games.zip

Appendix

...

Log File

Paste an excerpt of your log file here

[ghost]

In 5.10, the memory usage got up with another 200 MB.

After loading both databases, I have 2.6 GB RAM usage instead of 2.4 GB RAM on version 5.9.
So for those 2 specific database, the development version uses more memory then version 5.9.

I also didn't notice any speed improvement in loading/scrolling the items of the library.

[ThiloteE]

This is not the first time this has come up. If you go through issues with the label "performance", you will find a few things to look out for.

In most cases, using lots of RAM is not something that impacts performance, as RAM is faster than your hard-drive. It is also possible to cache the results of certain computing processes into the RAM instead of computing them over and over, again and again and this can increase performance.

Yes, JabRef uses a lot more RAM than other programms in comparison. This is partly by design and partly, because JabRef has been around for a long time and the code base is pretty big, so there is room for optimization. Also, JabRef is written in Java and Java by design uses lots of RAM https://en.wikipedia.org/wiki/Java_performance#Memory_use.

In general, as long as the RAM usage stays roughly constant, there is no problem. If your RAM usage constantly increases, without ever stopping and without you doing anything, then this would be a memory leak and would be a bug and needed to be fixed.

I am more concerned about JabRef becoming slow, when scrolling the entry list of the library.

[ThiloteE]

If I may ask, what is your current hardware? What is the name of the processor (CPU), motherboard, RAM and what kind of harddrive do you use (SSD vs HDD; pcie, m.2 or sata)? If you do not know, you can simply search for "Systeminformation" under Windows > Start. I suspect your CPU might hit its limit.

If you have a huge library, for now, there are only a few things you can do:

1. Try to find performance bottlenecks that are reproducible (e.g. enabling, disabling a preference) and reporting it here
2. Disable or do not use all known features that are known to have an impact on performance. Some are listed here: https://docs.jabref.org/faq#q-i-have-a-huge-library.-what-can-i-do-to-mitigate-performance-issues
3. Get a better and faster computer.

[ThiloteE]

By the way, since I know you use regex for your groups, it could also be that those slow down JabRef, as RegEx is known to be very slow.

[ghost]

By the way, since I know you use regex for your groups, it could also be that those slow down JabRef, as RegEx is known to be very slow.

In this libraries no group is made when I loaded it. The RAM goes up when loading library but then it stays the same.

Core i7 4.2 ghz, 32 Gb RAM, pcie samsung M2 2 TB, geforce RTX 3070

It's a gaming pc and It's never slow anywhere. I think it must be jabref

[Siedlerchr]

A high ram usage is per se nothing bad. JabRef loads all entries in memory. As long as the memory stays more or elss constant it's fine.

[koppor]

This especially refs #8906 and #7567

[calixtus]

The main table uses a standard javafx table, which prepares every cell in memory. Virtualizing the cells could help.
See https://github.com/palexdev/VirtualizedFX

[InAnYan]

some crazy idea: what if JabRef used a database for storing all entries? Not a List<BibEntry>, but a DB?

(Probably it won't change much, because RAM on modern OSes acts like a DB of some sort, pages are stored/loaded from disk..)

[koppor]

I will collect some "interesting" code points here - which are perfect for using a SQL data base:

    public static AuthorList parse(@NonNull final String authors) {
        return AUTHOR_CACHE.computeIfAbsent(authors, string -> {
            AuthorListParser parser = new AuthorListParser();
            return parser.parse(string);
        });
    }

    public Optional<String> getFieldLatexFree(Field field) {
        if (InternalField.KEY_FIELD == field) {
            // the key field should not be converted
            return getCitationKey();
        } else if (InternalField.TYPE_HEADER == field) {
            return Optional.of(type.get().getDisplayName());
        } else if (latexFreeFields.containsKey(field)) {
            return Optional.ofNullable(latexFreeFields.get(field));
        } else {
            Optional<String> fieldValue = getField(field);
            if (fieldValue.isPresent()) {
                // TODO: To we need FieldFactory.isLaTeXField(field) here to filter?
                String latexFreeValue = LatexToUnicodeAdapter.format(fieldValue.get()).intern();
                latexFreeFields.put(field, latexFreeValue);
                return Optional.of(latexFreeValue);
            } else {
                return Optional.empty();
            }
        }
    }

    /**
     * Returns a LaTeX-free version of this `Author`.
     */
    public Author latexFree() {
        if (latexFreeAuthor == null) {
            String first = getGivenName().map(LatexToUnicodeAdapter::format).orElse(null);
            String givenNameAbbreviated = getGivenNameAbbreviated().map(LatexToUnicodeAdapter::format).orElse(null);
            String von = getNamePrefix().map(LatexToUnicodeAdapter::format).orElse(null);
            String last = getFamilyName().map(LatexToUnicodeAdapter::format).orElse(null);
            String jr = getNameSuffix().map(LatexToUnicodeAdapter::format).orElse(null);
            latexFreeAuthor = new Author(first, givenNameAbbreviated, von, last, jr);
            latexFreeAuthor.latexFreeAuthor = latexFreeAuthor;
        }
        return latexFreeAuthor;
    }

---

Idea: NOT port the whole Maintable, EntryEditor, etc. to DB, but work on these two places to make use of SQL. I bet this gives boost of 50%. 😅