Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optionally specify limit for number of entities in a record. #373

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

blackwinter
Copy link
Member

@blackwinter blackwinter commented Oct 10, 2024

Just a proof-of-concept to gauge interest in such a feature; the immediate need has been rendered moot (due to the record's holdings being suppressed from publishing now).

This is a brute-force approach to dealing with OOM situations when Alma records have an excessive number of items (e.g. 99374518570506441: >12000 entities = ~10 GB heap for the Record instance). Loosely modeled after JAXP's totalEntitySizeLimit, except that we don't abort processing but instead skip the remaining entities.

Use Metafix instance setter setMaxEntityCount(int) or set system property org.metafacture.metafix.maxEntityCount=<int>.

Alternative options:

  • Increase maximum heap size for JVM.
  • Significantly reduce memory requirement for Record instances.

Possible optimizations (implemented in dc4f1af):

  1. Drop the setter and make the limit static final; this way the JVM can (potentially) eliminate the check altogether when no limit has been set (less than zero).
  2. Skip all remaining literals (instead of just the ones inside entities), thereby simplifying the check in literal().

@dr0i dr0i self-requested a review October 14, 2024 09:54
This is a brute-force approach to dealing with OOM situations when Alma records have an excessive number of items (e.g. 99374518570506441: >12000 entities = ~10 GB heap for the Record instance).

Use Metafix instance setter `setMaxEntityCount(int)` or set system property `org.metafacture.metafix.maxEntityCount=<int>`.

Alternative options:

- Increase maximum heap size for JVM.
- Significantly reduce memory requirement for Record instances.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Review
Development

Successfully merging this pull request may close these issues.

1 participant