-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve ConfigNode load/save perf #88
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NathanKell
changed the title
Rewrite PreFormatConfig to iterate through strings less
Improve ConfigNode load/save perf
Aug 25, 2022
…trictly slower even for 1m+ rows, tho)
…MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.
NathanKell
force-pushed
the
ConfigNodePerf
branch
from
August 26, 2022 04:01
0d5cf3a
to
1062b39
Compare
gotmachine
pushed a commit
that referenced
this pull request
Dec 7, 2022
* Rewrite PreFormatConfig to iterate through strings as little as possible. * Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho) * Patch writing, too, for a slight improvement in perf * Fix a lingering issue with trailing braces * Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there. * Update readme and Settings * Big improvements to save perf by not writing lines / flushing / using a puny buffer. * missed a newline
gotmachine
pushed a commit
that referenced
this pull request
Jan 1, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible. * Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho) * Patch writing, too, for a slight improvement in perf * Fix a lingering issue with trailing braces * Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there. * Update readme and Settings * Big improvements to save perf by not writing lines / flushing / using a puny buffer. * missed a newline
gotmachine
pushed a commit
that referenced
this pull request
Jan 30, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible. * Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho) * Patch writing, too, for a slight improvement in perf * Fix a lingering issue with trailing braces * Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there. * Update readme and Settings * Big improvements to save perf by not writing lines / flushing / using a puny buffer. * missed a newline
gotmachine
pushed a commit
that referenced
this pull request
Jan 30, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible. * Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho) * Patch writing, too, for a slight improvement in perf * Fix a lingering issue with trailing braces * Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there. * Update readme and Settings * Big improvements to save perf by not writing lines / flushing / using a puny buffer. * missed a newline
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rewrite PreFormatConfig to iterate through strings less and to support skipping checking for {} and //
Also patch writing for a little bit of extra perf.
The preformat patch includes, disabled, siimav's parallel implementaiton; it's disabled because at the moment it's strictly slower than stock code AFAICT.
This yields up to an 8x boost (or so) in read/write performance for sfs, and lower but still substantial gains elsewhere.
For testing, add this to a file in gamedata and then load craft/saves.
Note that the skip nodes will only apply after MM loads; before that you'll just get the regular speedup.