Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve ConfigNode load/save perf #88

Merged
merged 8 commits into from
Aug 26, 2022
Merged

Improve ConfigNode load/save perf #88

merged 8 commits into from
Aug 26, 2022

Conversation

NathanKell
Copy link
Contributor

@NathanKell NathanKell commented Aug 25, 2022

Rewrite PreFormatConfig to iterate through strings less and to support skipping checking for {} and //
Also patch writing for a little bit of extra perf.

The preformat patch includes, disabled, siimav's parallel implementaiton; it's disabled because at the moment it's strictly slower than stock code AFAICT.

This yields up to an 8x boost (or so) in read/write performance for sfs, and lower but still substantial gains elsewhere.

For testing, add this to a file in gamedata and then load craft/saves.

@KSP_COMMUNITY_FIXES:FINAL
{
	@CONFIGNODE_PERF_SKIP_PROCESSING_KEYS
	{
		item = attPos0
	}
	@CONFIGNODE_PERF_SKIP_PROCESSING_SUBSTRINGS
	{
		item = attRot
	}
}

Note that the skip nodes will only apply after MM loads; before that you'll just get the regular speedup.

@NathanKell NathanKell changed the title Rewrite PreFormatConfig to iterate through strings less Improve ConfigNode load/save perf Aug 25, 2022
@NathanKell NathanKell merged commit fb90bee into dev Aug 26, 2022
@NathanKell NathanKell deleted the ConfigNodePerf branch August 26, 2022 04:02
gotmachine pushed a commit that referenced this pull request Dec 7, 2022
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
gotmachine pushed a commit that referenced this pull request Jan 1, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
gotmachine pushed a commit that referenced this pull request Jan 30, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
gotmachine pushed a commit that referenced this pull request Jan 30, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant