-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Json with more error checking and parsing #360
Conversation
…y retained and default settings enforced.
As I test, I decided to fuzz the index you provided @joyfullservice, and write a tool to fuzz one in the future. This allows us to build huge If you're ok with it, I'd like to put the new testing file in the \testing\ directory. |
Here is what I am getting after building from your branch:
|
…x without exposing the file names (potentially releasing confidential or private data).
… of this function in my environment; 1-2 seconds worth in ultra large json files.
… string charachter; stop execution if that's not true. Add chunk size increases for performance tuning.
Interesting; my mind is BLOWN, because it's a solid 3-5 seconds per loop (15-17) for me on the same code. |
I did a little more testing, and noted that you are doing 5 iterations for each of these tests. The 0.5 seconds I was getting on the performance side was for a single iteration of loading JSON content. Here is a comparison after adding in a couple other tests for side-by-side comparison.
From this, it appears that there is no appreciable performance difference between the different approaches taken. As to the actual performance on my machine, I am running an older i5 (Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz) and the file is located on my Samsung EVO SSD with Rapid Mode on and no disk encryption. (Although we are probably looking at in-memory processing here.) Hopefully that gives you a little more comparative data for evaluating the performance enhancements. 😄 |
Well, after looking into it, I think it may actually be Access's fault on my end. Intel(R) Xeon(R) E-2276M CPU @ 2.80GHz 2.81 GHz, 32GB Ram, with 1TB SSD. Access 365, 64bit, Version 2202. And then I found this, which indicates I might have a version susceptible to this... https://answers.microsoft.com/en-us/msoffice/forum/all/vba-code-is-very-slow-in-office-365-version-of/e9b7bc57-baa3-4c73-b229-47f0a460ccd8 |
After several days of head banging, I will probably end up reverting all the string buffer tooling. However; I've found that if I turn it into a class, it becomes a lot more useful, and can handle automatic property loading (useful in my other project). I'll be cleaning this up shortly. |
…nd ensure consistent testing from machine to machine.
@joyfullservice: can you pull this in and see if the new test routine still works on your end? I made some tweaks that help (me) a little bit, but before I roll those into the formal PR, I want to ensure I didn't break anything else. |
@joyfullservice bump |
Since the last PR didn't go as expected with a performance hit, I'm trying to track down the performance limiting items.
To do this, I opted to move the thing to a class (for now), which allows for easier default options settings.
As of now, I'm getting a minimum parse time of 3 seconds for the giant vcs-index file with even the original parser, not 0.51s reported by @joyfullservice. I'm not sure what is going on, because it should be considerably faster, so I'm putting it up here for @joyfullservice to hopefully test on his end and see if his results match mine.