Improve PHP parsing performance

[nylen]

Fixes #1646 where the PHP parser introduced in #1152 caused a performance regression.

Previous parsing times displayed evidence of O(n^2) complexity - from #1646 (comment):

 358 words: 0.06s
 921 words: 0.33s
1438 words: 0.77s
1793 words: 1.13s

Current parsing times (using a post not containing any WP blocks, generated the same way as the above example) are significantly improved, with O(n) complexity:

 500 words: 0.04s
1000 words: 0.08s
1500 words: 0.11s
2000 words: 0.13s

Read on for details about what changed.

---

Using this excellent StackOverflow answer, I set up xhprof to do granular profiling of PHP code. Here were the initial results when running RUN_SLOW_TESTS=y phpunit -c phpunit-xhprof.xml --filter large_post:

We're calling mb_strlen and mb_substr too many times (the post being tested is the Gutenberg demo content, which has around 10k characters), and these functions are too slow.

mb_strlen is really easy to fix, because it's always called against the full length of the input string. Fixed in nylen/phpegjs@9f6b0f8.

mb_substr is a bit more difficult to get rid of, but it's important to realize what these multibyte string functions actually do. They can't know the length of the string in advance, because UTF-8 characters are variable-length. So, every substring operation needs to loop through the entire string until it reaches its starting position. This is the source of the previous O(n^2) complexity.

This is fixed in nylen/phpegjs@3bfd473 by splitting the parse input into an array of UTF-8 characters when a parsing run begins. Direct array lookups are far faster than looping over a multibyte string, and WordPress core already uses this approach in other places. (Like core does in the code block below that one, we need to detect when PCRE Unicode support is not available, and use an alternative method. This is a task for a separate PR.)

Final performance results indicate further room for improvement, and there are some more obvious things to try, such as eliminating the input_substr function for substrings that are known to be of length 1. However, the major issues with the initial version are fixed and we can continue iterating later:

[westonruter]

@mtias I think the parsing performance problem should be fixed prior to the 0.4 release.

[nylen]

Yep, that's the plan :)

@westonruter I am pretty satisfied with test coverage on these changes; do you see any issues here?

[westonruter]

Excellent performance debugging here!

[westonruter]

Instead of adding a long-content.html fixture, should it instead just create the content programmatically?

[nylen]

It's not clear to me how to do this in a simple way. It would require executing Node.js, much like generating the PHP parser, which is also checked into the repo. I'd prefer to leave this as-is for now.

[westonruter]

This hard-coded number seems problematic in the long run because machines will get faster and faster that we develop on. What if, and this doesn't seem feasible either, we obtain the CPU speed and use that a multiplier for some threshold value? The other thought that came to mind is to use register_tick_function() to count the ticks that happen when parsing the large content, but I tried it out and on master and on this branch the result is both 11, probably because I was only declaring ticks in the unit test file and not the parser.

[westonruter]

The error_log here may have an undesirable effect on the phpunit output, with the results being added inline with the progress when running all tests:

[nylen]

Regarding the last couple of review comments here: I would like to build a process that runs these "slow tests" every hour or so against master, then reports the results somewhere. I plan for this to handle many of the cases described at #939 (comment).

For performance tests, PHPunit may not be the best tool. Ideally the performance test mechanism should store past results, provide a history, and compare them against previous results from the same machine.

That's all for later - the important thing for now is to fix the performance degradation included in the 0.3.0 release.

[dmsnell]

Very good work here @nylen, and I'm sorry for jumping to a false conclusion earlier about the source of the performance issues.

Curious:

• do we have any comparison to how fast the JS-based parser performs on the same documents on the same machine?
• do you see any other major performance issues here?

In light of this I'm going to close our #1681 and try to address other parser-related tasks. Thanks for staying persistent on this.

[nylen]

do we have any comparison to how fast the JS-based parser performs on the same documents on the same machine?

I just ran some more tests on parsing the fixture introduced in this PR.

JavaScript :  11ms
PHP 5.6    : 150ms
PHP 7.0    :  17ms
PHP 7.1    :  19ms

Fortunately PHP 7.x is our official recommendation.

do you see any other major performance issues here?

Nothing major really.

I am not a huge fan of the current approach of splitting the entire input string into an array at the beginning of the parse, but I don't see a good way around it, other than proving that we can safely use substr (sounds tricky and unlikely).

Fixing #1611 should provide further improvements (the peg_regex_test block at the bottom of the second performance graph).

We can get rid of a bunch more input_substr function calls (the red block at the bottom of the second performance graph). These calls aren't very complicated so I'm not sure how much it will actually help.

[dmsnell]

Ouch on 5.6 but those are good numbers otherwise. Keeping the PHP parse within a magnitude of the JS parse I think is a satisfactory target. Getting them within half the runtime of JS is good!