Incorrect reporting of endTokenPos attribute in recent versions of nikic/php-parser library.

[lisachenko]

There are several bug reports in goaop framework itself about breaks in code:

goaop/framework#451,
goaop/framework#452.

Need to release a hotfix patch version to prevent installation of broken versions.
@nikic, ping

[nikic]

PHP-Parser 4.7.0 canonicalizes token_get_all() output to the PHP 8 format, using T_NAME_* tokens. Does that sound like something that could affect you? E.g. are you using token positions from PHP-Parser, but doing your own token_get_all call?

[lisachenko]

@nikic From what I can see, php-parser reports incorrect position of tokens for nodes. And I'm heavily use this information to adjust original source code (eg to find position where code should be changed). I'm using plain token_get_all() and information from php-parser to find correct position for node in token stream.

Am I right that indexes reported by php-parser can be used to specify an index for array returned by the token_get_all() for the same file?

[nikic]

@lisachenko The token positions returned by php-parser refer to positions into the tokens returned by $lexer->getTokens(), which are based on token_get_all(), but not exactly the same, due to some normalization of PHP version differences. Probably the 4.7 release was the first one where the difference became visible for your usage.

[lisachenko]

Ok, @nikic! Thanks for the quick response, I should update my code to follow new logic.

[xfuturomax]

@lisachenko are there any news or plans about using new version of php-parser? I want to upgrade my phpunit to version 9, and it requires "nikic/php-parser": "^4.8"

[lisachenko]

@xfuturomax unfortunately, I’m busy with my new job, thus no time for open-source now. If you want - you can contribute this, it shouldn’t be very hard to adopt new version, need only to use proper methods instead of get_tokens_all() function call.

[xfuturomax]

goaop/framework#454
#106

I've tested with nikic/php-parser v4.10, and all code was generated correct.