PHP-Parser broken on PHP 8.0.0-dev

[ondrejmirtes]

Hi, I prepared a minimal reproducing repo:

https://github.com/ondrejmirtes/php-8-parser-repro

The script to show the error is to run:

                composer install
                php --version
                vendor/bin/php-parse ConstantStringType.php

The current output (can be seen in GitHub Actions https://github.com/ondrejmirtes/php-8-parser-repro/runs/873878820?check_suite_focus=true) is:

PHP 8.0.0-dev (cli) (built: Jul 15 2020 00:48:57) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.0-dev, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.0-dev, Copyright (c), by Zend Technologies
====> File ConstantStringType.php:
PHP Notice:  Undefined array key 338 in /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/lib/PhpParser/Lexer.php on line 284
PHP Fatal error:  Uncaught TypeError: PhpParser\Lexer::getNextToken(): Return value must be of type int, null returned in /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/lib/PhpParser/Lexer.php:324
Stack trace:
#0 /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/lib/PhpParser/ParserAbstract.php(208): PhpParser\Lexer->getNextToken()
#1 /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/lib/PhpParser/ParserAbstract.php(158): PhpParser\ParserAbstract->doParse()
#2 /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/lib/PhpParser/Parser/Multiple.php(51): PhpParser\ParserAbstract->parse()
#3 /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/lib/PhpParser/Parser/Multiple.php(32): PhpParser\Parser\Multiple->tryParse()
#4 /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/bin/php-parse(71): PhpParser\Parser\Multiple->parse()
#5 {main}
  thrown in /home/runner/work/php-8-parser-repro/php-8-parser-repro/vendor/nikic/php-parser/lib/PhpParser/Lexer.php on line 324

Of course I'm not sure what exactly in ConstantStringType is causing this, and whether it's a bug of PHP 8, or the PHP-Parser.

Thank you.

[ondrejmirtes]

BTW if it's any help, the error didn't occur in build that happened on July 9th, but started showing up in builds since July 12th. But I went through commits in php-src between those dates and couldn't find anything that could have caused it (nothing related to tokens or so).

[nikic]

This is because PHP-Parser does not support match yet.

[ondrejmirtes]

Is it because I'm calling a static method \Nette\Utils\Strings::match? Shouldn't that be a T_STRING to the PHP token_get_all() function?

[ondrejmirtes]

The difference can be seen here: https://3v4l.org/eOan2/rfc#output

All PHP versions up until 8.0-alpha 2:

T_OPEN_TAG
T_FUNCTION
T_WHITESPACE
T_STRING
(
)
:
T_WHITESPACE
T_STRING
T_WHITESPACE
{
}
T_WHITESPACE
T_STRING
(
)
;

php-master:

T_OPEN_TAG
T_FUNCTION
T_WHITESPACE
T_MATCH
(
)
:
T_WHITESPACE
T_STRING
T_WHITESPACE
{
}
T_WHITESPACE
T_MATCH
(
)
;

If it's at all possible, I'd be for preserving the original behaviour. It'd also make writing sniffs for PHP_CodeSniffer easier.

[ondrejmirtes]

Alright, I just read the RFC (https://wiki.php.net/rfc/match_expression_v2) and noticed it's supposed to continue to work as method name: https://3v4l.org/HPLCd#output

The tokens comparison is now:

Original:

T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
{
T_WHITESPACE
T_STATIC
T_WHITESPACE
T_FUNCTION
T_WHITESPACE
T_STRING
(
)
:
T_WHITESPACE
T_STRING
T_WHITESPACE
{
}
T_WHITESPACE
}
T_WHITESPACE
T_NS_SEPARATOR
T_STRING
T_NS_SEPARATOR
T_STRING
(
)
;
T_WHITESPACE

New:

T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
{
T_WHITESPACE
T_STATIC
T_WHITESPACE
T_FUNCTION
T_WHITESPACE
T_MATCH
(
)
:
T_WHITESPACE
T_STRING
T_WHITESPACE
{
}
T_WHITESPACE
}
T_WHITESPACE
T_NS_SEPARATOR
T_STRING
T_NS_SEPARATOR
T_MATCH
(
)
;
T_WHITESPACE

I'd be for preserving T_STRING there.

/cc @iluuu1994 Would it be possible?

[nikic]

@ondrejmirtes The method name will continue to work, it's just not supported as a semi-reserved identifier in PHP-Parser yet.

[ondrejmirtes]

Yeah, I'm just arguing for token_get_all() to interpret it as T_STRING in a class method name as it'd make life of people working with the token list easier.

[nikic]

@ondrejmirtes It will be interpreted as T_STRING if you enable TOKEN_PARSE mode in token_get_all(). That's exactly what the flag is for :)

[ondrejmirtes]

Alright, thanks! :) I learnt something today :)

[nikic]

I've merged #672, so this should be fixed now (if you use dev-master).

[ondrejmirtes]

Thank you very much!