PHP 8.5: Recursive tokenization during `token_get_all` result on error

[gharlan]

Description

The following code:

a.php:

<?php

set_error_handler(function (int $errno, string $msg) {
    require __DIR__.'/b.php';
    echo 'error handler called: ', $msg;
});

$code = '<?php (double) $x;';
$tokens = token_get_all($code, \TOKEN_PARSE);

echo "\n\n";
foreach ($tokens as $token) {
    if (is_string($token)) {
        echo $token, "\n";
    } else {
        echo token_name($token[0]) . ' "' . $token[1], "\"\n";
    }
}

b.php:

<?php 1;

Calling php a.php resulted in this output:

error handler called: Non-canonical cast (double) is deprecated, use the (float) cast instead

T_OPEN_TAG "<?php "
T_OPEN_TAG "<?php "
T_LNUMBER "1"
;
T_WHITESPACE "
"
T_DOUBLE_CAST "(double)"
T_WHITESPACE " "
T_VARIABLE "$x"
;

But I expected this output instead:

error handler called: Non-canonical cast (double) is deprecated, use the (float) cast instead

T_OPEN_TAG "<?php "
T_DOUBLE_CAST "(double)"
T_WHITESPACE " "
T_VARIABLE "$x"
;

Not sure about the error handler call. Is it intended that the error handler is called for deprecations inside the code given to token_get_all?
But at least I do not expect to get the tokens of b.php inside the tokens of the code given to token_get_all.

PHP Version

PHP 8.5.0-dev (cli) (built: Aug 16 2025 23:33:23) (NTS)
Copyright (c) The PHP Group
Built by Shivam Mathur
Zend Engine v4.5.0-dev, Copyright (c) Zend Technologies
    with Zend OPcache v8.5.0-dev, Copyright (c), by Zend Technologies

Operating System

macOS

[devnexen]

verified on master, not happening on 8.4.x

[devnexen]

@nielsdos do you think it is more a change in the engine ? Do not think it comes from the tokenizer extension.

[mvorisek]

simpler reproducer: https://3v4l.org/AO49k/rfc#vgit.master

[DanielEScherzer]

Probably caused by the deprecations coming from within zend_language_scanner.l rather than during compilation, e.g.

php-src/Zend/zend_language_scanner.l

Lines 1636 to 1641 in c98b173

	<ST_IN_SCRIPTING>"("{TABS_AND_SPACES}("integer"){TABS_AND_SPACES}")" {
	if (PARSER_MODE()) {
	zend_error(E_DEPRECATED, "Non-canonical cast (integer) is deprecated, use the (int) cast instead");
	}
	RETURN_TOKEN(T_INT_CAST);
	}

CC @Girgias

[hakre]

IIRC tokenization was error free for a long time. if it now emits, the diagnostic messages should probably be collected and optionally be available, e.g. an out parameter on token_get_all().

That the previous(?) token is returned looks worry-some to me, too, collecting instead of immediate emission may solve that, too.

[keradus]

if it now emits, the diagnostic messages should probably be collected and optionally be available, e.g. an out parameter on token_get_all().

I do not think this case should trigger any error in first place. if code is valid but deprecated, tokenizer should work as-is (same output as under 8.4)

[TimWolla]

The only other non-fatal in the scanner is a E_COMPILE_WARNING. See also: #18510.