If you have memory-usage critical solution, you should keep in mind, that strtok function holds input string parameter (or reference to it?) in memory after usage.

<?php
function tokenize($str, $token_symbols) {
    $word = strtok($str, $token_symbols);
    while (false !== $word) {
        // do something here...
        $word = strtok($token_symbols);
    }
}
?>
Test-cases with handling ~10MB plain-text file:
Case #1 - unset $str variable
<?php
$token_symbols = " \t\n";
$str = file_get_contents('10MB.txt'); // mem usage 9.75383758545 MB (memory_get_usage() / 1024 / 1024)); 
tokenize($str, $token_symbols); // mem usage 9.75400161743 MB
unset($str); // 9.75395584106 MB
?>
Case #1 result: memory is still used

Case #2 - call strtok again
<?php
$token_symbols = " \t\n";
$str = file_get_contents('10MB.txt'); // 9.75401306152 MB
tokenize($str, $token_symbols); // 9.75417709351
strtok('', ''); // 9.75421524048
?>
Case #2 result: memory is still used

Case #3 - call strtok again AND unset $str variable
<?php
$token_symbols = " \t\n";
$str = file_get_contents('10MB.txt'); // 9.75410079956 MB
tokenize($str, $token_symbols); // 9.75426483154 MB
unset($str);
strtok('', ''); // 0.0543975830078 MB
?>
Case #3 result: memory is free

So, better solution for tokenize function:
<?php
function tokenize($str, $token_symbols, $token_reset = true) {
    $word = strtok($str, $token_symbols);
    while (false !== $word) {
        // do something here...
        $word = strtok($token_symbols);
    }

    if($token_reset)
        strtok('', '');
}
?>