o An (-'ve valued, typically -1) EOF must never be allowed in
token_string, as it be converted to 255 -- a legitimate value.
o Comparing against a specific eof() (-1, typically) is more costly than
detecting +'ve/-'ve. Since EOF is the only non-positive value allowed
we can use the simpler test.
o Removed unnecessary test for token_string size, as it is already
tested in the method, and must never occur in correct code; used an
assert instead.
546e148b24