scan-token ( -- str )
Factor handbook » The language » Parsing words » Reading ahead

Next:

scan-word-name ( -- string )

Vocabulary
lexer

Inputs
None

Outputs

str

a string

Word description
Reads the next token from the lexer. Tokens are delimited by whitespace, with the exception that " is treated like a single token even when not followed by whitespace. This word throws unexpected-eof on end of input. To output f on end of input, use ?scan-token instead.

Notes
This word should only be called from parsing words.

Definition

USING: kernel ;

IN: lexer

: scan-token ( -- str )
    ?scan-token [ "token" throw-unexpected-eof ] unless* ;