Vocabulary
lexer

Inputs and outputs

str

a string

Word description
Reads the next token from the lexer. Tokens are delimited by whitespace, with the exception that " is treated like a single token even when not followed by whitespace. This word throws unexpected-eof on end of input. To output f on end of input, use (scan-token) instead.

Notes
This word should only be called from parsing words.

Definition

USING: kernel ;

IN: lexer

: scan-token ( -- str )
(scan-token) [ "token" throw-unexpected-eof ] unless* ;