parse-token ( lexer -- str/f )

Vocabulary

lexer

Inputs

lexer

a lexer

Outputs

str/f

a string or f

Word description

Reads the next token from the lexer. Tokens are delimited by whitespace, with the exception that " is treated like a single token even when not followed by whitespace.

Definition

USING: kernel ;

IN: lexer

: parse-token ( lexer -- str/f )
    dup parse-raw [ skip-comments ] [ drop f ] if* ;