Vocabulary
lexer

Inputs and outputs

end	a string
seq	a new sequence of strings

Word description
Reads a sequence of tokens until the first occurrence of end. The tokens remain as strings and are not processed in any way. This word is equivalent to map-tokens with an empty quotation.

Notes
This word should only be called from parsing words.

See also
each-token, map-tokens, (parse-until), parse-until

Definition

IN: lexer

: parse-tokens ( end -- seq ) [ ] map-tokens ;