tokenize ( input -- ast )


Vocabulary
simple-tokenizer

Inputs
inputa string


Outputs
asta sequence of strings


Word description
Tokenize a string. Supported syntax:
foo bar baz - simple tokens
foo\ bar - token with an escaped space
"foo bar" - quoted token


Definition