tokenize ( input -- ast )

Vocabulary

simple-tokenizer

Inputs

input

a string

Outputs

ast	a sequence of strings

Word description

Tokenize a string. Supported syntax:

•	foo bar baz - simple tokens
•	foo\ bar - token with an escaped space
•	"foo bar" - quoted token

Definition

USING: accessors namespaces peg peg.private ;

IN: simple-tokenizer

: tokenize ( input -- ast )
    [ \ ( gensym ) perform-parse ] with-scope
    check-parse-result ast>> ;