default-tokenizer ( -- tokenizer )

Vocabulary

peg.ebnf.private

Inputs

None

Outputs

tokenizer

an object

Definition

USING: kernel peg peg.parsers ;

IN: peg.ebnf.private

: default-tokenizer ( -- tokenizer )
    T{ tokenizer-tuple
        { any [ any-char ] }
        { one [ token ] }
        { many [ [ = ] curry any-char swap semantic ] }
    } ;