lexer
The lexer

Next:

<lexer> ( text -- lexer )

Vocabulary
lexer

Variable description
Stores the current lexer instance.

Class description
An object for tokenizing parser input. It has the following slots:

text	the lines being parsed; an array of strings
line	the line number being parsed; unlike most indices this is 1-based for friendlier error reporting and integration with text editors
column	the current column position, zero-based

Custom lexing can be implemented by delegating a tuple to an instance of this class and implementing the skip-word and skip-blank generic words.

Definition

USING: arrays math strings vectors ;

IN: lexer

TUPLE: lexer
    { text array initial: { } } { line fixnum initial: 0 }
    { line-text string initial: "" }
    { line-length fixnum initial: 0 }
    { column fixnum initial: 0 }
    { parsing-words vector initial: V{ } } ;

Methods

USING: kernel lexer lexer.private sequences ;

M: lexer skip-blank
    shebang?
    [ [ nip length ] change-lexer-column ]
    [ [ (skip-blank) ] change-lexer-column ] if ;


USING: lexer lexer.private ;

M: lexer skip-word [ (skip-word) ] change-lexer-column ;