Vocabulary
lexer

Variable description
Stores the current lexer instance.

Class description
An object for tokenizing parser input. It has the following slots:

•	text - the lines being parsed; an array of strings
•	line - the line number being parsed; unlike most indices this is 1-based for friendlier error reporting and integration with text editors
•	column - the current column position, zero-based

Custom lexing can be implemented by delegating a tuple to an instance of this class and implementing the skip-word and skip-blank generic words.

Definition

USING: arrays math strings vectors ;

IN: lexer

TUPLE: lexer
{ text array initial: { } } { line fixnum initial: 0 }
{ line-text maybe{ string } initial: f }
{ line-length fixnum initial: 0 }
{ column fixnum initial: 0 }
{ parsing-words vector initial: V{ } } ;

Methods

USING: lexer ;

M: lexer skip-blank [ t skip ] change-lexer-column ;

USING: combinators kernel lexer lexer.private math ;

M: lexer skip-word
[
{
{ [ 2dup quote? ] [ drop 1 + ] }
{ [ 2dup shebang? ] [ drop 2 + ] }
[ f skip ]
} cond
] change-lexer-column ;