A variable that encapsulate internal parser state:

lexer

Creating a default lexer:

<lexer> ( text -- lexer )

A word to test of the end of input has been reached:

still-parsing? ( lexer -- ? )

A word to advance the lexer to the next line:

next-line ( lexer -- )

Two generic words to override the lexer's token boundary detection:

skip-blank ( lexer -- )

skip-word ( lexer -- )

Utility combinator:

with-lexer ( lexer quot -- newquot )