shlex-split ( string comments? posix? -- tokens )
Shell-like lexical analysis

Prev:	parse-shlex ( string -- tokens )
Next:	shlex-quote ( string -- quoted )

Vocabulary
shlex

Inputs

string	a string
comments?	a boolean
posix?	a boolean

Outputs

tokens

an array

Word description
Splits a string like Python's shlex.split(string, comments, posix). With comments enabled, unquoted # starts a comment through the next newline. POSIX mode recognizes backslash escapes outside single quotes; inside double quotes only a double quote or backslash loses its escape. Non-POSIX mode retains quotes, recognizes them only at the start of a token, ends a token at a closing quote, and treats backslashes literally.

Errors
Throws shlex-unclosed-quote for an unmatched quote and shlex-missing-escape for an unfinished POSIX escape.

Definition

USING: arrays ascii combinators kernel sequences strings ;

IN: shlex

:: shlex-split ( string comments? posix? -- tokens )
    V{ } clone :> tokens
    V{ } clone :> token! f :> started?! f :> quote! f
    :> escaped?! f :> comment?! [
        started? [
            token >string tokens push
            V{ } clone token! f started?!
        ] when
    ] :> finish-token string [| ch |
        {
            { [ comment? ] [ ch 10 = [ f comment?! ] when ] }
            {
                [ escaped? ]
                [
                    quote
                    [ ch quote = ch 92 = or ~quotation~ unless ]
                    when ch token push f escaped?!
                ]
            }
            {
                [ quote ]
                [
                    ch quote =
                    [ posix? ~quotation~ unless f quote! ] [
                        posix? quote 34 = and ch 92 = and
                        ~quotation~ ~quotation~ if
                    ] if
                ]
            }
            {
                [ comments? ch 35 = and ]
                [
                    t comment?! posix?
                    [ finish-token call ] when
                ]
            }
            { [ ch blank? ] [ finish-token call ] }
            {
                [ posix? ch 92 = and ]
                [ t started?! t escaped?! ]
            }
            {
                [ ch "'\"" member? posix? started? not or and ]
                [
                    ch quote! t started?! posix?
                    [ ch token push ] unless
                ]
            }
            [ ch token push t started?! ]
        } cond
    ] each escaped? [ shlex-missing-escape ] when quote
    [ quote shlex-unclosed-quote ] when finish-token call
    tokens >array ;