parse-robots.txt ( string -- sitemaps rules-seq )

Vocabulary

robots

Inputs

string

an object

Outputs

sitemaps	an object
rules-seq	an object

Definition

USING: kernel robots.private sequences ;

IN: robots

: parse-robots.txt ( string -- sitemaps rules-seq )
    normalize-robots.txt [
        [ <rules> dup ] dip [ parse-robots.txt-line drop ] with
        each
    ] map ;