normalize-robots.txt ( string -- sitemaps seq )

Vocabulary

robots.private

Inputs

string

an object

Outputs

sitemaps	an object
seq	an object

Definition

USING: assocs combinators.short-circuit kernel sequences
splitting splitting.monotonic unicode ;

IN: robots.private

: normalize-robots.txt ( string -- sitemaps seq )
    split-lines [ [ blank? ] trim ] map [ "#" head? ] reject
    harvest
    [ ":" split1 [ [ blank? ] trim ] bi@ [ >lower ] dip ]
    map>alist [ first "sitemap" = ] partition [ values ] dip [
        {
            [ [ first "user-agent" = ] both? ]
            [ nip first "user-agent" = not ]
        } 2||
    ] monotonic-split ;