parse-robots.txt-line ( rules seq -- rules )

Vocabulary

robots.private

Inputs

rules	an object
seq	an object

Outputs

rules

an object

Definition

USING: accessors arrays assocs calendar.parser combinators
kernel math.parser sequences splitting ;

IN: robots.private

: parse-robots.txt-line ( rules seq -- rules )
    first2 swap {
        { "user-agent" [ add-user-agent ] }
        { "allow" [ add-allow ] }
        { "disallow" [ add-disallow ] }
        { "crawl-delay" [ string>number >>crawl-delay ] }
        { "request-rate" [ string>number >>request-rate ] }
        {
            "visit-time"
            [
                "-" split1 [ hhmm>duration ] bi@ 2array
                >>visit-time
            ]
        }
        [ pick unknowns>> push-at ]
    } case ;