url>robots ( url -- robots )

Vocabulary

robots

Inputs

url

an object

Outputs

robots

an object

Definition

USING: kernel robots.private urls ;

IN: robots

: url>robots ( url -- robots )
    >url dup get-robots.txt parse-robots.txt <robots> ;