get-robots.txt ( url -- robots.txt )

Vocabulary

robots.private

Inputs

url

an object

Outputs

robots.txt

an object

Definition

USING: http.client kernel ;

IN: robots.private

: get-robots.txt ( url -- robots.txt )
    >robots.txt-url http-get nip ;