Handbook
Glossary
rules
Vocabulary
robots
Definition
IN:
robots
TUPLE:
rules
user-agents allows disallows visit-time request-rate
crawl-delay unknowns
;