Vocabulary
robots

Definition

TUPLE: rules
user-agents allows disallows visit-time request-rate
crawl-delay unknowns ;