thw-theorie-web/robots.txt

9 lines
95 B
Plaintext
Raw Normal View History

User-agent: *
Disallow:
User-agent: WebReaper
Disallow: /
User-agent: Slurp
Crawl-delay: 180