9 lines
103 B
Plaintext
9 lines
103 B
Plaintext
|
User-agent: *
|
||
|
|
Disallow:
|
||
|
|
|
||
|
|
User-agent: WebReaper
|
||
|
|
Disallow: /
|
||
|
|
|
||
|
|
User-agent: Slurp
|
||
|
|
Crawl-delay: 180
|