#Default for missing robots.txt

User-agent: *
Disallow: 
Crawl-delay: 20