# disallows all Web Spiders for the entire site
User-agent: *
Disallow: /