Disallow: /*?o= Disallow: /*&o= Disallow: *?*creg= Disallow: /cdn-cgi/ Request-rate: 1/15 User-agent: Googlebot Crawl-delay: 15 User-agent: MSNBot Crawl-delay: 15 User-agent: bingbot Crawl-delay: 15 User-agent: Slurp Crawl-delay: 15 User-agent: MJ12bot Crawl-delay: 15 User-agent: Yandex Crawl-delay: 20 User-agent: Amazonbot Crawl-Delay: 55 User-agent: * Crawl-Delay: 20