User-agent: * Disallow: /members.phtml Disallow: /mystores/buylink.html Disallow: /mystores/buylink.phtml Disallow: /entry.phtml #Crawl-Delay: 10 #User-agent: BecomeBot Crawl-Delay: 20 #This one would make repeated calls to relative links #/page.html, then ../page.html, then ../../page.html, etc User-agent: Pompos Disallow: / #picsearch.com picture search User-agent: psbot Disallow: / #Korean Telecom User-agent: nhnbot Disallow: / #Korean Telecom User-agent: NaverBot Disallow: / #Ask Jeeves/Teoma User-agent: Ask Jeeves Disallow: /mystores/search.phtml #MSN User-agent: msnbot Disallow: /mystores/search.phtml #These things walk through the bookstore all day #User-agent: Wget #Disallow: / #These things walk through the bookstore all day User-agent: Httrack Disallow: / #Japan internet multifeed User-agent: gazz Disallow: /mystores/ #Nameprotect.com - DAMNED IP crawler User-agent: NPBot Disallow: / #ShopWiki private comparison shopper User-agent: ShopWiki Disallow: /