Robots.txt Tutorial

Comprehensive Robots.txt Tutorial, on how to block and allow individual robots. Etc.

The robots.txt file is used to disallow robots, (such as search engine crawlers), from visiting parts, or sections of a website.

Use the robots.txt file to disallow indexing of content that you don't want indexed, such as decorative images, borders, or button images. Etc.

Tutorial

  1. Using the asterisk (*) wildcard
  2. Blocking certain robots
  3. Block a directory using robots txt

Articles

  1. Robots.txt and Security
  2. Blocking bad robots
  3. Using the Robots Meta tag

Robots.txt example

 User-agent: Google
 User-agent: Yahoo
 Disallow: /

 User-agent: msnbot
 Allow: /

Links

  1. BlueBoden's Blog

  2. Broadcasting Service