By using this site you agree to the use of cookies by Brugbart and our partners.

Learn more

Robots.txt Tutorial

Comprehensive Robots.txt Tutorial, on how to block and allow individual robots. Etc.

Edited: 2015-04-15 00:30

The robots.txt file is used to disallow robots, (such as search engine crawlers), from visiting parts, or sections of a website. Most robots will read your robots.txt file, and follow the rules that you have placed there – keep in mind however, that this file alone will not prevent access to robots that do not use robot.txt files!

Use the robots.txt file to disallow indexing of content that you don't want indexed, such as decorative images, borders, or button images. Etc.

It should not be necessary to block JavaScript and CSS files, and some search engines may even use such files to better rank your pages, so blocking them in robots.txt might not be such a good idea.

Tutorial

  1. Using the asterisk (*) wildcard
  2. Blocking certain robots
  3. Block a directory using robots txt
  4. How to link your Sitemap in Robots.txt

Articles

  1. Robots.txt and Security
  2. Blocking bad robots
  3. Using the Robots Meta tag

Robots.txt example

 User-agent: Google
 User-agent: Yahoo
 Disallow: /

 User-agent: msnbot
 Allow: /