The Robots meta tag is used to control whether robots are allowed to cache, and index a page, as well as whether its allowed to crawl the links found on a page. The tag was originally intended for those who couldn't control the Robots.txt file on their websites, but its use has since expanded.
The robots meta tag should be placed in the head section of pages.
You can use the robots meta tag, in place of the rel="nofollow" attribute.
You should keep in mind, that the robots meta tag only works on a per page basis. If you tell robots not to crawl links found on your page, then keep in mind that they might still be able to crawl the linked pages from other locations on the web.
The robots meta tag takes the value of a comma separated list.
|all||Robots are both allowed to index, and crawl the links on a page.|
|none||Nothing is allowed.|
|index||Robots are allowed to index the page.|
|noindex||Tells robots that the page should not be indexed.|
|follow||Robots are allowed to follow links on a page.|
|nofollow||Tells robots that links on the page should not be followed, or pass on pagerank. Easier then having to use rel="nofollow" on the individual links.|
|noarchive||Robots are not allowed to keep a cached version of the page.|
None and noarchive
It seems unclear if a value of none will disallow caching as well as indexing and following. Feel free to discuss this in the comments.
The name attribute
Some robots allow you to target them specifically, by replacing the all targeting robots value, with the name of the robot that you want to target. See the examples section for further information.
Robots meta tag examples
Remember that the tag should be placed inside the head element on pages, like shown below:
<head> <meta name="robots" content="nofollow"> </head>
To use multiple values in the same tag, separate the values with comma.
<meta name="robots" content="noindex,nofollow">
You can also target all bots, and then add specific rules for other bots to follow. I.e.
<meta name="robots" content="noindex,nofollow"> <meta name="googlebot" content="index,follow">
<meta name="googlebot" content="noindex,nofollow">
<meta name="slurp" content="noindex,nofollow">
<meta name="msnbot" content="noindex,nofollow">