By using this site you agree to the use of cookies by Brugbart and our partners.

Learn more

The Robots Meta Tag

How to use the robots meta tag, and a list of allowed values.

Edited: 2015-12-04 15:38

The Robots meta tag is used to control whether robots are allowed to cache, and index a page, as well as whether its allowed to crawl the links found on a page. The tag was originally intended for those who couldn't control the Robots.txt file on their websites, but its use has since expanded.

The robots meta tag should be placed in the head section of pages.

You can use the robots meta tag, in place of the rel="nofollow" attribute.

You should keep in mind, that the robots meta tag only works on a per page basis. If you tell robots not to crawl links found on your page, then keep in mind that they might still be able to crawl the linked pages from other locations on the web.

Accepted values

The robots meta tag takes the value of a comma separated list.

Value:Description:
allRobots are both allowed to index, and crawl the links on a page.
noneNothing is allowed.
indexRobots are allowed to index the page.
noindexTells robots that the page should not be indexed.
followRobots are allowed to follow links on a page.
nofollowTells robots that links on the page should not be followed, or pass on pagerank. Easier then having to use rel="nofollow" on the individual links.
noarchiveRobots are not allowed to keep a cached version of the page.

None and noarchive

It seems unclear if a value of none will disallow caching as well as indexing and following. Feel free to discuss this in the comments.

The name attribute

Some robots allow you to target them specifically, by replacing the all targeting robots value, with the name of the robot that you want to target. See the examples section for further information.

Robots meta tag examples

Remember that the tag should be placed inside the head element on pages, like shown below:

<head>
 <meta name="robots" content="nofollow">
</head>

To use multiple values in the same tag, separate the values with comma.

<meta name="robots" content="noindex,nofollow">

Execptions

You can also target all bots, and then add specific rules for other bots to follow. I.e.

<meta name="robots" content="noindex,nofollow">
<meta name="googlebot" content="index,follow">

Google

<meta name="googlebot" content="noindex,nofollow">

Yahoo

<meta name="slurp" content="noindex,nofollow">

Bing

<meta name="msnbot" content="noindex,nofollow">