By using this site you agree to the use of cookies by Brugbart and our partners.

Learn more

Block a directory using robots txt

How to block a directory and its subdirectories using the robots.txt file.

Edited: 2011-07-24 18:48

Sometimes you have a directory containing decorative images, or temp files. And sometimes you may not want this to be indexed in the search engines. You might also have a directory with some private files, which you don't want to have indexed.

Keep in mind that the Robots.txt file isn't intended for access control. If you want to keep some data safe, then you shouldn't have made it publicly accessible in the first place. See also: Robots.txt and Security

The below will block access to a directory called "images" as well as its subdirectories.

User-agent: *
Disallow: /images/

Multiple directories? No problem!

User-agent: *
Disallow: /images/
Disallow: /temp/
Disallow: /cgi-bin/