Robots.txt and Security

Potential security issues around using Robots.txt to block indexing of content in members sections.

You should not rely to heavily on robots.txt, if there is something that you don't to spread around the Internet, then either don't host it on your site, or implement a decent server-side security mechanism instead.

Not all robots respect robots.txt, and some might even chose to ignore it entirely. Its therefor best that you use other security mechanisms, to prevent access to content.

Robots.txt and Security

There are quite a few security issues around using Robots.txt, none however critical. Robots.txt is mainly useful if you want to control how the major known search engines will access your site, not as a security mechanism.

In addition, listing secret directories in the robots text file, could inform hackers of otherwise unknown locations on your server. Its therefor important, that you have other security mechanisms in place. Simply providing members of your site with a secret URL, is not enough to prevent access, especially not if you list this URL in your robots.txt file to prevent it from showing up in the search results.

Post comment

Links that you insert are not nofollowed, but will be removed by admins if they are considered spam.

[url=Absolute URL for page]TITLE[/url]

You should insert code boxes around code examples, which will be automatically syntax highlighted.

[code1 html|css|javascript|php|sql]Your Code Here[/code1]

You may want to read our Privacy Policy before submitting your comment.