By using this site you agree to the use of cookies by Brugbart and our partners.

Learn more

Using the asterisk wildcard with the robots.txt file

Prevent access to all robots, except for the ones that you specifically allow.

Edited: 2014-12-28 00:01

The asterisk (*) wildcard can be used to prevent access for all robots, except the ones that you specifically list. The star should be placed just after the User-agent: part in your robots.txt file.

The below robots.txt example shows how to block or disallow access for all robots, and only allow Google, Bing, and Yahoo.

User-agent: msnbot # Bing
Allow: /
User-agent: Slurp # Yahoo
Allow: /
User-agent: Google
Allow: /

# All other robots
User-agent: *
Disallow: /