Using the asterisk wildcard with the robots.txt file

Prevent access to all robots, except for the ones that you specifically allow.

The asterisk (*) wildcard can be used to prevent access for all robots, except the ones that you specifically list. The star should be placed just after the User-agent: part in your robots.txt file.

The below robots.txt example shows how to block or disallow access for all robots, and only allow Google, Bing, and Yahoo.

User-agent: msnbot # Bing
Allow: /
User-agent: Slurp # Yahoo
Allow: /
User-agent: Google
Allow: /

# All other robots
User-agent: *
Disallow: /

Links