Robots.txt

by Darrell Kingsley — last modified Mar 13, 2014 02:14 PM
This file restricts access to your site for well-behaved spiders and bots

The biggest gotcha is the user-agent disallow statements are not additive, so if you have some rules for all user agents, and then some different rules for Googlebot, Googlebot will only pay attention to the Googlebot rules. To get all the rules to apply, you have to specify the "all bot" rules again within the Googlebot rules:

User-agent: *
Disallow: /my_folder

User-Agent: Googlebot
Disallow: /my_other_folder

Googlebot only ignores my_other_folder. To get it to ignore both you need:

User-Agent: Googlebot
Disallow: /my_other_folder
Disallow: /my_folder