I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (
/) URL, but nothing else.
I'd prefer to do this in robots.txt. How can I achieve this?
It is already answered in Stackoverflow, you just need to define the useragent name there. Here we only specify Googlebot, so that rule is not applied to other search engines.
user-agent: Googlebot Allow: /$ Disallow: /
As Google said (at the end of page). Use $ when you want to Match URLs that end with a specific string. Here /$ will say allowed to index all the URL that end with / for example
exapmle.com/file/ but the next line say don't crawl all the subdolfer, so
exapmle.com/file/ and similar others files which end with / or any filename is not allowed to crawl.
Note: There is no need to specify other search engine bot if you want to allowed to crawl, for example.
User-Agent: bingbot Allow: /
Weather you add above line or not, it doesn't make any difference. People use robots.txt to block something, but by default it is already allowed to crawl when you don't specify. There are so many website on internet, they don't have robots.txt at all, so all crawler by default crawl their site.