I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else.
I'd prefer to do this in robots.txt. How can I achieve this?
-
I'm assuming you know how robots.txt works, since you prefer it. What rules have you tried?– GDVSNov 29, 2019 at 13:14
-
1Do you want Google to only be able to crawl the home page, or only be able to index the home page? These are two different goals with different best ways to achieve them. Sometimes Google is known to index pages without crawling them.– Maximillian Laumeister ♦Nov 29, 2019 at 18:23
-
1It is very unusual for a site to only have their home page indexed in search engines. You tagged this as "SEO" and what you are requesting is the opposite of SEO. It is really bad for SEO to have just one page in search engines.– Stephen Ostermiller ♦Nov 30, 2019 at 12:10
-
@StephenOstermiller there are other search enginges than google, hence the SEO-tag.– saturnusringarNov 30, 2019 at 16:32
-
@MaximillianLaumeister ideally, both: only crawl the home page AND only index the home page. All other bots (more or less) will be allowed to crawl and index the site.– saturnusringarNov 30, 2019 at 16:33
|
Show 2 more comments
1 Answer
It is already answered in Stackoverflow, you just need to define the useragent name there. Here we only specify Googlebot, so that rule is not applied to other search engines.
user-agent: Googlebot
Allow: /$
Disallow: /
As Google said (at the end of page). Use $ when you want to Match URLs that end with a specific string. Here /$ will say allowed to index all the URL that end with / for example example.com/ and exapmle.com/file/ but the next line say don't crawl all the subdolfer, so exapmle.com/file/ and similar others files which end with / or any filename is not allowed to crawl.
Note: There is no need to specify other search engine bot if you want to allowed to crawl, for example.
User-Agent: bingbot
Allow: /
Weather you add above line or not, it doesn't make any difference. People use robots.txt to block something, but by default it is already allowed to crawl when you don't specify. There are so many website on internet, they don't have robots.txt at all, so all crawler by default crawl their site.
-
1It would be helpful to address the other part of the question and also state the rule for robots.txt to allow all other crawlers. Nov 30, 2019 at 15:44
-
Thanks, I'm not sure if this does what I want though?! Is the Allow and Disallow-rules applied only for Googlebot? Nov 30, 2019 at 16:35
-
Yes, it is apply for only Googlebot, since we did not specify other search engine bot, so they will crawl everything.– GoylloDec 1, 2019 at 5:07
-
Will this work for him if his site address doesn't end with "/"? As in:
https://www.example.comas opposed tohttps://www.example.com/?– gnickoDec 3, 2019 at 21:17 -
1@Goyllo a late reply here, nevertheless. I will apply these rules now. Thanks for the help! Dec 8, 2019 at 1:46