2

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else.

I'd prefer to do this in robots.txt. How can I achieve this?

  • I'm assuming you know how robots.txt works, since you prefer it. What rules have you tried? – GDav Nov 29 '19 at 13:14
  • 1
    Do you want Google to only be able to crawl the home page, or only be able to index the home page? These are two different goals with different best ways to achieve them. Sometimes Google is known to index pages without crawling them. – Maximillian Laumeister Nov 29 '19 at 18:23
  • 1
    It is very unusual for a site to only have their home page indexed in search engines. You tagged this as "SEO" and what you are requesting is the opposite of SEO. It is really bad for SEO to have just one page in search engines. – Stephen Ostermiller Nov 30 '19 at 12:10
  • @StephenOstermiller there are other search enginges than google, hence the SEO-tag. – saturnusringar Nov 30 '19 at 16:32
  • @MaximillianLaumeister ideally, both: only crawl the home page AND only index the home page. All other bots (more or less) will be allowed to crawl and index the site. – saturnusringar Nov 30 '19 at 16:33
3

It is already answered in Stackoverflow, you just need to define the useragent name there. Here we only specify Googlebot, so that rule is not applied to other search engines.

user-agent: Googlebot
Allow: /$
Disallow: /

As Google said (at the end of page). Use $ when you want to Match URLs that end with a specific string. Here /$ will say allowed to index all the URL that end with / for example example.com/ and exapmle.com/file/ but the next line say don't crawl all the subdolfer, so exapmle.com/file/ and similar others files which end with / or any filename is not allowed to crawl.

Note: There is no need to specify other search engine bot if you want to allowed to crawl, for example.

User-Agent: bingbot
Allow: /

Weather you add above line or not, it doesn't make any difference. People use robots.txt to block something, but by default it is already allowed to crawl when you don't specify. There are so many website on internet, they don't have robots.txt at all, so all crawler by default crawl their site.

| improve this answer | |
  • 1
    It would be helpful to address the other part of the question and also state the rule for robots.txt to allow all other crawlers. – Stephen Ostermiller Nov 30 '19 at 15:44
  • Thanks, I'm not sure if this does what I want though?! Is the Allow and Disallow-rules applied only for Googlebot? – saturnusringar Nov 30 '19 at 16:35
  • Yes, it is apply for only Googlebot, since we did not specify other search engine bot, so they will crawl everything. – Goyllo Dec 1 '19 at 5:07
  • Will this work for him if his site address doesn't end with "/"? As in: https://www.example.com as opposed to https://www.example.com/? – Greg Nickoloff Dec 3 '19 at 21:17
  • 1
    @Goyllo a late reply here, nevertheless. I will apply these rules now. Thanks for the help! – saturnusringar Dec 8 '19 at 1:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.