3

I'm trying to create a robots.txt file that allows bots to access ONLY the home page and no other page. Below is the content of the robots.txt file based on the research I have done.

Is this correct?

user-agent: *
Allow: /$
Disallow: /
5
  • Why allow bots only on the main page? Most sites get the majority of their search engine traffic to deep URLs. Preventing search engine bots from crawling deep pages may significantly hurt your SEO. Commented Mar 7, 2022 at 17:57
  • Thanks. Yes its something to reconsider. My website is mainly a listings website for a niche product so the pages other than than the home page are the categories and the individual listings pages.
    – The Oracle
    Commented Mar 8, 2022 at 9:37
  • I would want to have pages like that indexed in search engines if I were you. Commented Mar 8, 2022 at 10:22
  • Thanks for advise. Why is that? just to increase surface area?
    – The Oracle
    Commented Mar 8, 2022 at 10:34
  • 1
    Essentially yes. Search engines index content. Your home page is going to contain less than 1% of the content on your site. Your home page will rank for searches for your brand name, but your deep pages will rank for people searching for the particular products you are selling or product categories. Ranking for that type of query is very lucrative compared to brand traffic. Commented Mar 8, 2022 at 10:37

1 Answer 1

2

To answer your specific question... yes, these robots.txt directives would do as you are asking. Blocking everything except the root directory. It is correct to have the Allow directive first for the few crawlers that use a first-match policy (as opposed to longest match).

The easiest way to test this is to use Google's own robots.txt tester:

Root directory is "allowed":

Root directory is allowed in Google's robots.txt tester

Everything else is "blocked":

Everything else is blocked in Google's robots.txt tester

However, whether you should be doing this or not is another matter. From the discussion in comments, it sounds as if these "other" pages form part of your content and should be crawlable (and indexed) by search engines after all. (?)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.