I want to allow Google bots to crawl only my index.php and the content of some folders. Would that work?

User-agent: *

Allow: /index*.php
Allow: /folder1/
Allow: /folder2/
Allow: /folder3/
Allow: /folder4/

Disallow: /

The idea is for Google to crawl URLs like:

  • http://example.com/folder1/discussionA/topic/
  • http://example.com/folder1/discussionB/topic/
  • http://example.com/folder1/discussionC/topic/
  • http://example.com/folder2/discussionD/topic/

and exclude everything else.

  • 1
    What is the idea behind the * in /index*.php?
    – MrWhite
    Dec 28 '18 at 13:09
  • You say Google, but use User-agent: *. Do you want to target only Google’s bot, or any bot?
    – unor
    Dec 28 '18 at 17:27

I generally focus on what I don't want them to crawl, but I think your sample seems on target. I would, however, remove the space between the first line and the second:

User-agent: *
Allow: /index*.php
Allow: /folder1/

Mozilla explains it in more detail

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.