0

I need to allow only specific pages at some site. Let's say:

/
/page1 
/page2
...

Now I don't get how to whitelist the root page at robots.txt

Disallow: /         # block everything
Allow: /page1
Allow: /page2
Allow: ?            # how to allow the / ???

And I'm stuck here, because Allow: / will allow everything back

Is it even possible?

0

You should be able to use a $ sign which indicates the end of the string.

For example

Allow: /$

If you go to the bottom of Google's robots.txt help, you can see a similar example.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.