0

This is simple question but also I have a confusion.

Below is the code which I wrote in my robots.txt:

User-Agent: *
Disallow: /phpmyadmin/  
Disallow: /login/
Disallow: /newsletter/
Disallow: /articlesubmission/  
Allow: /

Sitemap: http://www.example.com/sitemap.xml

I want all bots to follow my site but they should not follow these URLs:

http://www.example.com/phpmyadmin/.....
http://www.example.com/login/.....
http://www.example.com/newsletter/..... 
http://www.example.com/articlesubmission/..... 

My doubt: Is my above robots.txt code is correct?
I followed this code from StackOverflow robots.txt.

  • 2
    Yes, your robots.txt is correct. – Zistoloen Jul 26 '13 at 8:52
  • 2
    @Danny Looks good :-) – dan Jul 26 '13 at 8:57
  • 1
    @Danny Based on the amazingly quick crawling and indexing StackOverflow receives, I don't think their webmaster is wrong ;-) – dan Jul 26 '13 at 15:17
  • 1
    @dan Thanks dan. I was little bit confused with this that's why though to ask you – ashutosh Jul 26 '13 at 15:53
2

Yes, you have defined your entries correctly.

Just to be clear, adding the name of a directory with a trailing slash will block that directory and everything within it; including if there is a live page on the directory address.

However your Allow: / is superfluous and will have no effect as you are already allowing the entire site apart from the directories you have disallowed, so this entry should be removed.

You can find more info on how robots.txt works with Google here: Robots.txt Specifications

  • Thank you for your answer.Now i understood. I am very confusing person. – ashutosh Jul 26 '13 at 15:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.