Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This is simple question but also I have a confusion.

Below is the code which I wrote in my robots.txt:

User-Agent: *
Disallow: /phpmyadmin/  
Disallow: /login/
Disallow: /newsletter/
Disallow: /articlesubmission/  
Allow: /

Sitemap: http://www.example.com/sitemap.xml

I want all bots to follow my site but they should not follow these URLs:

http://www.example.com/phpmyadmin/.....
http://www.example.com/login/.....
http://www.example.com/newsletter/..... 
http://www.example.com/articlesubmission/..... 

My doubt: Is my above robots.txt code is correct?
I followed this code from StackOverflow robots.txt.

share|improve this question

closed as off-topic by Stephen Ostermiller Jul 26 '13 at 21:45

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because it is unlikely to help anybody else. Examples include individual website reviews for SEO purposes and general website troubleshooting." – Stephen Ostermiller
If this question can be reworded to fit the rules in the help center, please edit the question.

2  
Yes, your robots.txt is correct. –  Zistoloen Jul 26 '13 at 8:52
2  
@Danny Looks good :-) –  dan Jul 26 '13 at 8:57
1  
@Danny Based on the amazingly quick crawling and indexing StackOverflow receives, I don't think their webmaster is wrong ;-) –  dan Jul 26 '13 at 15:17
1  
@Zistoloen thank you for the confirmation. –  Sundara Jul 26 '13 at 15:52
1  
@dan Thanks dan. I was little bit confused with this that's why though to ask you –  Sundara Jul 26 '13 at 15:53

1 Answer 1

Yes, you have defined your entries correctly.

Just to be clear, adding the name of a directory with a trailing slash will block that directory and everything within it; including if there is a live page on the directory address.

However your Allow: / is superfluous and will have no effect as you are already allowing the entire site apart from the directories you have disallowed, so this entry should be removed.

You can find more info on how robots.txt works with Google here: Robots.txt Specifications

share|improve this answer
    
Thank you for your answer.Now i understood. I am very confusing person. –  Sundara Jul 26 '13 at 15:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.