robots.txt and disallow

Question

Which one is correct?

User-agent: *
Disallow: index.php

or

User-agent: *
Disallow: /index.php

inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30

John Conde · Accepted Answer · 2011-05-02 13:13:23Z

3

The second one is better form as it clearly marks the index.php as being the in web root and not in some other subdirectory.

answered May 2, 2011 at 13:13

John Conde♦

86.4k27 gold badges147 silver badges241 bronze badges

Isn't the robots.txt used to block directories and not files?
– Alexis Wilke
Commented Feb 9, 2012 at 0:02
It can block both
– John Conde ♦
Commented Feb 9, 2012 at 0:12

Add a comment |

Rajesh Durai · Accepted Answer · 2011-05-02 21:00:39Z

You can use the following robots.txt file if you wish to block all robots from accessing your pages.

# robots.txt for www.domain.com to block all bots

User-agent: * 
Disallow: /

If you wish to block all all URLs that include a question mark (?) you can use the following format

# robots.txt for www.domain.com to block URL with ? mark

User-agent: *
Disallow: /*?

For more detailed information on how to write robots.txt visit the following link How to Create Robots.txt Files

2 Answers 2