7

I have the URL

http://www.example.com/shopping/books/?b=9

and the following robots.txt file:

User-agent: *
Disallow: /?b=9

But when I test this in Google Webmaster Tool's robots.txt tester it is showing allowed when it should be disallowed.

Whilst /?b=9 is fixed, /shopping/books will change with different categories and I need to block them all.

Please tell me what's wrong with my robots.txt.

3

3 Answers 3

6

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard - 0 or more instances of any character) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots ("Google, Bing, Yahoo, and Ask") support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

Reference:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#url-matching-based-on-path-values

4
  • Which should be follow ? Because my directory is /shopping/books/?b=9 Disallow /*/?b=9 Or Disallow: /*/*/?b=9 Nov 25, 2015 at 13:39
  • the answer is in his post
    – Froggiz
    Nov 25, 2015 at 13:46
  • It doesn't really matter, * matches any character (including /). Unless you specifically need to check for 2 path segments then use the former (as stated in my answer).
    – MrWhite
    Nov 25, 2015 at 13:47
  • 3
    Keep in mind that the wildcard (*) is supported by Google but not part of the robots.txt standard. Most bots won't understand it. Nov 25, 2015 at 14:39
0

The answer is on the link i posted :

Disallow: /shopping/*/*?b=9

* is a joker which mean "all"

3
  • We can't do that because numbers of url have as /shopping/mobile/?=9 , /shopping/bags/?=9....... So please give proper solution for this Nov 25, 2015 at 13:43
  • I updated my answer, but you should have found it by yourself
    – Froggiz
    Nov 25, 2015 at 13:52
  • May be wrong.... Didn't clear that Nov 25, 2015 at 13:54
0

I don't think there's such a way to do it in robots.txt and also whatever is advertised in robots.txt is also what can be advertised to hackers because robots.txt is a file accessible to all.

What I would suggest is to use your scripting language to detect for the query string you don't want people to access and if the query string matches, create a redirect to a relevant page people are allowed to access or take them to a page with a 410 HTTP code.

For example, in PHP, you can use either of these to block the b=9 parameter from being accessible:

<?php
if ($_GET['b']=="9"){
header("HTTP/1.1 410 Gone",true);
echo "This page is gone.";
exit();
}
?>

<?php
if ($_GET['b']=="9"){
header("HTTP/1.1 301 Redirect",true);
header("Location: http://example.com/newpage",true);
echo "This page moved <a href=\"http://example.com/newpage\">here</a>";
exit();
}
?>

If you are looking to specifically block just robots and not real users, then you could make the parameters accessible via POST only. Here's the HTML and PHP you can use:

Html:

<form action="phpscript.php" method="POST">
<input type="hidden" name="b" value="9">
<input type="submit" value="special page">
</form>

Php file named phpscript.php:

<?php
if ($_GET['b']=="9" && strtoupper($_SERVER['REQUEST_METHOD']) != "POST"){
header("HTTP/1.1 410 Gone",true);
echo "This page is gone";
exit();
}
?>

Only problem with the post method is that making post requests are generally non-cacheable based requests since they're primarily meant for user data submission.

2
  • I don;t want to go with 410. because need to stop crawling and indexing for this specific pages /?b=9 Nov 26, 2015 at 5:55
  • A 410 status sent to search engine bots means the page is "gone" and that they should stop scanning it. A 404 status means not found and that they can rescan it at a later point hoping it'd come back. Nov 26, 2015 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.