5

I have the URL

http://www.example.com/shopping/books/?b=9

and the following robots.txt file:

User-agent: *
Disallow: /?b=9

But when I test this in Google Webmaster Tool's robots.txt tester it is showing allowed when it should be disallowed.

Whilst /?b=9 is fixed, /shopping/books will change with different categories and I need to block them all.

Please tell me what's wrong with my robots.txt.

2

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard - 0 or more instances of any character) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots ("Google, Bing, Yahoo, and Ask") support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

Reference:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#url-matching-based-on-path-values

  • Which should be follow ? Because my directory is /shopping/books/?b=9 Disallow /*/?b=9 Or Disallow: /*/*/?b=9 – Amit Rajawat Nov 25 '15 at 13:39
  • the answer is in his post – Froggiz Nov 25 '15 at 13:46
  • It doesn't really matter, * matches any character (including /). Unless you specifically need to check for 2 path segments then use the former (as stated in my answer). – MrWhite Nov 25 '15 at 13:47
  • 1
    Keep in mind that the wildcard (*) is supported by Google but not part of the robots.txt standard. Most bots won't understand it. – Stephen Ostermiller Nov 25 '15 at 14:39
0

The answer is on the link i posted :

Disallow: /shopping/*/*?b=9

* is a joker which mean "all"

  • We can't do that because numbers of url have as /shopping/mobile/?=9 , /shopping/bags/?=9....... So please give proper solution for this – Amit Rajawat Nov 25 '15 at 13:43
  • I updated my answer, but you should have found it by yourself – Froggiz Nov 25 '15 at 13:52
  • May be wrong.... Didn't clear that – Amit Rajawat Nov 25 '15 at 13:54
0

I don't think there's such a way to do it in robots.txt and also whatever is advertised in robots.txt is also what can be advertised to hackers because robots.txt is a file accessible to all.

What I would suggest is to use your scripting language to detect for the query string you don't want people to access and if the query string matches, create a redirect to a relevant page people are allowed to access or take them to a page with a 410 HTTP code.

For example, in PHP, you can use either of these to block the b=9 parameter from being accessible:

<?php
if ($_GET['b']=="9"){
header("HTTP/1.1 410 Gone",true);
echo "This page is gone.";
exit();
}
?>

<?php
if ($_GET['b']=="9"){
header("HTTP/1.1 301 Redirect",true);
header("Location: http://example.com/newpage",true);
echo "This page moved <a href=\"http://example.com/newpage\">here</a>";
exit();
}
?>

If you are looking to specifically block just robots and not real users, then you could make the parameters accessible via POST only. Here's the HTML and PHP you can use:

Html:

<form action="phpscript.php" method="POST">
<input type="hidden" name="b" value="9">
<input type="submit" value="special page">
</form>

Php file named phpscript.php:

<?php
if ($_GET['b']=="9" && strtoupper($_SERVER['REQUEST_METHOD']) != "POST"){
header("HTTP/1.1 410 Gone",true);
echo "This page is gone";
exit();
}
?>

Only problem with the post method is that making post requests are generally non-cacheable based requests since they're primarily meant for user data submission.

  • I don;t want to go with 410. because need to stop crawling and indexing for this specific pages /?b=9 – Amit Rajawat Nov 26 '15 at 5:55
  • A 410 status sent to search engine bots means the page is "gone" and that they should stop scanning it. A 404 status means not found and that they can rescan it at a later point hoping it'd come back. – Mike Nov 26 '15 at 16:58
-1

Doesn't a self altering text configuration file suggest an issue with your directories and actual ability to reach/edit that file? Not to cause panic, but... the input you entered changed....I don't think it's a text file issue.

  • Which question are you answering? – MrWhite Nov 27 '15 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.