15

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing?

http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404 on random generated links as above all starting with /+ like above link.

I was wondering if there is a quick way other than to manually remove these URLs from the Google Webmaster Tools?

Can we block this with robots.txt to URLs starting with + sign?

migrated from serverfault.com Nov 22 '16 at 11:43

This question came from our site for system and network administrators.

  • 2
    There is nothing special about + (plus) in the URL-path, it is just a character like any other. – MrWhite Nov 22 '16 at 13:01
  • You could Apache redirect (in .htaccess) to a file or directory which robots.txt forbids robots ot access – Mawg Nov 23 '16 at 12:47
  • @Mawg What's the point of doing that? – MrWhite Nov 23 '16 at 22:08
  • To keep well behaved robots out? – Mawg Nov 23 '16 at 23:00
  • 2
    Independent of the issue with the URLs, you might want to read How do I deal with a compromised server? – Jonas Schäfer Nov 24 '16 at 7:34
30

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404

A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped from the search engines (ie. Google). If you block crawling then the URL could still remain indexed. (Note that robots.txt primarily blocks crawling, not indexing.)

If you want to "speed up" the de-indexing of these URLs then you could perhaps serve a "410 Gone" instead of the usual "404 Not Found". You could do something like the following with mod_rewrite (Apache) in your root .htaccess file:

RewriteEngine On
RewriteRule ^\+ - [G]
14

I'm going to answer the 2nd question.

I was wondering if there is a quick way other than to manually remove these URLs from the google webmaster tools?

https://developers.google.com/webmasters/hacked/docs/clean_site

Google is explicitly states that removal through Google Search Console (the new name of webmaster tools) is the fastest.

If the hacker created entirely new, user-visible URLs, you can have these pages more quickly removed from Google search results by using the Remove URLs feature in Search Console. This an entirely optional step. If you simply delete the pages and then configure your server to return a 404 status code, the pages will naturally fall out of Google's index with time.

But they also understand that this is not feasible for some cases:

The decision to use URL Removal will likely depend on the number of new, unwanted pages created (too many pages might be cumbersome to include in Remove URLs), as well as the potential damage these pages could cause users. To keep the pages submitted through URL Removal from ever appearing in search results, be sure the pages are also configured to return a 404 File not Found response for the unwanted/removed URLs.

So while you can block these pages in robots.txt -- you are not taking either of the corrective steps as explained by google.

4
User-Agent: *  
Disallow: /+

should do what you want. It will tell the robot to not request all URLs starting with a +.

2

If you really want to use robots.txt this would be a simple answer to your question. Also i have included a link to where you can read on the specifications on robots.txt.

User-agent: *
Disallow: /+

Read about robots.txt specs

But one other alternative might be to use .htaccess to make a rewrite rule (if you use Apache etc) to catch them and perhaps tell Google a better return HTTP code or to simply redirect the traffic to some other page.

  • 2
    There is no need for the * (asterisk) at the end of the URL-path. It should be removed for greatest spider-compatibility. robots.txt is already prefix matching, so /+* is the same as /+ for bots that support wildcards, and for bots that don't support wildcards then /+* will not match at all. – MrWhite Nov 22 '16 at 12:59
  • You are right, i just wrote that based on his question about Googlebot. I have edited it to reflect better compatibility against multiple bots. – davidbl Nov 22 '16 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy