Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

Question

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing?

http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404 on random generated links as above all starting with /+ like above link.

I was wondering if there is a quick way other than to manually remove these URLs from the Google Webmaster Tools?

Can we block this with robots.txt to URLs starting with + sign?

Answer 1

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404

A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped from the search engines (ie. Google). If you block crawling then the URL could still remain indexed. (Note that robots.txt primarily blocks crawling, not indexing.)

If you want to "speed up" the de-indexing of these URLs then you could perhaps serve a "410 Gone" instead of the usual "404 Not Found". You could do something like the following with mod_rewrite (Apache) in your root .htaccess file:

RewriteEngine On
RewriteRule ^\+ - [G]

Answer 2

I'm going to answer the 2nd question.

I was wondering if there is a quick way other than to manually remove these URLs from the google webmaster tools?

https://developers.google.com/webmasters/hacked/docs/clean_site

Google is explicitly states that removal through Google Search Console (the new name of webmaster tools) is the fastest.

If the hacker created entirely new, user-visible URLs, you can have these pages more quickly removed from Google search results by using the Remove URLs feature in Search Console. This an entirely optional step. If you simply delete the pages and then configure your server to return a 404 status code, the pages will naturally fall out of Google's index with time.

But they also understand that this is not feasible for some cases:

The decision to use URL Removal will likely depend on the number of new, unwanted pages created (too many pages might be cumbersome to include in Remove URLs), as well as the potential damage these pages could cause users. To keep the pages submitted through URL Removal from ever appearing in search results, be sure the pages are also configured to return a 404 File not Found response for the unwanted/removed URLs.

So while you can block these pages in robots.txt -- you are not taking either of the corrective steps as explained by google.

Answer 3

User-Agent: *  
Disallow: /+

should do what you want. It will tell the robot to not request all URLs starting with a +.

Answer 4

If you really want to use robots.txt this would be a simple answer to your question. Also i have included a link to where you can read on the specifications on robots.txt.

User-agent: *
Disallow: /+

Read about robots.txt specs

But one other alternative might be to use .htaccess to make a rewrite rule (if you use Apache etc) to catch them and perhaps tell Google a better return HTTP code or to simply redirect the traffic to some other page.