1

My site was hacked about 4-6 months ago and as a result lots of pages were created on my site. These pages no longer exist and were removed a week after the hack but they still exist in googles search results and i have over 1,000 crawl errors in google search console. I thought google would have removed these pages by now.

The pages all seem to refer to the following path:

mydomain.com/glpkvn[number here]/lity/[number here]

Where [number here] is a randomly generated number.

What is the best way to remove these from the google search results and also tidy up my google search console?

Thanks

  • This is tough. Because of the sheer number of pages, I do not think the Google Search Console - Remove URL option is necessarily a good idea. I would have thought that a 404 would have helped by now or at least started to remove the URLs enough to give you some confidence. I suggest using the robots.txt file for this. However, how exactly is unclear. I do not understand the mention of the random number where the example does not contain numbers. Can you clarify this in your question? Cheers!! – closetnoc Aug 26 '17 at 15:38
  • Sorry, i must have used reserved symbols with the greater than and less than signs – Olly Aug 26 '17 at 15:57
  • I have edited it now to show what i mean – Olly Aug 26 '17 at 15:57
  • Okay. Someone should be able to answer this better now. This should be fairly simple to solve. I have a friend coming up from about 2 hours away so I will be away for much of the rest of the day. However, in the late evening I should have some time if you do not have an answer. Cheers!! – closetnoc Aug 26 '17 at 16:18
  • Sorry that it took me so long to respond. You have a couple of options. Cheers!! – closetnoc Aug 27 '17 at 18:03
3

What is the best way to remove these from the google search results

Make such affected pages return a 410 HTTP code.

You can use the apache mod_rewrite module (or equivalent) and write a server configuration script that checks to see if a certain pattern exists in the URL and if it does, then user will see a 410 page.

If your webserver is apache, then in the document root folder create a file named .htaccess and depending on your specific situation, you can add any of these lines:

RewriteRule ^glpkvn([0-9]+)/lity/([0-9]+)$ [R=410,L]

This line (above) checks to see if the URL is http://example.com/glpkvn####/lity/#### (where #### is any number of numerical digits) and if theres a match, then rule processing stops and user gets sent to a page with a 410 HTTP status.

RewriteRule ^glpkvn(.*)$ [R=410,L]

This line checks to see if the URL starts with http://example.com/glpkvn and if it does then return the 410 HTTP status page.

If you want to allow case insensitivity (meaning you want the URL to start with http://example.com/glpkvn or http://example.com/GLPkvn) then add an NC in the options like so:

RewriteRule ^glpkvn(.*)$ [R=410,L,NC]

The reason you need to use 410 status is because 410 stands for gone forever and it signals to Google that it should never attempt to access the page again.

and also tidy up my google search console?

Do the above steps first then when you access search console, delete the bad URLs.

  • Thanks for the above. It turns out that whatever hacked it also got to other sites on the same hosting plan. If for example on one of the other sites i had was full of spam links such as www.website.com/<varyingnumber>/<varyingnumber>/frrpvut would i be correct in thinking the rule would be RewriteRule ([0-9]+)/([0-9]+)/frrpvut [R=410,L] would work? Thanks again – Olly Nov 1 '17 at 21:19
  • The rules would also require a dash e.g. RewriteRule ^glpkvn([0-9]+)/lity/([0-9]+)$ - [R=410,L] – xylar Jun 1 '18 at 11:46
2

While I am a firm believer of a 410 over a 404 error response, this is highly dependent upon Google actually visiting each page one at a time. If your site does not enjoy frequent visitations from Googlebot as a result of being considered a highly fresh and trendy site, then this would mean that it could take quite some time for Google to find each page before removing them.

When a site is hacked, it is often quite impossible that each URL be removed using the Remove URL option in Google Search Console though this remains an option with limitations of course. More on this later.

One potentially faster option is to use the robots.txt file.

Google will visit the robots.txt each time it visits your site providing that it has not fetched a fresh copy of the robots.txt file within 24 hours. This is seen as a reasonable compromise to fetching the robots.txt file each time Google visits or fetching the robots.txt too infrequently. Prior, there was no standard for this and there were always detractors for either reading the robots.txt file too frequently or not frequently enough. Yes. Sometimes Google cannot win.

When the robots.txt is fetched, it is saved within the index and applied as Googlebot goes about it's business. However, there is also a process that applies regular expressions (regex) rules easily derived from the rules found within the robots.txt and removes URLs and pages found within the index. This is not done immediately, likely to avoid short-term mistakes made by the webmaster, however, because robots.txt is taken very seriously as pivotal rules mechanism for well behaved robots, Google will apply it fairly quickly. It may still take days or weeks, however, it is done in bulk.

For this reason, the robots.txt is often the fastest way to remove URLs providing that they can be specified by a pattern. While not every search engine treats the robots.txt directives equally, fortunately, Google does allow wildcards giving you a serious advantage.

User-agent: Googlebot
Disallow: /glpkvn*/

According to page: https://support.google.com/webmasters/answer/6062596?hl=en&ref_topic=6061961 under Pattern-matching rules to streamline your robots.txt code, you will see a similar example.

Google does not guarantee that the URLs will be removed and states that it will take some time to remove the URLs.

https://support.google.com/webmasters/answer/7424835?hl=en&ref_topic=6061961#h17

https://support.google.com/webmasters/answer/7424835?hl=en&ref_topic=6061961#h18

However, it has been my experience that this method works and works faster than waiting for Google to fetch each page one at a time.

One warning. If you do block Google from fetching these pages via the robots.txt file, Google will not see a 404 or 410 error for the page. You have to choose one method or another. Google does recommend using the Google Search Console to remove URLs.

I prefer to wait for Google to remove pages naturally using a 404. A 410 error is faster since each 404 is retested several times before removing. However, given that your site has been hacked and these pages remain within the search results, it may be wise to attempt to remove the pages using another method. I have personally removed pages in bulk using this method though it was a couple of years ago. Which one you use is up to you.

  • Thanks for all the above, i did put a few into the Google Search Console to remove the URLs but it seemed to suggest this was only a temporary fix hence why i thought i would ask on here?I will look at amending the robots.txt file as you seem to suggest this will result in faster results than the .htacess file. When the listings are removed from google i presume i will be safe to tidy up my robots.txt afterwords? – Olly Aug 29 '17 at 18:39
  • Yep. You got it! I have personally used the robots.txt to remove pages a couple of years ago. It can still take some time. I added a thought just now to the answer where I explain that not immediately applying the robots.txt rules by Google is likely to avoid short-term webmaster mistakes. It used to be just a few days or one or two weeks. When it appears all is cleared, you can restore your robots.txt file as you like. Still, it does not hurt to leave a rule in longer just in case. Cheers!! – closetnoc Aug 29 '17 at 18:45
  • I followed the above for the robots.txt recommendation but found that google now says "A description for this result is not available because of this site's robots.txt" . Do these ever dissappear or should i try following one of the other suggestions also? Thanks again! – Olly Nov 1 '17 at 21:00
  • Assuming that Google is giving this message for the URLs you do not want and not site wide, then that may be normal. Google, mistakenly in my opinion, will sometimes list URLs in the SERPs where there is a link and where the robots.txt file disallows it. Ultimately, these should just go away. I am a fan of using 404 errors or 410 errors where you can. If you do, remove the entry from robots.txt so that the errors can be seen. As a note, I have never had an issue with using robots.txt. There must be links to these bad pages somewhere on the net. Cheers!! – closetnoc Nov 1 '17 at 23:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.