0

Many months ago I deactivated one store (localised to a language) within a Magento site, leaving the english store the only store live.

Looking in my Apache access logs, and also summarised in Logwatch, every day I still get GET requests from web crawlers (e.g. yahoo) that result in 404s, because they're trying to access pages from the deactivated site.

The sitemap for the deactivated store was removed long ago. Why are these 404s still happening and how would I go about stopping them?

0

This is normal business. Search engines are notoriously slow. The reason for this is simple- the Internet is flippin' HUGE!!

The best thing to do with the least amount of work is to let the 404's happen. Do not stop this process. This is the primary way that search engines know that a page is gone.

The very best thing to do is to issue a 410 error if you can. But that can take work that is not always easy to do.

The difference between a 404 (temporarily gone) and a 410 (gone) is that the 410 is immediate whereas the 404 is not. With the 404, Google for example, will retry the page several times over a period of time before determining that the page is truly gone. This is the standard way of course, but not as fast as a 410.

You do not want to stop or block this process. Otherwise, the search engines cannot know these pages have been removed.

If you have bots accessing your site that you do not want or appreciate, you can block those of course and you may want to for a period at least.

  • It must be very slow, the store was taken offline at least 9 months ago – Michael Oct 14 '15 at 7:57
  • @kaska That is quite a while, but no that uncommon. It all depends upon how fresh Google thinks the pages were on how fast it will access them. There is a certain number of retries (I do not know the number- have not counted) before it gives up and figures out the page is truly gone. Still if a link exists anywhere on the net, it is possible that Google will follow that link from time to time just to make sure. Any new link will certainly be followed. Frustrating I know. But what is Google the machine to do?? – closetnoc Oct 14 '15 at 13:26
0

web crawler goes in, where it is allowed. You can prevent their come ins with

  • deleting the folder,
  • closing it with htpassword,
  • placing in the root of the folder a htaccess with Header set X-Robots-Tag "noindex, nofollow"
  • check, that urls from this folder don't have backlinks from outta space

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.