0

I manage an ecommerce website that slightly refreshes its product/category collection every night. Sometimes products/categories get disabled and links to them disappear from the website but despite that they get crawled by Google and shown as 404s.

Is it possible that Google crawls website level-by-level and by the time it reaches deeper into links, higher-level pages are updated, some links removed and then it return 404s for these links (that were previously linked on higher levels which were already crawled)?

Is it possible to get rid of these errors?

2

If a URL has previously been crawled (and possibly indexed) then Google will continue to crawl that URL for a considerable time after its removal - when it is returning a 404 Not Found. These will naturally appear in the Google Search Console crawl error report. Webmasters make mistakes; a 404 isn't necessarily saying that the page is never coming back.

Google doesn't simply get the list of URLs to crawl from the current crawl of your site, if that is what you are suggesting? AFAIK Google does not necessarily crawl a website "level-by-level".

Alternatively, you can return a 410 Gone for these pages - to send a stronger signal to Google that these pages are truly gone and are never coming back. Google is then more likely to drop the page from its index quicker.

Or, if you have an alternative (ie. very similar / replacement) page to redirect to then redirect instead.

2
  • Thank you for the answer. 404s in this case are not webmaster's mistake, these are categories/products switched on/off based on availability so there is no other website to redirect to and this resource can come back in the future. So the possible solution would be to leave unavailable products and categories visible so that they can be crawled and no 404 errors appear. But then how to solve it when we have a category with 50 products and all are unavailable. Or maybe it's ok to ignore these 404s, even if there's thousands of them? – Alan Sep 14 '16 at 12:50
  • Ignoring the 404s is OK. The annoyance is that they pollute the report. Keeping the page live (200 OK) and simply stating "product unavailable" could be a solution. This might even be a better response for users (unless this is what your 404 response stated anyway). – MrWhite Sep 14 '16 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.