1

I am helping on a website, it was redone a few months ago. We have 301 redirection links set, uploaded a new sitemap (with the updated links) and submitted it. But only one page is getting indexed (the homepage). When I check the Craw Errors page there are multiple (300+) 404 errors and a few server errors (error 500). I have clicked on those links and checked where they are being linked to from and I found that it is the old sitemap (the new sitemap was uploaded in the old ones place). So I deleted the sitemap (from the website and webmaster tools) and uploaded the sitemap under a different name and submitted it. It's been a few weeks now and there is still only 1 page indexed (the homepage). So how come the old sitemap is being indexed and crawled (even though it was overwritten and then deleted a few weeks ago) rather than the new one?

Improve this question
3
  • If there are 301 redirects in place, why are there so many 404 errors?
    – Stephen Ostermiller ♦
    Jan 15, 2014 at 20:50
  • I'm not sure, I was told it was in place (it might have been removed).
    – user35233
    Jan 15, 2014 at 21:01
  • What do you mean by this: uploaded the sitemap under a different name - was the site verified under that?
    – dan ♦
    Jan 16, 2014 at 5:15

1 Answer 1

Reset to default
1

First things first, a sitemap is only necessary if a search engine crawler is encountering difficulty crawling your website or your website contains a complex structure and hierarchy several levels deep. A well built website rarely actually requires a sitemap and with inbound links pointing to the site, a search engine crawler will do their business in crawling and indexing what it is allowed and instructed to do so.

So in this sense, I wouldn't worry overly about your sitemap.

However, it sounds like there is likely more critical issues on your website than whether the sitemap is being picked up (or it could be that it is, but all the links in the new sitemap are 404'ing due to other issues).

I'd suggest you test a few URL's (versions before the redirect and after the redirect) in a header response tool and see what header response code is being returned.

  • Any URL's you wish to be indexed should return a 200 OK code.
  • Any old URL's that are being redirected to new destinations should return a 301 redirect code.
  • Any new URL's that have old URL's redirecting to them should be returning a 200 OK code (although you could well find that they are in fact returning a 404 code due to a problem with the redirects).

I'd also recommend using a tool such as Screaming Frog or Xenu to crawl your website, either of these will display what header response code is being returned for every URL's on your website, highlighting any crawlability issues.

Within Google Webmaster Tools, you can also fetch URL's as GoogleBot to see what is being returned when it visits your URL's. You'll also be able to see a list of the URL's returning 404's so you can drill down deeper into the root/cause of the problem.

Improve this answer
2
  • I created the sitemap using a sitemap generator and all the links on it are linking to content that is there (I have checked). Other search engines have indexed the new website (i.e. yahoo and bing) and I have used the fetch url as googlebot and it's finding pages fine (i.e. the fetch status is a success), I have submitted those pages as well, but nothing has changed,
    – user35233
    Jan 16, 2014 at 10:01
  • I'll say it again but forget the sitemap for a moment, it's irrelevant. Please can you go to your Google Webmaster Tools, pull up a URL that is 404'ing and paste it either in a comment here or in your original question so we can see what the problem is? Did you use any of those tools to crawl your website and see if the mass of 404 errors are also reported via these?
    – zigojacko
    Jan 16, 2014 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.