13

I am working on an e-commerce website. I used to have a sitemap with category Ids, something like:

<url>
    <loc>https://my-domain.com/home/browse/2/45/139</loc>
    <changefreq>daily</changefreq>
</url>
<url>
    <loc>https://my-domain.com/home/browse/5/60/160</loc>
    <changefreq>daily</changefreq>
</url>

In the above site map 2/45/139 represents a category:

fashion/women/tops-and-shirts

and 5/60/160 represents a different category:

sports/team-sports/football

Around 3 months ago, I decided to change my website's sitemap and use the category names instead of category Ids, so the new site map looks like this:

<url>
    <loc>https://my-domain.com/home/browse/fashion/women/tops-and-shirts</loc>
    <changefreq>daily</changefreq>
</url>
<url>
    <loc>https://my-domain.com/home/browse/sports/team-sports/football</loc>
    <changefreq>daily</changefreq>
</url>

I have submitted the new sitemap to Google months ago and everything is working... but every time I go to Google Search Console, I see Google is showing me errors about pages with old sitemap structure... or Google is adding random parameters to the url, e.g. page=59 and then complains that this page has error... it returns error because it does not exist!

enter image description here

How can I resolve these errors?

28

You have a misconception of what a sitemap is.

The sitemap is used to audit the site crawl by the search engine bot. The sitemap and crawling a site are two different and independent things. Google will continue to crawl your site independently of any sitemap. The sitemap will be used to audit/see if Google is able to properly crawl your site. For example, if pages are found in your sitemap and Google has not seen the page, Google may add the page to the fetch queue to be included.

The converse is not true. If a page is not found in the sitemap, Google will not remove it from it's index. Why? Because Google found it by crawling the site.

What you seem to believe is the sitemap is the be all - end all authority that Google uses to know what pages exist on any particular site. This is not the case. The crawl is. The sitemap only helps Google know whether they can properly crawl your site and, if not, what pages Google are missing that should be added to the fetch queue.

Your expectation, that Google will no longer try to access pages because these pages are no longer in your sitemap, is incorrect. Sitemaps are cached and only checked periodically. Why? Because it is an audit process.

You do have a real problem you need to solve.

You are returning a 500 error for pages that are not being found. This is bad. Your site should be returning a 404 Not Found error. The 500 error is a system error and Google will treat the condition as temporary. If your site returned a 404 error, Google will still continue to try the page for a number of tries over a period of time until it decides the page no longer exists. If at all possible, you want to issue a 410 Removed error for pages that you have removed. If this is too much work or not possible, the 404 will amount to the same thing over time.

You do need to fix your 500 error.

  • 17
    …or, better yet, return a 301 redirect to the corresponding new URL, like Stephen Ostermiller suggests. This also helps any visitors coming to your site through old links from other sites, or from stale search engine results that haven't been updated yet. – Ilmari Karonen Oct 25 at 13:12
  • 4
    It's worth mentioning that the behaviour OP wrongly expects from a sitemap can be, to a degree, implemented from a robots.txt – GeoffAtkins Oct 25 at 13:29
  • 4
    I wouldn't recommend robots.txt in this situation. You do want Googlebot to crawl your old URLs once you implement the redirects. Robots.txt can prevent crawling, but it won't always prevent indexing. There isn't a great pattern to use anyway. Best would probably be ten rules like Disallow: /home/browse/1, assuming that none of the new categories start with a digit. – Stephen Ostermiller Oct 25 at 19:04
  • 4
    I appreciate this answer's content, but it leaves out the most critical piece of information: the OP's customers might have bookmarked or shared page links, and without redirects, these users or their friends will end up on dead-links. This will lead to a bad user experience and lost revenue. – Spencer D Oct 26 at 2:56
18

Closetnoc is correct about sitemaps. Don't expect them to limit what URLs Google will crawl and index. In fact sitemaps have little to no influence over SEO. See The Sitemap Paradox

Google won't complain about errors from your old URLs if you redirect them. When you change your site's URL structure it is best to redirect all the old URLs to their corresponding new URLs. Redirecting is better for search engines because it preserves your SEO value and rankings (usually). It is better for users because if they do happen to get to the old URL, they are automatically taken to the new URL.

So make sure your site implements proper redirects that use the "301 Permanent" status:

/home/browse/2/45/139 -> /home/browse/fashion/women/tops-and-shirts
/home/browse/5/60/160 -> /home/browse/sports/team-sports/football

Google doesn't add random parameters to the URLs. All URLs that it crawls it finds somewhere. It likely found the links to the pagination on your own site. Googlebot also has dumb heuristics where it scans JavaScript for string literals that look like they could be URLs and crawls those. The parameters could also come external links. Occasionally other sites can randomly link to your site in weird broken ways.

If you no longer have pagination, it is fine to redirect those requests too. Even if you never had pagination, it would probably be fine to redirect to remove pagination parameters.

0

There are 2 things you need to do.

  1. Check for a link like this one on your page:

    something

This should be:

<a href="https:domain.com/?page=59"> something </a>

This is because you are using a "/" delimiter in the URL, which is a directory seperator. So if google crawler is on domain/home/browse/fashion

and it finds a link like the first one on this page, then it will add that link to this page, and not to main domain. So google will look for: domain/home/browse/fashion/?page=59 and not: domain/?page=59

  1. Now that you are using category names instead of numerical identifier, then you should redirect those old pages to new ones. This way you will not lose any search ranks which any of those old pages used to get.

If you do not redirect, and simply change the URL in html link and sitemaps, then google will consider these (category name URLs) as new pages and might not rank them.

Without redirect, you will also lose any link juice which is coming to your site because someone else linked to those old pages.

The 500 internal error is probably coming from a broken function. A function which earlier needed a numerical value, but you are now passing an alphabetical value to it, which is causing a fatal error of some kind. Look into your error logs for details about this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.