2

I am trying to increase the traffic to my website but I am getting weird Crawler Errors from Google. Google is telling me that it visited the URL below, but it doesn't exist. My sitemap.xml doesn't have any invalid URLs.

http://www.example.com/AboutUs/AboutUs

But the URL above that Google visited is not written in any part of my website, where does Google find that URL? the correct version of the URL is below

http://www.example.com/AboutUs

Another example is like below, the URL below is not valid

http://www.example.com/ContactUs/AboutUs

The correct URL is

http://www.example.com/ContactUs

or

http://www.example.com/AboutUs

Why does Google misunderstand my URLs? How can I fix this?

migrated from webapps.stackexchange.com Sep 15 '17 at 12:17

This question came from our site for power users of web applications.

  • 2
    Is http://www.example.com/AboutUs/ (with the trailing slash) a URL that resolves? If so, does it link to the contact page using relative linking? My guess is that it links with <a href="ContactUs" instead of <a href="/ContactUs" or <a href="http://example.com/ContactUs" – Stephen Ostermiller Sep 15 '17 at 15:10
1

In addition to Stephen's comment below your question, which should help you troubleshoot the issue (i.e. make sure the URL's don't resolve; ensure the links are absolute), consider the following:

The 404's that Google returns in GSC are not always reason for concern. If Google can't find a page or a resource that doesn't exist on your site, isn't ranking in SERPs, or isn't in the index, it is perfectly okay to let it go to a 404. As Google says in GSC, "Generally, 404s don't harm your site's performance in search, but you can use them to help improve the user experience."

If you're using a CMS, those have the ability to produce such errors. You may automate a URL generation process via a plugin/module, and may later find that something's been generated incorrectly. If this can be traced to a plugin, you can either modify your plugin settings and run multiple tests; figure out what's going wrong in the plugin's code (PHP, JS) and if it's conflicting with another plugin; or failing that, use a different plugin.

Some CMS platforms, like Drupal, will generate URL's for nodes like blocks and taxonomies, which should not appear in search. Be on the lookout for this functionality as well, and make sure you "noindex, follow" everything you should.

You can further troubleshoot by using Screaming Frog, or any of the numerous Chrome browser add-ons, to analyze your broken links on the website and see where they're coming from. GSC also gives you this info from its Crawl Errors -> Linked From tab. This will help you discover if the links are internal or coming from another website, allowing you to either fix the links or contact the site owner.

Also, make sure you're using canonical tags on your pages. These tags should be present on every page you want in the index (so all those pages in your XML sitemap), pointing to itself. That way, any additional URL version that is accidentally generated will lead to the page that's meant to be in the index.

-1

1st Option:

find directories in your servers, google will read your all servers paths and files and redirect it to its original urls.

2nd Option:

Fetch as google. for these pages.

  • I don't see how finding directories on the server is going to fix the broken links or help him figure out where Google found the links. Similarly fetch as Google will show that they are broken, but not where Google found them. – Stephen Ostermiller Sep 15 '17 at 15:07
-1

well you can provide google with a sitemap that would help them index your website better and try using canonical links,

and see your URL Parameters in google webmaster tools and you can remove those urls in Remove URLs in google webmaster tools.

  • They said they have a sitemap and these URLs are not in it. I don't see how the URL parameters tool would help since there are no parameters on these URLs. – Stephen Ostermiller Sep 15 '17 at 15:29
  • he may be using #AboutUs in href tag. – M0ns1f Sep 15 '17 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.