0

I am getting lots of crawl errors, Google is adding main domain to the URL:

sitemapxml:

<url><loc>http://www.somedomain.com/whateveryouwant.php</loc><changefreq>monthly</changefreq></url>

From Google Webmaster Tools:

Not found:

http://www.somedomain.com/whateveryouwant.php

When clicking it shows:

http://www.somedomain.com/www.somedomain.com/whateveryouwant.php

As you can see the domain URL is duplicated.

On every page on this site I am using a header and a footer, I have navigation links on these, and they are "included" on each page:

<? include "header.html ?>

The links on the header are absolute:

<a href="/whateveryouwant.php" class="menuFooter">Whatever</a>

Is this affecting crawling?

It is happening almost on every page, any ideas?

Improve this question
2
  • Have you tried fetch as Googlebot? You may have some errant rewriting going on that uses bot or user-agent detection. I've seen similar issues with responsive websites that attempt to detect mobile browsers.
    – jeffatrackaid
    Commented Jul 18, 2014 at 15:18
  • Don't forget to approve an answer if you like it, thanks!
    – Jérôme Verstrynge
    Commented Aug 25, 2014 at 17:32

4 Answers 4

Reset to default
1

Your sitemap example looks to be correct. I would make sure that your whole sitemap is correct before taking any further steps.

Otherwise, it has to be something on your website. But there is not enough information here to determine what exactly is going on.

Check your header and footer includes if the pages listed in Google Webmaster Tools appear to be navigational links. I am not sure what lots is. For some it can be thousands, for others it can be a dozen. If the crawl errors exceed your navigational links, then likely the problem exists somewhere else.

But I have an idea!!

If you can hit your live site, I suggest using a site scanner such as screaming frog http://www.screamingfrog.co.uk/broken-link-checker/ specifically to check for broken links. You can download a free copy from http://www.screamingfrog.co.uk/seo-spider/ which will let you spider 500 pages which should be more than enough to tell you what is wrong and where.

Improve this answer
1
  • thank you for your comments, I will give it a try, downloading seo-spider now.
    – Nicoli
    Commented Jul 18, 2014 at 15:07
1

The link you refer to in your example is not absolute, because it does not contain the http://www.mysite.com/ part. However, there is nothing wrong in using relative paths in your link (i.e, starting with '/'). This is authorized and ok. It is not an issue.

Browse your page and check the page source for the duplication you are reporting. If you see, then the issue in on your side. If you don't see it, use the Fetch as GoogleBot feature in Google Webmaster Tools. If everything still seems ok on your side, then notify the issue on the Google Webmaster Tools forum.

Improve this answer
0

If you are getting crawl errors, then yes it is affecting crawling. I'm guessing how you have your page files and your include files organized on your server is incorrect. Do you have your header.html in a separate directory than the whateveryouwant.php file?

Improve this answer
5
  • All files are on root.
    – Nicoli
    Commented Jul 17, 2014 at 22:35
  • And when you say you're links are absolute, do you have them as http:// www.{url}.com/whateveryouwant.php?
    – user1795832
    Commented Jul 17, 2014 at 22:39
  • No, just with a slash: "/whateveryouwant.php". I will try using the full url ad your example. Thank you!!!
    – Nicoli
    Commented Jul 18, 2014 at 14:42
  • Found the problem, please read my own answer. Thank you!
    – Nicoli
    Commented Jul 18, 2014 at 18:10
  • Absolute links are actually ones that have http: // www in there. The slash in there is still technically a relative URL.
    – user1795832
    Commented Jul 18, 2014 at 23:53
0

Finally found the error.

The problem was in the canonical link:

I had:

<link href="http:www.somedomain.com/whateveryouwant.php" rel="canonical"/>

It should be

<link href="http://www.somedomain.com/whateveryouwant.php" rel="canonical"/>

The "//" were missing.

The Seo-Spider recommended by @closetnoc did help me to find the problem.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.