4

In my website, most of the pages has a linkedin sharing button written in this way:

<li>
    <a id="share_linkedin" 
       href="/link/share/linkedin/Dance+Combo+%28Jazz%2C+Tap%2C+Contemporary+Dance%29+%28Age+16+or+above%29" 
       rel="external nofollow" 
       class="share_icon" title="LinkedIn" target="_blank">
        <span></span>
    </a>
</li>

I'm sure this is the only code that I use in all the pages for the button.

I have also blocked crawling of all the /link/share/ URLs in the robots.txt file.

However, I saw many "Restricted by robots.txt" errors in Google Webmaster. These errors are all associated to /link/share/linkedin/ URLs. Almost every day I got hundreds of such errors, I'm afraid these daily errors will cause negative impact on the ranking of the website.

So I'm wonder why google will report error for "nofollow" links, and what's wrong with my codes?

Update: I've got 400 such errors in 4/9/2011 and 370 in 3/9/2011. If there is nothing wrong with the website, then is there an option to get rid of it?

Improve this question
3
  • 2
    When you ask about "getting rid of errors", do you mean in the Webmaster Tools, or in your web server logs?
    – Mufasa
    Commented Sep 6, 2011 at 10:48
  • I mean hiding the "restricted by robots.txt" errors in Webmaster tool so that I can easily focus on other errors that really need my action. Now there are hundreds of "restricted by robots.txt" error every day that I can't notice any "real" errors.
    – LazNiko
    Commented Sep 7, 2011 at 0:41
  • 1
    You've restricted following the link in robots.txt. Therefore the spider is restricted from reading the link to find the nofollow directive and generates the error because robots.txt overrides all. Catch 22 situation with chicken/egg precedence error.
    – Fiasco Labs
    Commented Oct 11, 2014 at 17:43

3 Answers 3

Reset to default
5

According to Google:

In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web. However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap.

Google doesn't gaurentee that they won't crawl the URL, just that the contents won't affect your PageRank. In fact, Wikipedia entry nofollow explicitly claims that Google does crawl "nofollow" links, but again, doesn't use the contents to affect your PageRank any.

The Restricted by robots.txt message is fine--it won't affect your PageRank either. It is informational only.

You could drop the "external" part of the attribute though; using the standard rel="nofollow" is probably safer for crawler parsers that aren't as robust.

Improve this answer
1
  • After reading the wikipedia entry about nofollow, I think my situation is a prove that google does FOLLOW the nofollow links.
    – LazNiko
    Commented Sep 29, 2011 at 13:21
1

"Restricted by robots.txt" errors in Google Webmaster.

In other words, your nofollow directive is being noted by Google, obeyed and thus not an error. Working as designed by Google and not anything to worry about.

Improve this answer
3
  • I think it's a bit suspicious. If the nofollow is working, then google shouldn't mention robots.txt anyway. Normally when it mentions robots.txt, meaning that it does try to crawl the page but denied by robots.txt, isn't it? also, it generate so many such errors every day that makes users hard to find the "real" error that should take actions...it seems not very reasonable. Is there a way to get rid of such errors?
    – LazNiko
    Commented Sep 6, 2011 at 4:39
  • 1
    Well, here's what Google officially says about it at google.com/support/webmasters/bin/answer.py?answer=35235 with the following: Google was unable to crawl the URL due to a robots.txt restriction. This can happen for a number of reasons. For instance, your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error. You may have specifically set up a robots.txt file to prevent us from crawling this URL.
    – Fiasco Labs
    Commented Sep 7, 2011 at 6:22
  • So in order to remove the Google error, either remove the "nofollow" or remove the link.
    – Fiasco Labs
    Commented Sep 7, 2011 at 6:24
1

It's hard to say with 100% certainty but you probably have nothing wrong with your code. A link with nofollow will not be followed but if another page links to it, even from another website, Google will crawl it unless you block it through another means (robots.txt, x-robots-tag, etc). So Google obviously knows that pages exists and may want to crawl but is correctly not doing so because of your robots.txt and thus is reporting it to you). I wouldn't worry about since you're ultimately getting the results you want. Of course you should check to make sure all of your links are correctly nofollowed.

Improve this answer
1
  • Yes, I'm sure all the links for sharing buttons have "nofollow" attribute (the codes for sharing button has been extracted to one single file and all pages will load that file to show the buttons.). Also I think the crawling is initiated from my website instead of external pages, because google generates tens of such errors almost every day (I have 6k pages having the sharing buttons). Isn't it strange? I'm also afraid that such large number of daily error will cause negative impact to the calculation of the website's quality.
    – LazNiko
    Commented Sep 6, 2011 at 4:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.