3

I was reading Google Webmaster Guidelines and came across a phrase I do not understand.

Make reasonable efforts to ensure that advertisements do not affect search engine rankings. For example, Google's AdSense ads and DoubleClick links are blocked from being crawled by a robots.txt file.

I tried searching Google and here for more explanation, but no luck.

Are they saying that we should be blocking something or should not be blocking something? And blocking what exactly?

I know they want sites to use rel="nofollow" on paid links. But those would be external links that are not controlled by my robots.txt file.

Can someone explain this to me, with examples if possible?

Improve this question

2 Answers 2

Reset to default
2

My interpretation of this is that they want you to block any pages which are nothing but advertisements. For example, you may have an ad server on an "ads.example.com" subdomain. You should place a robots.txt file to stop ads on this server being crawled.

The second sentence is slightly confusing, but my interpretation reads "For example, we [Google] block Google AdSense ads with robots.txt". In other words, "For example, look at how we do it in http://gan.doubleclick.net/robots.txt".

Improve this answer
1

Google does not want paid links to be counted by the search engine as "recommendation" links that change the Pagerank and reputation of the page that is linked to. When Googlebot can crawl sponsored links, Google calls it "link selling". Google is willing to apply penalties for link selling that include:

  • Lowering the reported Pagerank for the site in the Google Toolbar (to make the site look less valuable to those buying links.)
  • Making the rankings in the search results worse for the site selling links.
  • Removing a site entirely from the Google search results

Here is an article that discusses link selling and its penalties.

When you have sponsored links on your site, Google wants you to take steps to ensure that Googlebot does not crawl those links and view them as your endorsement of the site being linked to. Technical measures that can be employed are:

  • Serving the ads with JavaScript and blocking those scripts with robots.txt.
  • Using rel=nofollow attributes on each link to prevent them from passing Pagerank.
  • Redirecting links through a tracking script which has a URL that is blocked in robots.txt.

As Google notes in their document, you don't have to worry about this with most ad networks such as AdSense and DoubleClick. They handle the required blocking for you. They use a combination on the techniques -- the ads are served via JavaScript then the JavaScript code to serve the ads and the ads themselves are listed in robots.txt.

The only time that you really need to worry about this is if you are selling ads yourself and directly linking to other sites.

Improve this answer
2
  • "Serving the ads with JavaScript and blocking those scripts with robots.txt." You seem to be saying to only block these scripts when you are serving requests from googlebots. This sounds dangerous as it could get misinterpreted as cloaking.
    – Kevin Wheeler
    Commented Nov 8, 2015 at 2:06
  • Using robots.txt to prevent Googlebot from crawling content is the Google approved way of doing things. It is never cloaking. I am not saying that you should block scripts when serving requests from Googlebot. I'm saying that you should use robots.txt to instruct Googlebot never to crawl those scripts in the first place.
    – Stephen Ostermiller ♦
    Commented Nov 8, 2015 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.