We have affiliates that link to our site with a querystring. e.g.

<a href="http://www.mysite.com/affiliate?id=123">Affiliate Link</a>

In my robots.txt file, i have:

Disallow: /affiliate/*?*

What would a search engine do with the link? Will it hit our site, then stop because it's disallowed? I'm guessing it'll index it, but will it index with the query string?

Ultimately, we want the link to our site to be followed, but we don't want the query string'd link to be ranked for obvious reasons. How does this work and how can this be achieved?

If the original link had a rel="nofollow", will google completely ignore that link?


An affiliate ID used for tracking purposes can be considered similar to a session ID, in that different affiliate links can lead to the same page and content, thus resulting in duplicate content.

Therefore to disallow affiliate links from being crawled (resulting in duplicate content), from the "Pattern matching" section in Google Webmaster Tools - Block or remove pages using a robots.txt file:

The Disallow: / *? directive will block any URL that includes a ? (more specifically, it will block any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string).

Even more specific to URL's with ?id= in them, you could simply have:

Disallow: /*?id=

If the affiliate links redirect to custom affiliate pages with similar content, you should specify a canonical URL in these pages to point to the preferred version of the page you want indexed. For more on this, see: Google Webmaster Tools - About rel="canonical"

The "nowfollow" attribute in rel="nofollow" tells search engines not to follow the links on a page, or don't follow this specific link, which isn't necessary if you're blocking them in your robots.txt. As Google indicates, this gives webmasters more "granular control" to tell robots not to crawl specific links or pages.


If the original link had a rel="nofollow", will google completely ignore that link?

No, it only means, that you do not trust this link and Google will not pass your site's authority to the recipient.

Disallow: /affiliate/*?*

This directive says Google not to index all the URLs with /affiliate/, which have some additional parameters with ?. But the directory /affiliate/ will be accessible for Google.

To resolve all your question about robots.txt file use Google Webmasters Tools' tool. That tool let you check all the URLs you want for availability to crawl.

  • 2
    This is incorrect, there is an extra forward slash so it wont match. it should be defined as: Disallow: /affiliate?id=
    – Max
    Jul 11 '13 at 7:03
  • Yes, you're right. I have lost my attention in the moment with the extra forward slash. Jul 11 '13 at 7:15
  • 1
    Also from Google Webmaster Tools - rel="nofollow": How does Google handle nofollowed links? In general, we don't follow them.
    – dan
    Jul 11 '13 at 8:04
  • This question had been asked on 04.03.2009. Since that time Google says many times, that their googlebot follow "nofollowed" links, but do not pass PR and authority. Nofollow only means that you do not trust the site you are linked to, no more. Jul 11 '13 at 8:47
  • 1
    If you read on in that link, Google goes on to state: However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap. So "In general it doesn't follow them..." would be a lot more accurate, and documented, than "No".
    – dan
    Jul 11 '13 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.