2

I have a problem which I have not encountered before. On a website I am working on, www.example.com, I have several affiliate links which are of the form www.example.com/out/blahblah

The redirect is a 302, as I have heard that 302 is the recommended redirect type for affiliate links.

The problem is that if i do a site:example.com inurl:out search, I can see the redirected urls show in serp, with the title and meta description of the targeted url.

I did not notice until today. And, as a curiosity I checked other big sites that i know have affiliate links and I noticed the same thing happens to some of them too. Some however do not have these links indexed and others have them indexed with the "this page is blocked by robots.txt" message because they probably have that.

example: see how yoast.com has /out/ blocked in robots , and it shows that message : https://www.google.com/search?q=site%3Ayoast.com+inurl%3Aout&rlz=1C1GIGM_enRO664RO664&oq=site%3Ayoast.com+inurl%3Aout&aqs=chrome.0.69i59j69i58.6783j0j4&sourceid=chrome&es_sm=93&ie=UTF-8

same for nuts and botls media : https://www.google.com/search?q=site:nutsandboltsmedia.com+inurl:go&rlz=1C1GIGM_enRO664RO664&biw=1920&bih=995&noj=1&filter=0

and check out carriedils.com which has exactly the problem I have : https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=site%3Acarriedils.com%20inurl%3Ago

What is the cause of this? Any recommendations?

1

It has been long been the case that Google indexes some 302 redirect URLs.

Years ago you this was used by black hats for "hijacking" listings. They could create a link that 302 redirected to another page and Google would sometimes put their redirect URL into the SERPs in place of the legitimate page. The black hat would then change the redirect (or even cloak it) and steal organic traffic coming from the Google search results.

Google changed their algorithms to favor the actual page 99% of the time instead of the redirect. Now 302 hijacking no longer works. Matt Cutts discussed this on his blog. He gives an example of why Google might want to occasionally index the redirect rather than the destination page:

Let’s take an example from the tiny fraction of the time that we may reserve the right to show the source page for a 302 off-domain redirect. If you run wget on www.sfgiants.com, you’ll get a 302 redirect to a different domain, and the url that you’ll get is pretty ugly: http://sanfrancisco.giants.mlb.com/NASApp/mlb/index.jsp?c_id=sf. Please set aside that you are probably a site owner or webmaster for a second, and try to step into the shoes of a regular user on the street. If we had a taste test, how many users would prefer to click on sfgiants.com and how many would prefer to click on sanfrancisco.giants.mlb.com/NASApp/mlb/index.jsp?c_id=sf? Normal users usually like short, clean urls. They are less likely to say “mlb.com? I wonder what that stands for? Hmm. Maybe major league baseball? Is that the officially licensed name, I wonder? It probably is. Yes, it looks like sanfrancisco.giants.mlb.com/NASApp/mlb/index.jsp?c_id=sf is the correct url from my query.”

So if you have a URL that redirects, Google may choose to index it. They may choose to do so especially if the destination URL is not crawlable. For this to happen, Google has to be able to crawl the URL.


For redirect URLs listed in robots.txt it is another case completely. Google may choose to index a URL that is listed in robots.txt even when it can't crawl it if it feels that the inbound links to it are important enough. In such cases it uses the anchor text of links in the SERPs and does not crawl the URL at all. Google has no knowledge that the URL redirects. This is the case for the three real world examples that you give. If I search for the destination pages by name, I can find the destination pages on the normal site.

Most of the time these URLs blocked in robots.txt will only appear for site: searches. (I'm pretty sure I've seen Google's John Mueller say so, but I can't find the reference now.)


So if you don't want your affiliate links to show up in the search results, it appears than neither 302 redirects nor robots.txt is a foolproof solution. I would recommend that every affiliate link on your site has the rel=nofollow attribute on it. Then Google will be much less likely give the URLs enough link juice that it might think they are worth indexing.

  • Thank you for this excellent answer mate, much appreciated. I do have a follow up question perhaps you can help me with it. Will Google consider these urls as part of my website and penalize me for eventual duplicate content or thin content? I am referring to the ones that show in serp with my url but with the targeted page title and content. – ClawDuda Dec 9 '15 at 8:21
  • I marked this question as answered , obviously. However, I would appreciate your answer to the follow up question too. thank you in advanced. – ClawDuda Dec 9 '15 at 8:22
  • 1
    The URLs are part of your website. They shouldn't be considered duplicate content. When they are blocked by robots.txt, Google can't crawl them to see the contents. When they are redirects, Google understands that they are not duplicate. I've worked with sites that use robots.txt to block tons of similar links. Those sites were not hit by Google's thin content algorithm (Panda). – Stephen Ostermiller Dec 9 '15 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.