2

This very related to this question "X-Robots-Tag: noindex" on an HTTP 301 response, however I don't see a definitive answer.

We have subdomains we use for our affiliate links, the affiliates publish a link to the subdomain which in turn 302 redirects to the main site.

However the subdomains are being indexed (as you usually find with 302 redirected URLs) and I would like to remove them from from the index.

I thought about removing them in GWT using URL removal, but I think they may become indexed again, so I was wondering about the possibility of using an X-Robots-Tag HTTP header in conjunction with the 302 redirect.

Can you use both together, or will only 1 be seen? If both are seen, will the noindex work; or would you just go the route of removing in GWT?

Improve this question
2
  • Why not use a 301? There's no reason why you can't send both headers (it's just another header in the response). If Google doesn't "see it" it's because it's deliberately ignoring it.
    – MrWhite
    Feb 5, 2015 at 10:31
  • We don't want to use 301's or direct links as we don't want the links form our affiliates to pass any weight to our main site.
    – Max
    Feb 5, 2015 at 11:25

2 Answers 2

Reset to default
6

As of December 2022, Google will respect a X-Robots-Tag HTTP Response header that comes as part of a HTTP 302 response - Google Search Console will report: "Indexing allowed? No: 'noindex' detected in 'X-Robots-Tag' http header".

I've verified this in my role as a developer on theguardian.com, ie, having added a X-Robots-Tag HTTP Response header to the redirecting url https://www.theguardian.com/business/series/buying-black/latest :

% curl --silent -I https://www.theguardian.com/business/series/buying-black/latest
HTTP/2 302
location: /business/2022/dec/14/buying-black-history-could-help-everyone
...
x-robots-tag: noindex
...

...then even attempting to explicitly index the page in Google Search Console fails, which is exactly what we want:

Google Search Console says: "Page cannot be indexed: Excluded by ‘noindex’ tag"

So, thankfully it's no longer necessary to avoid 302 redirects (eg using an awful Javascript redirect) just to get Google to not index a redirecting url.

Improve this answer
1
  • Very interesting, thanks for the update.
    – Max
    Dec 19, 2022 at 3:06
3

Google considers the status code before looking at everything else, so the X-Robots-Tag will be ignored. We did some test a year ago with a client and this was not working.

The strategy that we used was to remove manually the URLs in the Search Console and add the folder to the robots.txt. An alternative is to use javascript redirects, but this is not very user friendly.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.