Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm modifying a site and it requires that, when a user comes from another site in the group, this site will be branded slightly differently. The best way of identifying the page they came from would the HTTP referrer (where available).

However I don't want search engines to see it as a different site. So does anyone know whether search engine spiders, (mainly google), pass through a HTTP referrer parameter?

share|improve this question

migrated from stackoverflow.com Sep 10 '11 at 14:00

This question came from our site for professional and enthusiast programmers.

Explain your rational with this sentence: "However I don't want search engines to see it as a different site." How does this remotely relate to the question? – Steve-o Sep 10 '11 at 13:15

Ususally they don't (e.g., Google bot doesn't).

From my log files:

crawl-66-249-72-20.googlebot.com - - [10/Sep/2011:14:50:38 +0200] "GET /mailman/admin/mailman HTTP/1.1" 200 2003 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" matteocorti.ch

The referrer entry ("-") is empty

share|improve this answer

Usually Referrers are not passed along but instead by GET url-encoding containing the search keywords. Headers can be set and deleted, - are simple ASCI text, and are just part of the HTTP specification


   message-header = field-name ":" [ field-value ]
   field-name     = token
   field-value    = *( field-content | LWS )
   field-content  = <the OCTETs making up the field-value
                    and consisting of either *TEXT or combinations
                    of token, separators, and quoted-string>

In PHP you can use the function header to set/overwrite and *header_delete* to remove headers, similar ones are available in separate packages in ruby and php.

As such when you write your own search engine, you can simply pass headers through. No major search engine will do so however, for a number of reasons, starting with legal arguments.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.