I'm modifying a site and it requires that, when a user comes from another site in the group, this site will be branded slightly differently. The best way of identifying the page they came from would the HTTP referrer (where available).

However I don't want search engines to see it as a different site. So does anyone know whether search engine spiders, (mainly google), pass through a HTTP referrer parameter?

  • Explain your rational with this sentence: "However I don't want search engines to see it as a different site." How does this remotely relate to the question?
    – Steve-o
    Sep 10 '11 at 13:15

Ususally they don't (e.g., Google bot doesn't).

From my log files:

crawl-66-249-72-20.googlebot.com - - [10/Sep/2011:14:50:38 +0200] "GET /mailman/admin/mailman HTTP/1.1" 200 2003 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" matteocorti.ch

The referrer entry ("-") is empty


Usually Referrers are not passed along but instead by GET url-encoding containing the search keywords. Headers can be set and deleted, - are simple ASCI text, and are just part of the HTTP specification


   message-header = field-name ":" [ field-value ]
   field-name     = token
   field-value    = *( field-content | LWS )
   field-content  = <the OCTETs making up the field-value
                    and consisting of either *TEXT or combinations
                    of token, separators, and quoted-string>

In PHP you can use the function header to set/overwrite and *header_delete* to remove headers, similar ones are available in separate packages in ruby and php.

As such when you write your own search engine, you can simply pass headers through. No major search engine will do so however, for a number of reasons, starting with legal arguments.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy