Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
I'm modifying a site and it requires that, when a user comes from another site in the group, this site will be branded slightly differently. The best way of identifying the page they came from would the HTTP referrer (where available).
However I don't want search engines to see it as a different site. So does anyone know whether search engine spiders, (mainly google), pass through a HTTP referrer parameter?
Ususally they don't (e.g., Google bot doesn't).
From my log files:
crawl-66-249-72-20.googlebot.com - - [10/Sep/2011:14:50:38 +0200] "GET /mailman/admin/mailman HTTP/1.1" 200 2003 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" matteocorti.ch
The referrer entry ("-") is empty
Usually Referrers are not passed along but instead by GET url-encoding containing the search keywords. Headers can be set and deleted, - are simple ASCI text, and are just part of the HTTP specification
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
message-header = field-name ":" [ field-value ]
field-name = token
field-value = *( field-content | LWS )
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>
In PHP you can use the function header to set/overwrite and *header_delete* to remove headers, similar ones are available in separate packages in ruby and php.
As such when you write your own search engine, you can simply pass headers through. No major search engine will do so however, for a number of reasons, starting with legal arguments.
