In the Hypertext Transfer Protocol (HTTP), HTTP header fields contain the operating parameters of an HTTP request or response. With the request or response line (first line of message), they form the message header.

learn more… | top users | synonyms

2
votes
1answer
39 views

How to invalidate Google Search results?

Due to a misconfiguration all HTTPS traffic to a specific server showed the wrong content for a while. Google indexed this content including urls that don't exist in the affected sites. For example: ...
3
votes
3answers
65 views

If i use 301 redirects (to bounce the requests), will the other destination be aware of my Server IP?

(Just for the knowledge) I wonder, let's say one day, when my Site is suddenly having some attacks at the login page (e.g: /login.php). If i don't bother about my Login page anymore, and i want to ...
5
votes
1answer
76 views

Looking for explanation of Apache behavior on manipulating headers

Tried to get an answer at ServerFault - they don't know, all real gurus are sitting here. The background of the issue: Googlebot creates non-existing URLs and tries to crawl them. On some URLs Apache ...
3
votes
1answer
43 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
2
votes
2answers
66 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
2
votes
1answer
23 views

Handling bots that request URLs in paths

In my server log, I found at least one IP address to be requesting a full URL in an awkward place. For example, the header the client sends to my server is this: GET ...
1
vote
0answers
50 views

Is a Server and Date HTTP header really necessary for error or redirecting pages?

On practically every URL I visit using the command-line tool CURL, there is always a server identification header and date header outputted regardless of the status of the request. For example, on my ...
1
vote
2answers
42 views

Is it a good idea to close the connection on a web hacker / spammer?

When I receive a request from a web spammer, such as a GET /wp-login.php, I am thinking of replying with an error (404) and closing the HTTP connection immediately with this header: Connection: close ...
2
votes
0answers
61 views

What strings do %{HTTP_HOST} and other mod_rewrite variables return? [closed]

If I have a htaccess file that has RewriteCond %{HTTP_HOST} in it, what is the HTTP_HOST? This is something I've been struggling to find out. To help me understand, could you tell me, in each example ...
2
votes
1answer
33 views

requesting for downloadable files via HEAD

I bet google is checking web pages via HTTP HEAD and HTTP GET request methods. So I'm thinking I should also allow proper header output for downloadable files from my website via HTTP HEAD request but ...
1
vote
1answer
34 views

Can servers ever return headers not beginning with HTTP?

When I checked URLs for headers, they always start with HTTP. For example, HTTP/1.1 301 Moved Permanently. If I remember, I think in the past, headers started with Status: followed by a code. ...
1
vote
1answer
156 views

How to serve for International Googlebots if we redirect users to subdomains for language matching?

So, we have a site and a few sub-domains with content in different languages and we're using Hreflang and canonical implementations for SEO. We do detect the language that users use in their browsers ...
1
vote
2answers
102 views

HTTP Header broken? Browsers don't open our pdf, but open all other pdfs

My Firefox and Chrome usually open PDF files very nicely, without asking and within the browser window. When clicking on a link to a PDF on our own website: Firefox asks if it should download them, ...
0
votes
1answer
65 views

Eliminating 'cookie' from 'vary:' without breaking website functionality

I read that some middle-man servers won't cache websites that have cookie set in the vary: header. For example, If my website issues a vary:accept-encoding,cookie as opposed to vary:accept-encoding ...
0
votes
1answer
37 views

Is it okay for url to end .html even if stored gzipped?

I have some html files stored on Amazon s3, some are quite large so the plan was to store them gzipped and return that as the encoding type when receive a request for them, is that how it works rather ...
2
votes
3answers
365 views

When do browsers clear cache and how can I make more use of 304 responses?

I have a website with some some content which changes frequently. I would like to benefit from good caching when this doesn't change (so would ideally use a longer Cache-Control header), but would ...
1
vote
1answer
94 views

When using resources from cache Firefox show a “304” response but Chrome says “220 (from cache)”

While working on some optimizations for my site, I noticed some things in Networking Timings for Chrome Developer Tools differ from those in Firefox Developer Tools. Very consistently Chrome loads a ...
1
vote
1answer
82 views

Vary: User-Agent in case of redirect?

Should we use Vary: User-Agent in our headers when the regular url www.mydomain.com is forwarded to m.mydomain.com? Or is it not needed for Google due the redirect?
1
vote
1answer
240 views

Earliest browsers that support if-modified-since or if-none-match headers

Based on my observations, google and probably many others are in love with websites that include the Etag and Last-modified headers along with the if-modified-since and if-none-match header checking ...
2
votes
2answers
60 views

Caching a changing site

I run a website where the static content changes at least 4 times a week and I'm trying to figure out the best header arrangement to use. Currently I have all caching related headers in use. I have ...
2
votes
1answer
128 views

Trying to add cache header to 301 pages made by ModRewrite

I have several URLs on my website that get redirected to other URLs using Mod-Rewrite. For example: RewriteRule ^section$ /newsection [NC,L,R=301] The above rule redirects ...
8
votes
1answer
489 views

If I only serve gzipped version of my content, should I add the vary accept encoding header?

I just moved a static site from a VPS to Amazon S3. I have decided to serve only gzipped version of my pages, since S3 is not a webserver I can't have logic based on headers. I also use Cloudfront as ...
0
votes
0answers
178 views

X-Robots-Tag: noindex on an HTTP 302 response to remove subdomain

This very related to this question "X-Robots-Tag: noindex" on an HTTP 301 response, however I don't see a definitive answer. We have subdomains we use for our affiliate links, the ...
0
votes
2answers
56 views

What is the proper response header when an address is not available but its parent is available and is displayed?

My website addresses has a hierarchy structure. When one page is requested, and it is not a valid address, it tries to find its parent, and if the later exists, it will show it. Now, my question is ...
7
votes
3answers
548 views

How should an expired content be handled?

Let's imagine this simplified case: An auction website has "auction detail" pages. A few weeks after the auction is over, the "auction detail" page is no more available. We are simply serving a ...
2
votes
1answer
392 views

Trailing slash on Root folder - Is it possible to remove it without 301?

I have noticed something very strange on the site I'm working on. We currently only show http://www.example.com. The URL http://www.example.com/ (slash at the end) gets 301'd to ...
10
votes
1answer
38k views

Font blocked from loading by Cross-Origin Resource Sharing policy: No 'Access-Control-Allow-Origin'

we are experiencing this error in Google Chrome. We thought everything was set-up correct. But maybe not? Font from origin http://skin.cdn.com has been blocked from loading by Cross-Origin ...
3
votes
2answers
1k views

CloudFlare Not Sending If-Modified-Since Headers

I'm trying to reduce my server load by providing 304 Not Modified responses for content where appropriate. CloudFlare is my middle man, so they should be sending If-Modified-Since headers whenever a ...
1
vote
0answers
437 views

Apache 404 Issue after upgrading from Mavericks to Yosemites

Good afternoon fellow StachExachangers, Background story: I have recently upgraded my macbook pro from Mavericks to Yosemites, everything went well except my local development environment got ...
1
vote
1answer
2k views

Caching of JS and CSS files – Apache mod_headers doesn’t take effect

As suggested by an answer on SO, I'm trying using a version number on my JS and CSS files so that the browser will only make a request when the JS or CSS version number is changed in my PHP file. To ...
0
votes
1answer
38 views

Does referrer header checking offer any real world security improvement?

At work we use a central portal that provides basic SSO functionality to other applications. In addition to verifying the SSO data sent, all of our existing in-house applications (used by the public) ...
1
vote
1answer
135 views

Did I use the proper way to remove pages from Google Webmaster Tools forever?

I have many pages from one year ago that were deleted and are returning a 404 error code, but two week ago after I cleared the content of my robots.txt, the errors in my Google Webmaster Tools reports ...
1
vote
2answers
563 views

Redirect to an affiliate using PHP or JavaScript redirect?

I was asked to introduce a unified self hosted URL-Shortener/Forwarder/Tracking Service for an website that earns money from affiliates. One example I found was one of the bigger german sites for ...
1
vote
1answer
237 views

Use different .htaccess file based on set header from browser

I want to have the ability to test .htaccess rules before saving them to the main .htaccess file. Is it possible to set a specific request header and have Apache use a set of rules from a different ...
0
votes
1answer
48 views

What is best practice for search engines when a website is under maintenance?

I need around a week to transition a heavily data-driven website from one back end to another. During that time I do plan to attempt to keep some pages live, but they won't all work well or look ...
4
votes
1answer
2k views

How to get keywords from Google HTTP referrer?

Can the http referrer URL from Google be decoded so that I can see the specific keywords? ...
1
vote
1answer
42 views

Can serving a different header code for crawlers and users be considered cloaking?

I have urls that point to pages I don't know yet if exist. Whenever reaching those, I perform a not-so-quick external sources check, and if exists generate the page on the fly. In order to avoid this ...
2
votes
1answer
1k views

Can I set Cache-Control in the .htaccess file for a specific part of a document (just a section of the <body>)

I noticed, by using Google Page Speed Insights, that if I structure my HTML to load the critical, above-the-fold content first, my page will improve performance - and it woks (with now current perf. ...
4
votes
1answer
4k views

Is it necessary to specify content-type in http response header?

Except for the html document, currently all the files (css, javascript and images) served from my web server do not specify the content-type in the http response header. I happen to notice this as I ...
4
votes
2answers
458 views

Why doesn't this web crawler obey requests to stop crawling my pages?

I have a hidden page that verifies internal payments from PayPal. The page has the following in its HTTP headers: X-Robots-Tag: noindex, nofollow But once in a while I see this in the page logs: ...
7
votes
1answer
350 views

Why is “29030400” seconds used as a common value for cache expiration?

I noticed that 29030400 is used very often in the Expires directives for static files. Google recommends to cache this kind of files up to 1 year (1 month at least). I did the math: 29030400 seconds ...
3
votes
1answer
409 views

Should we allow HEAD requests?

Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from ...
2
votes
1answer
369 views

Can a past Expires header value cause a site to not be indexed by Google?

I'm using this: <filesMatch "\.(html|htm|js|css)$"> FileETag None <ifModule mod_headers.c> Header unset ETag Header set Cache-Control "max-age=0, no-cache, no-store, ...
1
vote
1answer
312 views

how do we organize our site's (navigation) pages in Google search? [duplicate]

Is there a HTML technique for organizing my site's main pages in Google (navigation)? Or is it made by a Google's specific tool for creating a result layout like in this picture? Look at the ...
2
votes
1answer
6k views

IP Address headers (Remote_Addr) - When is this header sent by the browser?

I want to know when a browser includes the header - Remote_Addr in the HTTP request. I observed the request sent by the browser on loading stackoverflow.com, in the web console window of Firefox. ...
1
vote
1answer
651 views

Nginx Browser Caching using HTTP Headers outside server/location block

I am having difficulty setting the HTTP expires headers for Nginx outside of specific server (and then location) blocks. What I want is to something like the following: location ~* ...
1
vote
2answers
2k views

Is no cache meta tag bad for performance?

I see many websites, seemingly arbitrarily adding the no cache meta tag on every page. Like this: <META HTTP-EQUIV="PRAGMA" CONTENT="NO-CACHE"> <META HTTP-EQUIV="CACHE-CONTROL" ...
5
votes
1answer
875 views

How do Expires headers and cache manifest rules work together?

I find the W3C's official Offline Web Applications specification to be rather vague about how the cache manifest interacts with headers such as ETag, Expires, or Pragma on cached assets. I know that ...
4
votes
1answer
263 views

HTTP Headers caching

I am not totally sure bout HTTP headers, but from what I read its good to have some level of caching on static pages also I am not sure if Transfer Encoding: chunked is a good thing. I was not finding ...
2
votes
1answer
160 views

What's the best HTTP code for dynamically deleted pages?

I've built my own CMS and I'm looking for the best way to handle pages / items deletion. Actually, pages and items were deleted and cannot be restored. But since I have implemented a bin, I don't ...