7

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek at it! Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if ...


5

Official docs says You can’t configure CloudFront to add any of the following headers to requests that it sends to your origin: Cache-Control Connection Content-Length Cookie Host If-Match If-Modified-Since If-None-Match If-Range If-Unmodified-Since Max-Forwards Pragma Proxy-Authorization Proxy-Connection Range Request-Range TE Trailer ...


3

Apache Tomcat has gzip filter and it starts to zip from 2kb, my quick test tells that it's the lowest boundary, and you can increase it at least to 3-4kb. Because for 2kb you will get a similar size at the output + time penalty for zipping.


2

As far as I know Vary header is used for displaying website content via CSS or HTML appropriately for the User Agent who is accessing the site, right? So if your SPA also have different serve between mobile and desktop, you need Vary header for all responses. BTW, SSR not better in SEO, SSR and SPA now is same rank(just depend on the page content/meta), ...


1

Content Security Policy (CSP) Quick Reference Guide CSP-Reporting on Google Developers If you're just starting out with CSP, it makes sense to evaluate the current state of your application before rolling out a draconian policy to your users. As a stepping stone to a complete deployment, you can ask the browser to monitor a policy, reporting violations but ...


1

From How can I add header conditionally in nginx configuration? on StackOverflow this should be: map $legituser $mimetypeheadervalue { 0 'text/html'; default $arg_mimetype; } location / { more_set_headers "Content-Type: $mimetypeheadervalue"; if ($legituser = 0) { return 200 "you are not allowed to access this url, ...


1

Mozilla has a HTTP Caching FAQ that says: [In this case] then the expiration calculation is essentially the algorithm described in RFC 2616 section 13.2. ... The expiration time is computed as follows: expirationTime = responseTime + freshnessLifetime - currentAge where responseTime is the time at which the response was received according to the browser. .....


1

After posting the question a day back, I was researching Google official docs in parallel. Fortunately, I found the below phrases in https://developers.google.com/search/mobile-sites/mobile-seo/dynamic-serving we recommend that the server send a hint to request that Googlebot for smartphones also crawl the page, and thus discover the mobile content. This ...


1

Cloaking is only a problem if you treat search engine crawler user agents differently than normal users. Removing ads for a few normal users that have old browsers is not cloaking. You would still be treating Googlebot the same as the vast majority of human users. Search engines might use the vary on the User-Agent header if you were serving different ...


1

With thanks to @Stephen Ostermiller and @davidgo, .htaccess and $_SERVER variables are not for outgoing queries. If there is a question of what exactly is being sent in your outgoing queries, change your endpoint to https://httpbin.org/anything, which will return all of the headers and contents of your query.


1

It looks like you're getting the issue "Often Misused: HTTP Method Override" reported by Fortify's WebInspect scanner. To resolve this for my team I implemented a filter in the WebApp itself that listens for our bad headers (x-http-method, x-http-method-override, x-method-override), sets status to 405, and breaks if they are found. See code below. ...


1

If you are using the user agent to determine if www.example.com serves content or is redirected to m.example.com, then you should use Vary: User-Agent. You don't need to use the Vary: User-Agent heading if you have separate mobile and desktop sites without any redirects between them. You could also implement the redirects client side in JavaScript. Then ...


Only top voted, non community-wiki answers of a minimum length are eligible