Tag Info

Hot answers tagged

52

The short answer is "Because HTTP wasn't designed for it". Tim Berners-Lee did not design an efficient and extensible network protocol. His one design goal was simplicity. (The professor of my networking class in college said that he should have left the job to the professionals.) The problem that you outline is just one of the many problems with the ...


20

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty ...


11

Because they do not know what those resources are. The assets a web page requires are coded into the HTML. Only after a parser determines what those assets are can the y be requested by the user-agent. Additionally, once those assets are known, they need to be served individually so the proper headers (i.e. content-type) can be served so the user-agent ...


10

The web server software looks at the hostname in the HTTP request and uses that to determine which website to serve. For example, Apache has the NameVirtualHost configuration option which controls this behaviour. You can find a detailed explanation of how this process works in its documentation: https://httpd.apache.org/docs/2.2/vhosts/name-based.html


10

Your web browser doesn't know about the additional resources until it downloads the web page (HTML) from the server, which contains the links to those resources. You might be wondering, why doesn't the server just parse its own HTML and send all the additional resources to the web browser during the initial request for the web page? It's because the ...


8

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


7

Because, in your example, web server would always send CSS and images regardless if the client already has them, thus greatly wasting bandwidth (and thus making the connection slower, instead of faster by reducing latency, which was presumably your intention). Note that CSS, javascript and image files are usually sent with very long expire times for exactly ...


7

Working just within the browser, I couldn't tell if a particular direction was due to DNS or the webserver. To view webserver redirects in browsers, open up Developer Tools (either from the browser's menu, or by pressing F12), and click on the Network tab. It's a good idea to look for a Disable Cache option and check this so that all requests are done ...


7

It won't be any faster if the two are served by the same company. There are no opportunity for synergies between the domain registration, DNS, and web services. They don't use the same protocols, they don't use the same ports, they don't use the same types of servers. There isn't going to be an opportunity for any type of caching or reuse between the ...


6

I would have said that placing these sensitive files above the document root would be preferable. And perhaps easier to manage if they are all contained in a particular directory, however... Using .htaccess to prevent access to all .db and .exe files and return a 403 - Forbidden. <Files ~ "\.(db|exe)$"> Deny from all </Files> Unless you have ...


6

It's tough to get some good stats for this but if you look at this article "When does it cause network problems?" it would have been an advantage if there was a proper AAAA response. On this list of customer problems that could occurr however there is one router listed that indeed has problems when both A and AAAA are set up. But what's the impact? You'll ...


6

HTTP2 is based on SPDY and does exactly what you suggest: At a high level, HTTP/2: is binary, instead of textual is fully multiplexed, instead of ordered and blocking can therefore use one connection for parallelism uses header compression to reduce overhead allows servers to “push” responses proactively into client caches More is ...


5

In order for IIS to allow access to the file at all, it needs to be assigned a MIME-type. Use application/octet-stream and the browser will almost certainly treat it as a file it can't handle itself. (You could also experiment with application/x-whatever-you-want)


5

It sounds like your domain records have not propagated yet. This means that the domain name servers around the world that store a record of your domain information haven't updated with the latest details, so they are still 'sending' visitors to your old server. The solution is to wait. It can take between 0 and 48 hours for domain name changes to update ...


5

Digicert maintains pages for compatibility with certificate types: SAN certificate compatibility Wildcard certificate compatibility They note several server side compatibility problems with wildcard certificates but no client side problems. SAN certificates are problemic for some older browsers: Versions of major web browsers from before 2003. Older ...


5

Web client, usually a browser, opens a TCP socket to the server. The server software accepts the connection without knowing the specific site requested, and waits for an HTTP request to happen. The client sends the request, mainly composed of HTTP headers. One of these headers is the Host: example.com header, at this point the server is aware of the right ...


4

Without knowing a bit more about your context it's a bit hard to provide a better solution, however rather than writing a custom web browser that allows users to browse your site, and download things why not do something along the lines of: In your windows application, call a service on your website that gives you the relevant details of downloads ...


4

A .htaccess file works only on the Apache webserver, you're using Microsoft's IIS Webserver. See 'Add Expires or Cache Control Header to static content in IIS'


4

I've just started getting into hosting and am using WHMCS. You can find more info at http://www.whmcs.com/ You can put clients into it, separate by servers, keep track of products/services for each client, custom notes for the client or if they have multiple domains you can write notes for each domain on a client. It can automate a lot of things but If ...


4

Some search engines and bots send HEAD request to pages before sending the GET request for reasons like: Checking if the page size has changed Checking the last modified date etc. (Any other info the head would give them!) This would help large crawlers save a lot of bandwidth if they know a page has not been changed meanwhile and they don't have to ...


3

If your environnement is identical, i mean with the same physical path, ip etc.. you can just applicationHost.config. The file is by default located in the C:\Windows\System32\inetsrv\config it's an xml file, si if you rigth a little tool you could change your binding before restore the file in your dev environnement


3

Yep. Just add more types: AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript application/x-font-ttf


3

To answer my own question, the problem was with how Internet Explorer handles compatibility mode on intranet sites. Even though my web.config and my HTML code specified to always set IE=Edge (most recent rendering engine), the Internet Explorer browser itself has ultimate control over the rendering, and its default is to have intranet sites rendered in ...


3

Also you have to consider that you will also eat up 30GB x 4 of the websites data traffic per month, depending on the website this can be a huge problem for the operator and they will probably detect the bandwidth usage spike as an attack on their website.


3

Your best bet is to rent a Virtual Server for ~£15 a month for approximately six months. The reason being that there is only so much you can do with an offline host, e.g. something on your LAN - it is just not 'real-world' enough. The best operating system to learn with is Ubuntu simply because it has the best community support of all of the distros. With ...


3

I suspect "Password protect folder" puts a password on for people using web browsers. I think the best way to achieve this is to have separate ftp logons for the directories for a.com, b.com and c.com. Then only you, with access to the root, have access to the files for d.com. You could make the passwords the same so that people only have to remember one ...


3

No ISP DNS is required. Just setup your local DNS to use 8.8.8.8 and 8.8.4.4. If necessary you can set your router to distribute those addresses and bypass your local ISP altogether.


3

You could try using a program like HTTrack to perform a site mirror. This gives you a local browsable copy that you can archive.


3

You need to make a separate VirtualHost for each port, like so: #assuming you have this in your config NameVirtualHost *:8000 NameVirtualHost *:8001 NameVirtualHost *:8002 NameVirtualHost *:8003 # (...) Listen 8000 Listen 8001 Listen 8002 Listen 8003 # (...) Then each VirtualHost looks like this: <VirtualHost *:8000> ServerName localhost ...


3

You may not be hosting malware now but you have relatively recently: http://safebrowsing.clients.google.com/safebrowsing/diagnostic?site=goodtherapy.org The script could be randomly injected or perhaps your ISP is cleaning things for you in the background. In any event, your site has been and probably still is compromised and you should be doing a ...



Only top voted, non community-wiki answers of a minimum length are eligible