64

The short answer is "Because HTTP wasn't designed for it". Tim Berners-Lee did not design an efficient and extensible network protocol. His one design goal was simplicity. (The professor of my networking class in college said that he should have left the job to the professionals.) The problem that you outline is just one of the many problems with the ...


31

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty robots....


14

Your web browser doesn't know about the additional resources until it downloads the web page (HTML) from the server, which contains the links to those resources. You might be wondering, why doesn't the server just parse its own HTML and send all the additional resources to the web browser during the initial request for the web page? It's because the ...


14

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


13

The web server software looks at the hostname in the HTTP request and uses that to determine which website to serve. For example, Apache has the NameVirtualHost configuration option which controls this behaviour. You can find a detailed explanation of how this process works in its documentation: https://httpd.apache.org/docs/2.2/vhosts/name-based.html


11

Because they do not know what those resources are. The assets a web page requires are coded into the HTML. Only after a parser determines what those assets are can the y be requested by the user-agent. Additionally, once those assets are known, they need to be served individually so the proper headers (i.e. content-type) can be served so the user-agent ...


8

Because, in your example, web server would always send CSS and images regardless if the client already has them, thus greatly wasting bandwidth (and thus making the connection slower, instead of faster by reducing latency, which was presumably your intention). Note that CSS, JavaScript and image files are usually sent with very long expire times for exactly ...


8

Working just within the browser, I couldn't tell if a particular direction was due to DNS or the webserver. To view webserver redirects in browsers, open up Developer Tools (either from the browser's menu, or by pressing F12), and click on the Network tab. It's a good idea to look for a Disable Cache option and check this so that all requests are done from ...


8

It won't be any faster if the two are served by the same company. There are no opportunity for synergies between the domain registration, DNS, and web services. They don't use the same protocols, they don't use the same ports, they don't use the same types of servers. There isn't going to be an opportunity for any type of caching or reuse between the two....


8

Most people choose to pay for hosting for multiple reasons. Site availability: A personal computer isn't really built for serving a website to however many visitors you would get. Most people turn their PCs off from time to time, and most people have limited bandwidth for their internet connection. So, if your site suddenly got popular, most people's ...


7

HTTP2 is based on SPDY and does exactly what you suggest: At a high level, HTTP/2: is binary, instead of textual is fully multiplexed, instead of ordered and blocking can therefore use one connection for parallelism uses header compression to reduce overhead allows servers to “push” responses proactively into client caches More is ...


6

It's tough to get some good stats for this but if you look at this article "When does it cause network problems?" it would have been an advantage if there was a proper AAAA response. On this list of customer problems that could occurr however there is one router listed that indeed has problems when both A and AAAA are set up. But what's the impact? You'll ...


6

I would have said that placing these sensitive files above the document root would be preferable. And perhaps easier to manage if they are all contained in a particular directory, however... Using .htaccess to prevent access to all .db and .exe files and return a 403 - Forbidden. <Files ~ "\.(db|exe)$"> Deny from all </Files> Unless you have a ...


6

Web traffic Check the web logs to see which pages are being requested. If they're hitting a single page with a bad plug-in then it should show you the Page/Request URI that the user hit. Some things like scripts that resize image uploads can eat resources quickly. If you were getting a lot of traffic for the webserver then it would likely be someone ...


6

Digicert maintains pages for compatibility with certificate types: SAN certificate compatibility Wildcard certificate compatibility They note several server side compatibility problems with wildcard certificates but no client side problems. SAN certificates are problemic for some older browsers: Versions of major web browsers from before 2003. Older ...


6

Web client, usually a browser, opens a TCP socket to the server. The server software accepts the connection without knowing the specific site requested, and waits for an HTTP request to happen. The client sends the request, mainly composed of HTTP headers. One of these headers is the Host: example.com header, at this point the server is aware of the right ...


6

This isn't so much an SEO issue, as an issue as to whether your site would work at all. If it doesn't work in the browser then it's certainly going to hurt your SEO. Your HTML pages still need to return the text/html mime-type in order to be interpreted as HTML by the browser. ie. You need to send a Content-Type: text/html HTTP response header somehow. ...


5

It sounds like your domain records have not propagated yet. This means that the domain name servers around the world that store a record of your domain information haven't updated with the latest details, so they are still 'sending' visitors to your old server. The solution is to wait. It can take between 0 and 48 hours for domain name changes to update ...


5

Some search engines and bots send HEAD request to pages before sending the GET request for reasons like: Checking if the page size has changed Checking the last modified date etc. (Any other info the head would give them!) This would help large crawlers save a lot of bandwidth if they know a page has not been changed meanwhile and they don't have to crawl ...


5

There are several reasons to pay for web hosting. Reliability Your web host should have 99.9% uptime or better. It is hard to achieve that at home. Does your house ever lose power or have an internet outage? Do you have a backup generator and an uninterruptible power supply? Do you maintain multiple internet connections in case one goes out? Most web ...


4

I've just started getting into hosting and am using WHMCS. You can find more info at http://www.whmcs.com/ You can put clients into it, separate by servers, keep track of products/services for each client, custom notes for the client or if they have multiple domains you can write notes for each domain on a client. It can automate a lot of things but If ...


4

A .htaccess file works only on the Apache webserver, you're using Microsoft's IIS Webserver. See 'Add Expires or Cache Control Header to static content in IIS'


4

Without knowing a bit more about your context it's a bit hard to provide a better solution, however rather than writing a custom web browser that allows users to browse your site, and download things why not do something along the lines of: In your windows application, call a service on your website that gives you the relevant details of downloads available....


4

I'm assuming that the bind.php file is not actually a part of Joomla, but rather a malicious script that the hacker uploaded to your site. In particular, just looking at the request parameters, it appears likely that the script is being used to send e-mail spam, possibly using someone else's hijacked e-mail account. Here's what the request parameters in ...


4

I am wondering if [it] is necessary [to have these files?] No, it isn't strictly necessary to have any particular files on a web server, including any "default" index pages (the ones you listed). [W]hat happens if there is no such file name[?] The server will generally do one of five things: dump a directory list of the files that are available (...


4

Your attempting to access the site using a IP address when your virtual host file is setup to use a domain name, not an IP address... so it'll throw up the default folder, in this case /var/www/html/. Fix 1. Local Virtual host file The most common and easy method to fix the issue would be to edit your host file within Windows or Mac, this will allow you to ...


4

The TLDR; answer: Send out a 404 header, don't change the url. The longer answer: There are more options then you suggest, which I'm not going to list here (because it's not really needed). The most basic solution is just to serve a 404 header when the page does not exist. The reason for this is simple; This is what visitors expect. You open a URL and ...


4

the url remains the same, but the page has the status code 404, This is certainly the preferred option, as Martijn suggests. For a "custom" error page on Apache (ignoring the CMS option for a moment), this could be achieved with the ErrorDocument directive: ErrorDocument 404 /errors/my404.php it can be a default server 404 page Using the default ...


4

tl;dr Yes, you can enable MultiViews to serve extensionless URLs. ie. Where the file extension is omitted from a URL that would otherwise map to an existing file. However, be aware of potential conflicts with mod_rewrite. the server won't even open links to the site that are missing a file extension You make it sound that this should be normal (or ...


Only top voted, non community-wiki answers of a minimum length are eligible