2022 Developer Survey is open! Take survey.
67 votes
Accepted

When webservers send a page, why don't they send all required CSS, JS, and images without being asked?

The short answer is "Because HTTP wasn't designed for it". Tim Berners-Lee did not design an efficient and extensible network protocol. His one design goal was simplicity. (The professor of my ...
user avatar
35 votes
Accepted

Is there a difference between an empty robots.txt and no robots.txt at all?

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of ...
user avatar
  • 15k
16 votes

Is there a difference between an empty robots.txt and no robots.txt at all?

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler ...
user avatar
  • 398
15 votes

When webservers send a page, why don't they send all required CSS, JS, and images without being asked?

Your web browser doesn't know about the additional resources until it downloads the web page (HTML) from the server, which contains the links to those resources. You might be wondering, why doesn't ...
user avatar
  • 521
13 votes

How do websites share the same IP address?

The web server software looks at the hostname in the HTTP request and uses that to determine which website to serve. For example, Apache has the NameVirtualHost configuration option which controls ...
user avatar
  • 4,334
11 votes

When webservers send a page, why don't they send all required CSS, JS, and images without being asked?

Because they do not know what those resources are. The assets a web page requires are coded into the HTML. Only after a parser determines what those assets are can the y be requested by the user-agent....
user avatar
  • 85.8k
9 votes
Accepted

How do I determine if a redirection was due to DNS or the webserver?

Working just within the browser, I couldn't tell if a particular direction was due to DNS or the webserver. To view webserver redirects in browsers, open up Developer Tools (either from the browser's ...
user avatar
  • 15k
8 votes

When webservers send a page, why don't they send all required CSS, JS, and images without being asked?

Because, in your example, web server would always send CSS and images regardless if the client already has them, thus greatly wasting bandwidth (and thus making the connection slower, instead of ...
user avatar
8 votes
Accepted

Can having domain reg. + hosting with the same company make for a faster loading website?

It won't be any faster if the two are served by the same company. There are no opportunity for synergies between the domain registration, DNS, and web services. They don't use the same protocols, ...
user avatar
8 votes

Why do we have to pay for web hosting rather than do it ourselves?

Most people choose to pay for hosting for multiple reasons. Site availability: A personal computer isn't really built for serving a website to however many visitors you would get. Most people turn ...
user avatar
  • 1,900
7 votes

When webservers send a page, why don't they send all required CSS, JS, and images without being asked?

HTTP2 is based on SPDY and does exactly what you suggest: At a high level, HTTP/2: is binary, instead of textual is fully multiplexed, instead of ordered and blocking can therefore use ...
user avatar
  • 71
6 votes

How do websites share the same IP address?

Web client, usually a browser, opens a TCP socket to the server. The server software accepts the connection without knowing the specific site requested, and waits for an HTTP request to happen. The ...
user avatar
  • 181
6 votes
Accepted

Will it hurt SEO to remove the '.html' extension from static pages such that they get served as 'text/x-generic'?

This isn't so much an SEO issue, as an issue as to whether your site would work at all. If it doesn't work in the browser then it's certainly going to hurt your SEO. Your HTML pages still need to ...
user avatar
  • 41k
5 votes
Accepted

What are the benefits of utilizing a Web Server vs opening an HTML file directly via your browser

I tend to use a development server rather than file:/// URLs even when developing static sites. There are quite a few things that don't work from file:/// URLs. Just the first two points in my list ...
user avatar
5 votes

Why do we have to pay for web hosting rather than do it ourselves?

There are several reasons to pay for web hosting. Reliability Your web host should have 99.9% uptime or better. It is hard to achieve that at home. Does your house ever lose power or have an ...
user avatar
5 votes

Does server timezone affect SEO?

Does server timezone affect SEO? No. I understand that the geolocation of a server impacts search engine rankings. Only indirectly. Major search engines don't care where the files are coming from. If ...
user avatar
5 votes

Can I ban IP based on page GET then instant POST? (Spambots)

It could be possible to write some custom fail2ban rules to 95% solve this (ie with a short "findtime"). There might be edge cases with CGNAT and scripts but you can likely mitigate this by ...
user avatar
  • 5,825
4 votes

How to access a website hosted on a shared server (and IP), when its domain is not working?

Your hosting provider should send you temporary URL for your hosting account and website. If they didn't, check your cPanel or whatever control panel and take out the ip address and paste it here: ...
user avatar
4 votes

Server log what does "ho=<something>" mean?

I'm assuming that the bind.php file is not actually a part of Joomla, but rather a malicious script that the hacker uploaded to your site. In particular, just looking at the request parameters, it ...
user avatar
4 votes
Accepted

Is index.html required?

I am wondering if [it] is necessary [to have these files?] No, it isn't strictly necessary to have any particular files on a web server, including any "default" index pages (the ones you listed). ...
user avatar
4 votes
Accepted

apache virtual hosts on LAN

Your attempting to access the site using a IP address when your virtual host file is setup to use a domain name, not an IP address... so it'll throw up the default folder, in this case /var/www/html/. ...
user avatar
  • 32.6k
4 votes

404 page vs. 404 server answer

The TLDR; answer: Send out a 404 header, don't change the url. The longer answer: There are more options then you suggest, which I'm not going to list here (because it's not really needed). The ...
user avatar
  • 6,736
4 votes

404 page vs. 404 server answer

the url remains the same, but the page has the status code 404, This is certainly the preferred option, as Martijn suggests. For a "custom" error page on Apache (ignoring the CMS option for a moment),...
user avatar
  • 41k
4 votes
Accepted

Enabling Multiviews is how you make a web server able to open links with missing extensions?

tl;dr Yes, you can enable MultiViews to serve extensionless URLs. ie. Where the file extension is omitted from a URL that would otherwise map to an existing file. However, be aware of potential ...
user avatar
  • 41k
3 votes

When webservers send a page, why don't they send all required CSS, JS, and images without being asked?

Because it doesn't assume that these things are actually required. The protocol doesn't define any special handling for any particular type of file or user-agent. It does not know the difference ...
user avatar
3 votes
Accepted

How do I get ngix to stop listening on port 80?

I will say right from the start I do not know Amazon Linux EC2, but I did do a bit of research to try and help. You should be able to stop nginx with: nginx -s stop You should be able to stop it ...
user avatar
  • 32.5k
3 votes

Still prompted for a password after adding SSH public key to a server

TL;DR On Client side: open configuration file /etc/ssh/ssh_config; here look for PreferredAuthentications; make sure password comes after publickey and not viceversa In my case password was written ...
user avatar
  • 31
3 votes

How can I route traffic between two hosting servers?

You can do this with DNS if you use a service such as Route53: DNS GeoProximity Trafic Routing If you wanted to setup your own DNS server to do this kind of thing you could look into technologies such ...
user avatar
3 votes

What's the point of cross-origin resource sharing if there's a bunch of CORS proxies running?

I'd say the the focus of CORS is making the web safer for final users and most of it implementation is in fact incapsulated in modern browsers. What I mean is that it is not designed to only make ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible