35 votes
Accepted

Is there a difference between an empty robots.txt and no robots.txt at all?

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of ...
  • 15.1k
16 votes

Is there a difference between an empty robots.txt and no robots.txt at all?

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler ...
  • 398
9 votes
Accepted

How do I determine if a redirection was due to DNS or the webserver?

Working just within the browser, I couldn't tell if a particular direction was due to DNS or the webserver. To view webserver redirects in browsers, open up Developer Tools (either from the browser's ...
  • 15.1k
8 votes
Accepted

Can having domain reg. + hosting with the same company make for a faster loading website?

It won't be any faster if the two are served by the same company. There are no opportunity for synergies between the domain registration, DNS, and web services. They don't use the same protocols, ...
8 votes

Why do we have to pay for web hosting rather than do it ourselves?

Most people choose to pay for hosting for multiple reasons. Site availability: A personal computer isn't really built for serving a website to however many visitors you would get. Most people turn ...
  • 1,910
6 votes
Accepted

Will it hurt SEO to remove the '.html' extension from static pages such that they get served as 'text/x-generic'?

This isn't so much an SEO issue, as an issue as to whether your site would work at all. If it doesn't work in the browser then it's certainly going to hurt your SEO. Your HTML pages still need to ...
  • 42k
5 votes
Accepted

What are the benefits of utilizing a Web Server vs opening an HTML file directly via your browser

I tend to use a development server rather than file:/// URLs even when developing static sites. There are quite a few things that don't work from file:/// URLs. Just the first two points in my list ...
5 votes

Why do we have to pay for web hosting rather than do it ourselves?

There are several reasons to pay for web hosting. Reliability Your web host should have 99.9% uptime or better. It is hard to achieve that at home. Does your house ever lose power or have an ...
5 votes

Does server timezone affect SEO?

Does server timezone affect SEO? No. I understand that the geolocation of a server impacts search engine rankings. Only indirectly. Major search engines don't care where the files are coming from. If ...
5 votes

Can I ban IP based on page GET then instant POST? (Spambots)

It could be possible to write some custom fail2ban rules to 95% solve this (ie with a short "findtime"). There might be edge cases with CGNAT and scripts but you can likely mitigate this by ...
  • 6,580
4 votes
Accepted

Is index.html required?

I am wondering if [it] is necessary [to have these files?] No, it isn't strictly necessary to have any particular files on a web server, including any "default" index pages (the ones you listed). ...
4 votes

How can I route traffic between two hosting servers?

You can do this with DNS if you use a service such as Route53: DNS GeoProximity Trafic Routing If you wanted to setup your own DNS server to do this kind of thing you could look into technologies such ...
4 votes

How to access a website hosted on a shared server (and IP), when its domain is not working?

Your hosting provider should send you temporary URL for your hosting account and website. If they didn't, check your cPanel or whatever control panel and take out the ip address and paste it here: ...
4 votes
Accepted

apache virtual hosts on LAN

Your attempting to access the site using a IP address when your virtual host file is setup to use a domain name, not an IP address... so it'll throw up the default folder, in this case /var/www/html/. ...
  • 32.7k
4 votes

404 page vs. 404 server answer

The TLDR; answer: Send out a 404 header, don't change the url. The longer answer: There are more options then you suggest, which I'm not going to list here (because it's not really needed). The ...
  • 6,776
4 votes

404 page vs. 404 server answer

the url remains the same, but the page has the status code 404, This is certainly the preferred option, as Martijn suggests. For a "custom" error page on Apache (ignoring the CMS option for a moment),...
  • 42k
4 votes
Accepted

Enabling Multiviews is how you make a web server able to open links with missing extensions?

tl;dr Yes, you can enable MultiViews to serve extensionless URLs. ie. Where the file extension is omitted from a URL that would otherwise map to an existing file. However, be aware of potential ...
  • 42k
4 votes
Accepted

Why would this malfunctioning site pad its error page?

Internet Explorer (and now maybe Edge?) has a heuristic built in that for error pages that are small. For small error pages, (less than 512 bytes) it doesn't use what the server sends, but rather ...
3 votes

What's the point of cross-origin resource sharing if there's a bunch of CORS proxies running?

I'd say the the focus of CORS is making the web safer for final users and most of it implementation is in fact incapsulated in modern browsers. What I mean is that it is not designed to only make ...
3 votes

Still prompted for a password after adding SSH public key to a server

TL;DR On Client side: open configuration file /etc/ssh/ssh_config; here look for PreferredAuthentications; make sure password comes after publickey and not viceversa In my case password was written ...
  • 131
3 votes

Why I cannot see the root folder of my website?

Mate, you are in your web root. wp-admin wp-content wp-includes and rest of the files are the root directory of your WordPress. Level above are rest of the pages on server. From what I see, you ...
  • 3,858
3 votes

Can GoDaddy unlimited bandwidth hosting handle 8000 simultaneous users or should I purchase a dedicated server?

I've used GoDaddy's unlimited (and cheap) monthly plans, and a basic Wordpress site with more than about 3 users clicking around at the same time will give overload warnings (GoDaddy will email you if ...
3 votes

Should my web server listen on its IP address, or the public IP address with port forwarding?

It should listen on the IP address assigned to the PC on the LAN. When you're behind a router at home, the PC has no idea of outside world IP numbers. Remember that you are port forwarding, not IP ...
  • 4,608
3 votes
Accepted

How to check if a visitor is on my website?

If you're looking to only serve content to one person at a time, this would be a server level mod. As an example for Nginx, you'd limit the number of active connections to 1: server { limit_conn ...
  • 3,933
3 votes
Accepted

Installing mPDF PHP library for website without root access

To install mPDF root privileges are not required, FTP access will be sufficient and it should work fine on shared hosting environments. Simply unzip the files and subfolders and then copy them into a ...
3 votes

How can I select server size based on the visitors I am getting on my website?

You need more information than just visitor counts. The only way to get the correct server size for your website is to load test it. Or buy a small server and see if it holds up, upgrade it if it ...
3 votes
Accepted

How to prevent domain mirroring?

The good news is, Google should be able to recognize that you published the content first, meaning their copy is of less value. Be sure that you have some way to verify when your changes are published,...
  • 1,910
3 votes

403 forbidden error when IIS is binding ASP web app on port 8080 to be accessed via a subdirectory

I managed to get this working by creating a new rooms application pool. Then in default websites > rooms > basic settings I changed the application pool to the newly created rooms. I still don't ...
3 votes
Accepted

Are Plesk or cPanel helpful for bringing up new VPS servers?

Plesk and cPanel are for managing shared hosting. You would install one of them if you had several people hosting websites on the same machine or VPS. I would not recommend installing them if only ...

Only top scored, non community-wiki answers of a minimum length are eligible