66

The short answer is "Because HTTP wasn't designed for it". Tim Berners-Lee did not design an efficient and extensible network protocol. His one design goal was simplicity. (The professor of my networking class in college said that he should have left the job to the professionals.) The problem that you outline is just one of the many problems with the ...


33

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty robots....


15

Your web browser doesn't know about the additional resources until it downloads the web page (HTML) from the server, which contains the links to those resources. You might be wondering, why doesn't the server just parse its own HTML and send all the additional resources to the web browser during the initial request for the web page? It's because the ...


14

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


13

The web server software looks at the hostname in the HTTP request and uses that to determine which website to serve. For example, Apache has the NameVirtualHost configuration option which controls this behaviour. You can find a detailed explanation of how this process works in its documentation: https://httpd.apache.org/docs/2.2/vhosts/name-based.html


11

Because they do not know what those resources are. The assets a web page requires are coded into the HTML. Only after a parser determines what those assets are can the y be requested by the user-agent. Additionally, once those assets are known, they need to be served individually so the proper headers (i.e. content-type) can be served so the user-agent ...


9

Working just within the browser, I couldn't tell if a particular direction was due to DNS or the webserver. To view webserver redirects in browsers, open up Developer Tools (either from the browser's menu, or by pressing F12), and click on the Network tab. It's a good idea to look for a Disable Cache option and check this so that all requests are done from ...


8

Web traffic Check the web logs to see which pages are being requested. If they're hitting a single page with a bad plug-in then it should show you the Page/Request URI that the user hit. Some things like scripts that resize image uploads can eat resources quickly. If you were getting a lot of traffic for the webserver then it would likely be someone ...


8

Because, in your example, web server would always send CSS and images regardless if the client already has them, thus greatly wasting bandwidth (and thus making the connection slower, instead of faster by reducing latency, which was presumably your intention). Note that CSS, JavaScript and image files are usually sent with very long expire times for exactly ...


8

It won't be any faster if the two are served by the same company. There are no opportunity for synergies between the domain registration, DNS, and web services. They don't use the same protocols, they don't use the same ports, they don't use the same types of servers. There isn't going to be an opportunity for any type of caching or reuse between the two....


8

Most people choose to pay for hosting for multiple reasons. Site availability: A personal computer isn't really built for serving a website to however many visitors you would get. Most people turn their PCs off from time to time, and most people have limited bandwidth for their internet connection. So, if your site suddenly got popular, most people's ...


7

HTTP2 is based on SPDY and does exactly what you suggest: At a high level, HTTP/2: is binary, instead of textual is fully multiplexed, instead of ordered and blocking can therefore use one connection for parallelism uses header compression to reduce overhead allows servers to “push” responses proactively into client caches More is ...


6

Digicert maintains pages for compatibility with certificate types: SAN certificate compatibility Wildcard certificate compatibility They note several server side compatibility problems with wildcard certificates but no client side problems. SAN certificates are problemic for some older browsers: Versions of major web browsers from before 2003. Older ...


6

Some search engines and bots send HEAD request to pages before sending the GET request for reasons like: Checking if the page size has changed Checking the last modified date etc. (Any other info the head would give them!) This would help large crawlers save a lot of bandwidth if they know a page has not been changed meanwhile and they don't have to crawl ...


6

Web client, usually a browser, opens a TCP socket to the server. The server software accepts the connection without knowing the specific site requested, and waits for an HTTP request to happen. The client sends the request, mainly composed of HTTP headers. One of these headers is the Host: example.com header, at this point the server is aware of the right ...


6

This isn't so much an SEO issue, as an issue as to whether your site would work at all. If it doesn't work in the browser then it's certainly going to hurt your SEO. Your HTML pages still need to return the text/html mime-type in order to be interpreted as HTML by the browser. ie. You need to send a Content-Type: text/html HTTP response header somehow. ...


5

It sounds like your domain records have not propagated yet. This means that the domain name servers around the world that store a record of your domain information haven't updated with the latest details, so they are still 'sending' visitors to your old server. The solution is to wait. It can take between 0 and 48 hours for domain name changes to update ...


5

There are several reasons to pay for web hosting. Reliability Your web host should have 99.9% uptime or better. It is hard to achieve that at home. Does your house ever lose power or have an internet outage? Do you have a backup generator and an uninterruptible power supply? Do you maintain multiple internet connections in case one goes out? Most web ...


5

Does server timezone affect SEO? No. I understand that the geolocation of a server impacts search engine rankings. Only indirectly. Major search engines don't care where the files are coming from. If I set the timezone to a different country for administrative purposes (to help an offshore team working from that country) does it affect rankings in the US? I ...


5

It could be possible to write some custom fail2ban rules to 95% solve this (ie with a short "findtime"). There might be edge cases with CGNAT and scripts but you can likely mitigate this by being specific about the form page to scan for, and it should be practical if your form is not getting 1000s of hits per hour from a targeted group. That said, ...


4

Your hosting provider should send you temporary URL for your hosting account and website. If they didn't, check your cPanel or whatever control panel and take out the ip address and paste it here: http://67.23.***.**/~yourhostingloginname/


4

I'm assuming that the bind.php file is not actually a part of Joomla, but rather a malicious script that the hacker uploaded to your site. In particular, just looking at the request parameters, it appears likely that the script is being used to send e-mail spam, possibly using someone else's hijacked e-mail account. Here's what the request parameters in ...


4

I am wondering if [it] is necessary [to have these files?] No, it isn't strictly necessary to have any particular files on a web server, including any "default" index pages (the ones you listed). [W]hat happens if there is no such file name[?] The server will generally do one of five things: dump a directory list of the files that are available (...


4

Your attempting to access the site using a IP address when your virtual host file is setup to use a domain name, not an IP address... so it'll throw up the default folder, in this case /var/www/html/. Fix 1. Local Virtual host file The most common and easy method to fix the issue would be to edit your host file within Windows or Mac, this will allow you to ...


4

The TLDR; answer: Send out a 404 header, don't change the url. The longer answer: There are more options then you suggest, which I'm not going to list here (because it's not really needed). The most basic solution is just to serve a 404 header when the page does not exist. The reason for this is simple; This is what visitors expect. You open a URL and ...


4

the url remains the same, but the page has the status code 404, This is certainly the preferred option, as Martijn suggests. For a "custom" error page on Apache (ignoring the CMS option for a moment), this could be achieved with the ErrorDocument directive: ErrorDocument 404 /errors/my404.php it can be a default server 404 page Using the default ...


4

tl;dr Yes, you can enable MultiViews to serve extensionless URLs. ie. Where the file extension is omitted from a URL that would otherwise map to an existing file. However, be aware of potential conflicts with mod_rewrite. the server won't even open links to the site that are missing a file extension You make it sound that this should be normal (or ...


3

You can do this with DNS if you use a service such as Route53: DNS GeoProximity Trafic Routing If you wanted to setup your own DNS server to do this kind of thing you could look into technologies such as GeoDNS with BIND. A different approach would be to use load balancing software such as HAProxy to geolocate the user, and then pass the traffic to your ...


3

TL;DR On Client side: open configuration file /etc/ssh/ssh_config; here look for PreferredAuthentications; make sure password comes after publickey and not viceversa In my case password was written before publickey, so ssh would prompt me for password even though I had copied my pub_key onto server. This problem can be found out easily using verbose: ...


Only top voted, non community-wiki answers of a minimum length are eligible