4

Are these bots that index my site(s) or are they malicious by nature and should be banned using an IP deny filter? You tell us. If they're interacting with your site as any normal user would there is no need to block them. If they're using the site more than visitors from other regions then your site may appeal to them then users from other regions. ...


3

There is no way at this point to do that with Linode. Having used them a while back as a provider I believe they sent emails at the time to alert when usage approached 100% of the plan allowance so you could make the appropriate changes to reduce your costings but there is no way in advance to prevent the usage from going over without actually monitoring it.


3

By default, once a image is loaded it will be cached by the browser. Every additional request for this (exact) image url will be delivered out of the browser cache. Example: 250 Image tags with just 10 unique image urls. This will only waste the traffic of the size of these 10 images. You can check that easily by hitting the developer tools of your browser (...


3

I did some research for this question and found some interesting thins, such as: 1. Is it a fake crawler? -> https://stackoverflow.com/questions/15840440/google-proxy-is-a-fake-crawler-for-example-google-proxy-66-249-81-131-google-c Conclusion from the user: These 'crawlers' are not crawlers but are part of the live website preview used in the Google ...


3

Reduce Your File Size Before I address your CDN concerns, you should really try to reduce your file size for the podcasts. Unless your podcast episodes are 2 to 4 hours long, each, you shouldn't be reaching 50 megs per file. Try reducing the audio quality in your mp3 files (and use .mp3 file compression if you're not already - it's the podcast industry ...


3

398 (kb/sec) * 60 (sec/min) * 60 (min/hour) * 24 (hours/day) * 30 (days/month) / (8 (bits/byte)) / (1024 KB/MB) / (1024 MB/GB) = 123 GB/Month


2

CloudFlare works great. I use them for several sites. Here's a very basic summary of how it works: You update your DNS settings so that traffic to your site goes through CloudFlare first. Because they protect a large number of websites they are able to see things, like IP addresses, that are responsible for unusual activity. Once a certain threshold is ...


2

Anything that requests a resource (a.k.a. web page, image file, stylesheet, etc) from your server uses bandwidth. So crawling other sites from your server will count against your bandwidth limit.


2

Yes, using a CDN will reduce the bandwidth on your hosting account. Exactly how much may depend on the CDN used. If you use Amazon CloudFront, you can use S3 as the "origin server", which means you upload the files there and no bandwidth is used on your server. Other services use your regular website as the origin, meaning when the CDN is asked for a file, ...


2

Using a CDN does remove traffic from your main hosting server and does reduce bandwidth there. A CDN also serves files faster than your main hosting because it caches the files in multiple locations so that it can serve them from near where the user is located. For more information about what a CDN is and how it works, read the CDN article on Wikipedia.


2

As you mention, "unlimited" bandwidth isn't really unlimited. The ISP can't afford infinite bandwidth with a finite customer base and is counting on the fact that almost all sites do very little in the way of traffic to maintain the illusion of infinite resources. An unlimited plan is really meant to handle a brief and infrequent traffic spike (e.g. your ...


2

I am assuming Apache and .htaccess. From: https://perishablepress.com/creating-the-ultimate-htaccess-anti-hotlinking-strategy/ RewriteEngine on RewriteCond %{HTTP_REFERER} !^$ RewriteCond %{REQUEST_FILENAME} -f RewriteCond %{REQUEST_FILENAME} \.(gif|jpe?g?|png)$ [NC] RewriteCond %{HTTP_REFERER} !^https?://([^.]+\.)?domain\. [NC] ...


2

It's not possible to view an image in a web page without downloading it. Your browser downloads every image to a local cache folder for every image you view in your browser. So, in answer to your question, you will always be downloading 100kb. Now, if your goal is to offload your image downloads from your server, you should consider a Content Delivery ...


2

You could use VNstat to send you hourly reports via a cron, and then manually shut down your http server via a shell on your mobile device.


1

Find two new ISPs, bond your network, approach your original ISP for better service. If ISP number one doesn't offer quicker service and a larger cap then drop them. Another advantage of bonding is that if you lose one connection you're still up. Of course it costs more but you get what you pay for and if it's a business you are lucky they don't charge ...


1

If you haven't enabled hourly or daily Cost and Usage detail reports, do that now. These reports break down everything that generates a charge, by resource-id and the exact type of usage that generated the charge -- so traffic to the Internet, traffic using public IP addresses, etc., is all broken down by the specific instance, balancer, bucket, etc., that ...


1

If the URLs they are requesting follow some patterns, I think it is a good idea to block those pages. Based on your comment above, in your robots.txt you could add: User-agent: * Disallow: /data/ That will work as long as you don't have any real URLs beginning with data. The second thing you may want to do is decrease Google's crawl rate. You can do this ...


1

There isn't a 'one size fits all' solution to HTTP caching. The right solution for you depends on your site and the nature of your traffic. E.g. do most of your requests come from repeat visitors, or do people visit your site once and never return? If they are repeat visitors, how frequently do they visit? Are they likely to be checking regularly for ...


1

The first issue here as I see it is the misconception that you absolutely must maintain a minimum number of words per page to avoid thin content penalties. This used to be the case a few years back but these days even Google's own advice is don't write a site for Google write it for your consumers. Second matter is that Google fully expects the mobile ...


1

You should filter your own ip address so it doesn't show up at the results. Although I am not familiar with webalizer, google analytics have this option if you go to the filters and add an exclude command for your ip adresses.


1

If you host videos on on a server, there is no way you can stream those videos from that server without using the bandwidth of that server. However, here are some way that you can try to reduce the bandwidth utilisation: Cache the streamed portions of the video on the client browser. This way, the receiver can replay it without streaming it again, and hence ...


1

The first step would be to figure out if this bot reads robots.txt. You can define a crawl-delay in there for this bot. This question goes in more detail about what you have to do. You can disallow this bot to crawl your site via there too. Please note that rogue bots do not read, or simply ignore, robots.txt and that changes to this file might not work. If ...


1

Please check detailed bandwidth statistics via your control panel and you will know all the details regarding this. Moreover, the access logs will give you all the details regarding the ip addresses from which the site was accessed.


1

AdWords already does not count visits from the Googlebot or other known search engine crawlers, it is intelligent enough to recognise these as invalid clicks. As for blocking IP addresses or address blocks this is a bad idea as not all IP's in a block will be dangerous, as well as not all users on an IP will be dangerous. There is also no reason to ...


1

Here's the update in 2020: The mistake here is the assumption that bandwidth is 0.18c/GB, even if it's much less now on many platforms lets ignore this. There are server hosts all over the world which offer unlimited bandwidth on dedicated servers such as servers.com. Additionally cloud providers such as DigitalOcean provide 1TB of free data per droplet you ...


Only top voted, non community-wiki answers of a minimum length are eligible