4

Are these bots that index my site(s) or are they malicious by nature and should be banned using an IP deny filter? You tell us. If they're interacting with your site as any normal user would there is no need to block them. If they're using the site more than visitors from other regions then your site may appeal to them then users from other regions. ...


3

Technically speaking, Workers have no problem with serving 1000 TB/month and trillions of requests. You will not see any performance degradation at such traffic levels, nor will it negatively impact other users. The limit on requests that you mentioned only applies to the free tier. Legally speaking, the supplemental Terms of Service for Workers are found ...


3

You can definitely proxy heavy traffic through Workers, but, if you try to cache or serve content like videos, (excessive amounts of) photo/audio or other large files, then that may violate Section 2.8 of Cloudflare's ToS and result in a hold on your account: https://www.cloudflare.com/terms/ This is something to bear in mind as a self-serve customer. ...


3

There is no way at this point to do that with Linode. Having used them a while back as a provider I believe they sent emails at the time to alert when usage approached 100% of the plan allowance so you could make the appropriate changes to reduce your costings but there is no way in advance to prevent the usage from going over without actually monitoring it.


3

By default, once a image is loaded it will be cached by the browser. Every additional request for this (exact) image url will be delivered out of the browser cache. Example: 250 Image tags with just 10 unique image urls. This will only waste the traffic of the size of these 10 images. You can check that easily by hitting the developer tools of your browser (...


3

Reduce Your File Size Before I address your CDN concerns, you should really try to reduce your file size for the podcasts. Unless your podcast episodes are 2 to 4 hours long, each, you shouldn't be reaching 50 megs per file. Try reducing the audio quality in your mp3 files (and use .mp3 file compression if you're not already - it's the podcast industry ...


3

I did some research for this question and found some interesting thins, such as: 1. Is it a fake crawler? -> https://stackoverflow.com/questions/15840440/google-proxy-is-a-fake-crawler-for-example-google-proxy-66-249-81-131-google-c Conclusion from the user: These 'crawlers' are not crawlers but are part of the live website preview used in the Google ...


2

Yes, using a CDN will reduce the bandwidth on your hosting account. Exactly how much may depend on the CDN used. If you use Amazon CloudFront, you can use S3 as the "origin server", which means you upload the files there and no bandwidth is used on your server. Other services use your regular website as the origin, meaning when the CDN is asked for a file, ...


2

Using a CDN does remove traffic from your main hosting server and does reduce bandwidth there. A CDN also serves files faster than your main hosting because it caches the files in multiple locations so that it can serve them from near where the user is located. For more information about what a CDN is and how it works, read the CDN article on Wikipedia.


2

As you mention, "unlimited" bandwidth isn't really unlimited. The ISP can't afford infinite bandwidth with a finite customer base and is counting on the fact that almost all sites do very little in the way of traffic to maintain the illusion of infinite resources. An unlimited plan is really meant to handle a brief and infrequent traffic spike (e.g. your ...


2

I am assuming Apache and .htaccess. From: https://perishablepress.com/creating-the-ultimate-htaccess-anti-hotlinking-strategy/ RewriteEngine on RewriteCond %{HTTP_REFERER} !^$ RewriteCond %{REQUEST_FILENAME} -f RewriteCond %{REQUEST_FILENAME} \.(gif|jpe?g?|png)$ [NC] RewriteCond %{HTTP_REFERER} !^https?://([^.]+\.)?domain\. [NC] ...


2

It's not possible to view an image in a web page without downloading it. Your browser downloads every image to a local cache folder for every image you view in your browser. So, in answer to your question, you will always be downloading 100kb. Now, if your goal is to offload your image downloads from your server, you should consider a Content Delivery ...


2

You could use VNstat to send you hourly reports via a cron, and then manually shut down your http server via a shell on your mobile device.


1

If your needs are large enough then that means you should probably get your own AS (like an ISP) and sign up with a tier 1 provider for the peering arrangements, and you buy/rent servers preferably in multiple locations. There are more options than AWS. Regardless of you use you can negotiate discounts based on volume. To give you an idea, I run my own VPN ...


1

Find two new ISPs, bond your network, approach your original ISP for better service. If ISP number one doesn't offer quicker service and a larger cap then drop them. Another advantage of bonding is that if you lose one connection you're still up. Of course it costs more but you get what you pay for and if it's a business you are lucky they don't charge ...


1

If you haven't enabled hourly or daily Cost and Usage detail reports, do that now. These reports break down everything that generates a charge, by resource-id and the exact type of usage that generated the charge -- so traffic to the Internet, traffic using public IP addresses, etc., is all broken down by the specific instance, balancer, bucket, etc., that ...


1

If the URLs they are requesting follow some patterns, I think it is a good idea to block those pages. Based on your comment above, in your robots.txt you could add: User-agent: * Disallow: /data/ That will work as long as you don't have any real URLs beginning with data. The second thing you may want to do is decrease Google's crawl rate. You can do this ...


1

There isn't a 'one size fits all' solution to HTTP caching. The right solution for you depends on your site and the nature of your traffic. E.g. do most of your requests come from repeat visitors, or do people visit your site once and never return? If they are repeat visitors, how frequently do they visit? Are they likely to be checking regularly for ...


1

The first issue here as I see it is the misconception that you absolutely must maintain a minimum number of words per page to avoid thin content penalties. This used to be the case a few years back but these days even Google's own advice is don't write a site for Google write it for your consumers. Second matter is that Google fully expects the mobile ...


1

You should filter your own ip address so it doesn't show up at the results. Although I am not familiar with webalizer, google analytics have this option if you go to the filters and add an exclude command for your ip adresses.


1

If you host videos on on a server, there is no way you can stream those videos from that server without using the bandwidth of that server. However, here are some way that you can try to reduce the bandwidth utilisation: Cache the streamed portions of the video on the client browser. This way, the receiver can replay it without streaming it again, and hence ...


1

The first step would be to figure out if this bot reads robots.txt. You can define a crawl-delay in there for this bot. This question goes in more detail about what you have to do. You can disallow this bot to crawl your site via there too. Please note that rogue bots do not read, or simply ignore, robots.txt and that changes to this file might not work. If ...


1

AdWords already does not count visits from the Googlebot or other known search engine crawlers, it is intelligent enough to recognise these as invalid clicks. As for blocking IP addresses or address blocks this is a bad idea as not all IP's in a block will be dangerous, as well as not all users on an IP will be dangerous. There is also no reason to ...


1

Here's the update in 2020: The mistake here is the assumption that bandwidth is 0.18c/GB, even if it's much less now on many platforms lets ignore this. There are server hosts all over the world which offer unlimited bandwidth on dedicated servers such as servers.com. Additionally cloud providers such as DigitalOcean provide 1TB of free data per droplet you ...


Only top voted, non community-wiki answers of a minimum length are eligible