0

I am building a site like the home page of MSN, google news, yahoo. It is basically an aggregation site.

To save the server space, I thought to display image directly from the source without the need to download to my server.

Is there any problem with it? Let say I have 1,000 page views per day.

Image link will be like,

<img src="domain.com/abc.jpg/>

So I request the above image 1,000 times per day. Am I correct? So will there is a chance that my IP/Domain blacklisted by those site since I request too many times?

1
  • This is known as image hotlinking, and it will eat the bandwidth of the hosting site, who can blacklist your IP after learning about you. What you can do is contact the sources and tell them about the benefits and allow you to hot link their images, otherwise it is not the right way. Commented Jul 24, 2019 at 20:13

5 Answers 5

3
+25

You are/your server is not requesting any images. Your visitors are requesting them, via their browser, using their IP.

Therefore there will be no massiv download of images from your IP.

The only thing that could lead to blocking of your domain could be the referer. Each time a browser send a request to a server, in this case if it access domain.com/abc.jpg, it will tell domain.com from where it got the link.

If a server gets massiv request to images where the referer is not its domain itself, but your domain, it could trigger some (automatic) blocking.

1
  • A lot of sites already block remote image access for copyright and/or bandwidth reasons - the OP might be seeing the images because they are in a local cache.
    – winwaed
    Commented Jul 24, 2019 at 14:23
1

This is a difficult question to answer. It really depend on the websites you used to get images. Some websites are allowed to query the web page & some websites are not.

In the other hand some websites provide API's to access there websites to fetch data.

in both cases you need to read TOS or ask them directly.

Following method will do the trick if you really need to crawl the website but these action may lead to some other problems. ( may be a law suite, so I'm not going to recomand using them )

You can cache the queries for some time frame & this will reduce the requests to original website.

You can generate requests with different user agent, ip each time to avoid any request limits or blacklisting.

again, I'm strongly disagree with using these methods.

0

Was working on a an art platform that was doing just this: scraping other platforms/resources for images/data. Platform had a steady 500k visitors/month traffic, never was there an issue with this. But then again, who knows, you have situations like this as well. Honestly, consider reaching out to those sources asking for a permission.

0

Your requirement will make other hosting website more trouble. suggest you to save these Images on your Google Photos and embed these images on your website.

-2

There are several solutions to this:

1. Lazy Load Images As none of these images are your content, Lazy Loading them won't matter as Google won't index them for your site anyway. This defers the loading of images until the user scrolls down to see them. How to setup LazyLoad.

The benefit of this is fewer bytes to load on initial page load. Often not all images are necessary to be displayed in the viewport of the user. Once the user starts scrolling we need more and more content that can be loaded in.

2. Set a longer cache policy This will help with users presented with the same image on multiple pages, so won't affect landing page load times like Lazy Loading. But it will improve overall site speed if these images are used on the same page. One draw back you have here is that if you are pulling these in dynamically and they have the same file name, the cache policy would force the showing of the cached image. This is something you would need to look into, maybe set the cache policy to 1 or days if this is the case.

This can be done in the access file but depends on your server hosting, adding the following code to your htaccess file:

<FilesMatch "\.(?i:gif|jpe?g|png|ico|css|js|swf)$">

  <IfModule mod_headers.c>
    Header set Cache-Control "max-age=172800, public, must-revalidate"
  </IfModule>

</FilesMatch>

3. Reduce the number of images being called by reducing the number of images This is the least interesting answer for you as you will need to get rid of some of the images.

4. Compress images using htaccess This will allow you to set cache policies for each file type and ensure that images are compressed:

480 weeks

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
Header set Cache-Control "max-age=290304000, public"
</FilesMatch>

2 DAYS

<FilesMatch "\.(xml|txt)$">
Header set Cache-Control "max-age=172800, public, must-revalidate"
</FilesMatch>

2 HOURS

<FilesMatch "\.(html|htm)$">
Header set Cache-Control "max-age=7200, must-revalidate"
</FilesMatch>

<ifModule mod_gzip.c>
  mod_gzip_on Yes
  mod_gzip_dechunk Yes
  mod_gzip_item_include file \.(html?|txt|css|js|php|pl)$
  mod_gzip_item_include handler ^cgi-script$
  mod_gzip_item_include mime ^text/.*
  mod_gzip_item_include mime ^application/x-javascript.*
  mod_gzip_item_exclude mime ^image/.*
  mod_gzip_item_exclude rspheader ^Content-Encoding:.*gzip.*
</ifModule>

The first option is the best and easiest and will reduce load tines significantly. Let me know if this answered your question or if you need some clarification?

1
  • 1
    I am sorry, but this does not answer OPs question at all.
    – Mnea
    Commented Jul 24, 2019 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.