2

The home page of my website has some images of users, I want to know if I can stop the crawling of these images with robots.txt

Disallow: /img/users

My question is I want to know if that can make a difference from an SEO perspective by blocking these images with the robots file.

  • How do the URLs of the images look like? – unor Jul 19 '18 at 15:51
  • @unor like this www.example.com/img/users/nameofimage.png – sayou Jul 19 '18 at 17:25
2

Of course, you can block and there will be no problem with the SEO of the site. Maybe those images were generating some traffic, in that case, that's the only thing you are going to lose.

Make sure you use the correct user-agent:

By specifying Googlebot-Image as the User-agent, the images will be excluded from Google Image Search. It will also prevent cropping of the image for display within Mobile Image Search, as the image will be completely removed from Google's Image index. If you would like to exclude the images from all Google searches (including Google web search and Google Images), specify User-agent Googlebot.

User-agent: Googlebot
Disallow: /mg/users/ 

If they are already indexed, using robots.txt disallow will take too much to get them to disappear from serps. You can delete manually or block them through the server config.

2

I stop it with hotlink protection. its in most website controlpanels, but if you are serving this from a private server , in .htaccess, you would write:

     RewriteEngine on
      RewriteCond %{HTTP_REFERER} !^$
      RewriteCond %{HTTP_REFERER} !^http://(www\.)example.com/.*$ [NC]
      RewriteRule \.(gif|jpg|jpeg|bmp|zip|rar|mp3|flv|swf|xml|php|png|css|pdf)$ - [F]
2

Based on the Robots specification document, the best ways to handle it would either be via the X-Robots-Tag HTTP header technique, in which you may specify a noindex for any URL, include files:

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

To directly quote the documentation:

You can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags is not possible. Here's an example of adding a noindex X-Robots-Tag directive for images files (.png, .jpeg, .jpg, .gif) across an entire site:

<Files ~ "\.(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

Or to use the noimageindex to prevent crawling of images within that page in the markup. If you had something like a User directory in which all users had their images listed, this technique could be usefull.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.