2

A website I'm working on has a team section. Every team member has a dedicated site. On the site's bottom of each team member three other team members are displayed randomly.

Site

What happens here is that search engines index all four images. When I enter the name a specific team member into a search engine I get all four images as results. I want to show only the main image on top. So, how can I prevent search engines from indexing the three images at the bottom?

3

You can write that section of the page using JavaScript that is blocked by robots.txt. This is the general technique described in Preventing robots from crawling specific part of a page.

HTML

<img src=/images/main-person.jpg> Text about the main person.
<script src=/js/3-random-people.js></script>

Robots.txt

User-Agent: *
Disallow: /js/3-random-people.js

3-random-people.js

document.write('<img src=/images/other-person-1.jpg>')
document.write('<img src=/images/other-person-2.jpg>')
document.write('<img src=/images/other-person-3.jpg>')
1
  • Thank you for linking the other post and explaining it here. I thought there might be other techniques as the other post is relatively old.
    – Kiril
    May 14 at 12:32
2

You can prevent those (three) images being indexed by returning an X-Robots-Tag: noindex header as part of the HTTP response.

This is easier if there is something in the image filename that differentiates the images you don't want indexed, or perhaps they are in a different subdirectory. You can then simply prevent indexing for any images in that subdirectory.

For example, store these images in a /images/noindex subdirectory. And if you are using Apache then you can do something like the following in .htaccess or your server config:

# Prevent indexing of certain images
SetEnvIf Request_URI "^/images/noindex/" NOINDEX=1
Header set X-Robots-Tag "noindex" env=NOINDEX

Alternatively, you can prevent crawling of this subdirectory in robots.txt (which should also prevent indexing).

User-agent: *
Disallow: /images/noindex/

UPDATE: You don't necessarily need to duplicate these images if the images are already stored in /images and these images do need to be indexed when they appear on other pages.

You could just modify the URL of these secondary images to include /noindex (as above), and internally rewrite the request to remove the /noindex path segment in order to serve the actual image.

Although, naturally, you still lose the benefit of client-side caching, since it still looks like two images from a user-agent perspective.

For example...

  • Image is stored in /images/person_xyz.jpg
  • URL references /images/noindex/person_xyz.jpg

Using mod_rewrite in .htaccess, rewrite the request to remove /noindex and set an environment variable. Set the X-Robots-Tag response header based on this env var, similar to before:

RewriteEngine On

# Remove "/noindex" from image URL and set env var
RewriteRule ^(images/)noindex/([^.]+\.jpg)$ $1$2 [E=NOINDEX:1,L]

# Set "X-Robots-Tag: noindex" header if "/noindex" was present in URL
Header set X-Robots-Tag "noindex" env=REDIRECT_NOINDEX

Note that due to the rewrite (internal redirect), we need to check for REDIRECT_NOINDEX, rather than NOINDEX in the Header directive.

Alternatively, you could just append a query string to the image URL and avoid the rewrite. mod_rewrite is still required in order to examine the query string portion of the URL. For example, link to /images/person_xyz.jpg?noindex. Then, in your .htaccess file:

RewriteEngine On

# Set env var if query string set to "noindex"
RewriteCond %{QUERY_STRING} ^noindex$
RewriteRule ^images/ - [E=NOINDEX:1]

# Set "X-Robots-Tag: noindex" header if "noindex" was present in query string
Header set X-Robots-Tag "noindex" env=NOINDEX
3
  • Since there is only one image of each person I will have to serve the same image over two URLs: /images/person_xyz.jpg and /images/noindex/person_xyz.jpg. The image which is related to the team member site will be servers from the first URL and the random images of all team members will be served from the second URL.
    – Kiril
    May 11 at 10:33
  • 1
    Yes, you serve the same image over two URLs. However, you don't necessarily need to physically duplicate the image on the backend filesystem, you could just change the URL and internally rewrite the request. I've updated my answer. Aside The secondary images look smaller in your example, so if you did "duplicate" the image, this could perhaps be a smaller/optimised version?
    – MrWhite
    May 11 at 12:09
  • They are basically the same image just scaled down, so serving through a second URL should work without physically duplicating
    – Kiril
    May 14 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.