2

I have been using Raven Tools, and I want these dynamically generated thumbnails to be not crawled or not be flagged as No TITLE images.

I tried adding X-Robots-Tag "noindex, nofollow" on each image within my webserver (nginx), but Raven's Site Auditor still reads the images and flags them.

How do I prevent them from being crawled?

Edit: The problem is Raven Tools flag these dynamically generated images as NO TITLE, since a WP plug-in generates its files.

  • X-Robots-Tag is a header, so it can only be seen as they are crawled. It will prevent search engine indexing, but it can't prevent crawling. – Stephen Ostermiller Jan 23 at 12:36
  • 1
    Do the thumbnails have a common path? For example are they all in the /thumbnails/ directory? If so, or if you can make it so, you should be able to use robots.txt. – Stephen Ostermiller Jan 23 at 12:39
  • @StephenOstermiller, as far as I know, a plug-in generates the thumbnails on the fly, I need to confirm it though with our developers. – Batutoy Tria Jan 23 at 12:43
  • In what way do these images have "no title"? Images themselves don't normally have a "title"? Are you referring to the possibly "random" nature of the filename? I'm wondering if the correct response is really to "prevent crawling"? Or you just want the "SEO Tool" warning to go away? – MrWhite Jan 23 at 16:01
1

This is not a direct answer, but you should cache your dynamically generated thumbnails so they are generated only once, and then served only from the cache.

Generating thumbnails is a very expensive process (for server CPU and memory). So you really should avoid doing it all the times.

My suggestion then is:

  • When user requests a page that has thumbnails, page should check if the cached thumbnail image is present.
  • If a cached thumbnail do not exists yet, then the page must generate the thumbnail and save it to the cached thumbnails directory.
  • The page will always output the cached image URL, and never a URL for a thumbnail-generating script.
  • You should generate all thumbnails in a specific directory on your server (like /thumbs/), so you can add this directory to robots.txt and this way you can "ask" for the crawler to not index images on this folder.
  • Thanks for the response, the images were all cached on the server side before serving. The problem is, these dynamically generated images has no "title", which causes our SEO tools to flag them. – Batutoy Tria Jan 23 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.