4

I understand that you can do the following to generally prevent images or pages from getting indexed.

Add to the page's meta section:

<meta name="robots" content="noindex" />

Edit the robots.txt file:

User-agent: *
Disallow: /images/

User-agent: *
Disallow: /$folder/whatever.jpg

User-agent: *
Disallow: /$folder/*.gif$

User-agent: Googlebot-Image
Disallow: /$folder/*.gif$

Say you have a page which is indexed and you have an image referenced in a stylesheet (background image, whatever) or an image in an ordinary img tag,

<img src="/$folder/img.gif" alt="whatever title="yadda" />

So, according to the robots.txt the folder or specific image should not be indexed, however, the page is indexed (no no-index).

Will the search engines, in specific Googlebot-Image, honor the request? Index the page to the exclusion of the image in an img tag?

Will it also honor the request with regard to styled background images?

| improve this question | |
2

Will the search engines, in specific Googlebot-Image, honor the request? Index the page to the exclusion of the image in an img tag?

Yes. Regardless of how the image is referenced on your site (img element, CSS background-image, etc.), if it is blocked by robots.txt then Googlebot-Image will be unable to crawl that image, which will most probably prevent that image being indexed and being returned in Google image search.

If pages are blocked from being crawled in robots.txt then they can still be indexed (and potentially returned in the SERPs) if other external pages link to them (although this is a URL only result - no title or description appears in the SERPs as the pages themselves have not been crawled). However, I have not seen a similar result with regards to images. If the image is blocked by robots.txt then it's not indexed.

You can also apply the same robots "meta tag" (specifically, the X-Robots-Tag HTTP response header in the case of non-HTML resources) to images as you can to pages if you want to, although this will generally be unconditional - applying to all bots and user agents.

| improve this answer | |
  • Thx - I've read numerous articles that talk about how to set up the robots.txt but they don't specifically answer the question. I use the robots.txt for good crawlers and .htaccess for curbing those that don't behave. With regard to indexing, do these sitescraper services have the capability of finding pages/files that are not linked to anywhere? Say you have a hidden folder, .hidden, do not even reference it in robots.txt. Can it be scraped? Or indexed? – Bob M Feb 19 '17 at 20:06
  • A scraper that gathers content by spidering the site cannot find it if it is not linked to from anywhere (unless you have director indexes enabled). When search engines find supposedly hidden pages it's generally because the URL has been leaked somehow. (The problem is that it can be too easy for these URLs to be discovered.) – MrWhite Feb 19 '17 at 22:13
  • The reassurance is comforting. Hypothetically I store some development stuff in a hidden folder, camouflaging the name. If I were to add an entry into the robots.txt instructing search engines to stay out I'd imagine that would be like making up a personalized invitation to bad bots and scrapers to give it your best shot - so best not to do so? – Bob M Feb 19 '17 at 22:56
  • If you are storing sensitive information that must not be accessed then it should be protected by some kind of authentication. – MrWhite Feb 19 '17 at 23:47
0

I believe X-Robots-Tag "noindex" is ideal for this case.

Header set X-Robots-Tag "noindex"

you can see examples of how to noindex png, jp(e)g, and gif files. That's specifically for an Apache .htaccess file. The implementation would be slightly different on other servers - but you get the idea.

If things are not yet indexed, you may also be able to fix the problem with a robots.txt block - but once they're indexed, you'll need to use a noindex to remove them.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.