From what I learned, there's one way I could cause all URLs on a domain strictly serving images not to be indexed and that is with the x-robots-tag HTTP header. Now I check my logs and find out that google and even baidu are downloading the entire contents of the image URLs. I was hoping they'd stop downloading when they came across this line:
X-Robots-Tag: noindex, noimageindex
Either I formatted that line wrong (used wrong casing or wrong order of values or something), or search engines are just plain dumb and they just decide to download everything just to waste customer's money.
I looked into robots.txt and thought of using the noindex line but when I did, google complained about having no access to what they call an "important url" when it isn't important.
I don't want to block their IP's because I have text-based content on another domain running on the same server that I want them to index.
I'm tempted to offer search engines the equivalent of what users get if they requested the URL via the HEAD method (full headers but no actual content), but I might get penalized for content cloaking.
Is there something I can do to rectify this?
Disallow: /
syntax (notnoindex
). Make sure you're not linking to the root of the image domain anywhere, if that's what the error in webmaster tools was complaining about. If that error persists, allow Google to crawl the root of the domain but not images (i.e. doDisallow: /images
or similar depending on what your URLs look like).