I have the following in my robots.txt file:
# Block Googlebot-Image from cache User-agent: Googlebot-Image Disallow: /media/catalog/product/cache/ # Crawlers Setup User-agent: * # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /errors/ ...
But I am still seeing this in my logs:
18.104.22.168 - - [23/Jul/2016:06:30:12 -0700] "GET /media/catalog/product/cache/1/image/1800x/040ec09b1e35df139433887a97daa66f/n/d/nd01120a10g.jpg HTTP/1.1" 200 106745 "-" "Googlebot-Image/1.0"
Am I doing something wrong?
The thing is, I don't want those cached images indexed because they get deleted and I don't want 404 penalties.