I have the following in my robots.txt file:

# Block Googlebot-Image from cache
User-agent: Googlebot-Image
Disallow: /media/catalog/product/cache/

# Crawlers Setup
User-agent: *

# Directories
Disallow: /404/
Disallow: /app/

Disallow: /cgi-bin/
Disallow: /downloader/

Disallow: /errors/

But I am still seeing this in my logs: - - [23/Jul/2016:06:30:12 -0700] "GET /media/catalog/product/cache/1/image/1800x/040ec09b1e35df139433887a97daa66f/n/d/nd01120a10g.jpg HTTP/1.1" 200 106745 "-" "Googlebot-Image/1.0"

Am I doing something wrong?

The thing is, I don't want those cached images indexed because they get deleted and I don't want 404 penalties.

  • First things first: there is no such thing as a 404 penalty. 404s are a part of how the web is supposed to work. Google does not write the rules. Secondly, you do have to give Google time to notice the change. In theory, the robots.txt file should be read prior to any request when the robots.txt file has not been read within 24 hours. This is Googles public statement, however, sometimes this does not happen exactly how Google specifies and takes a bit longer. Otherwise, it all looks goo to me.
    – closetnoc
    Jul 23 '16 at 16:25
  • I've often read that Google recommends using Webmaster Tools to identify 404 errors, and use 301 rewrites to point those URLs to an existing (and relevant) alternative. This implies Google does consider this a factor (one of thousands) that weighs in on your overall "score" with them. So I'm inclined to disagree with you.
    – Nathan
    Jul 25 '16 at 10:01
  • Quite the contrary! Google wants 404 errors for pages that do not exist. Google also pings the dog snot out of you if they think you have a soft 404. Google does like a 301 where it is appropriate, however, G does not recommend replacing a valid 404 with a 301. It does recommend a 301 for valid old page to new page use where it is relevant. Cheers!!
    – closetnoc
    Jul 25 '16 at 15:22
  • Can you point to any authoritative citations on this?
    – Nathan
    Jul 25 '16 at 17:42
  • Each of these have a valid purpose and Google prefers to have a clean index. support.google.com/webmasters/answer/181708?hl=en webmasters.googleblog.com/2011/05/do-404s-hurt-my-site.html It does not take too many searches to know what Google wants. I found these in just a couple of seconds. Stay away from the SEO sites for what Google wants. Cheers!!
    – closetnoc
    Jul 25 '16 at 17:53

I've noticed that Google does the same thing to some of my sites, but they don't actually index the content they've crawled.

Read here: https://support.google.com/webmasters/answer/35308?hl=en

According to Google, using that in the robots.txt prevents them from indexing the content, it doesn't say that it will stop crawling it entirely.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.