Googlebot ignores robots.txt

Question

Two months ago i disallowed some directories from crawling with robots.txt. Since then i monitor logfiles everyday and realize, that Googlebot absolutely ignores robots.txt.

Really, it crawls every single url, it was crawling before directories were disallowed for crawling.

Search console test displays, as all urls from disallowed directories would be allowed! Only live tests of search console display urls as disallowed - it says, disallow rules are correct and working.

According to disallowed crawling urls from disallowed directories should appear in index without snippets. But, urls from disallowed directories appear with snippets, despite of their caching date is from this week.

All main rules i know about Google and websites aren't working there.

Any ideas, what could be happen here? Example url.

Answer 1

That is the longest robots.txt file I've ever seen. It is 88K. Google officially has a limit of 500K, but I suspect the length of the file has something to do with it.

I don't see any other obvious problems with your robots.txt:

• The file appears well formed.
  • It doesn't have extra new lines
  • It has only one section User-agent: * which should apply to all robots
  • It only has Disallow: directives and a Sitemap: directive, which should be fine.
• The robots.txt file is in the right place at the root directory of your domain
• The robots.txt file is returning 200 OK HTTP status
• The robots.txt file has an appropriate Content-Type: text/plain; charset=utf-8
• Your Disallow: /samer.kassem-saad/* rule should match your example URL of /samer.kassem-saad/kfz-versicherung.php
• You say that your robots.txt file has been in place more than 24 hours, and I believe you.

One thing I notice is that all your rules have a wildcard * in them, even when they don't need to. These two rules should be equivelant:

• Disallow: /samer.kassem-saad/*
• Disallow: /samer.kassem-saad/

They are the same because without wildcards, robots.txt rules are all "starts with" rules. A star at the end of the rule shouldn't ever be necessary. While Googlebot (and other search engine bots) should understand the wildcard syntax, most robots do not. Removing the trailing star would make your robots.txt file understood by far more robot user agents. Your only rule that actually needs the wildcards is Disallow: *widgets/adp-suche*/*.

To try to make your robots.txt shorter, you could try combining some rules. A single Disallow: /werner. rule could replace nine other rules, assuming that there isn't any /werner* pages you do want to have crawled. Searching Google for site:www.signal-iduna.de inurl:/werner doesn't show any URLs that aren't disallowed, so I think you do have some room to combine a lot of your rules.

The only crawler I know of that would have a problem with combined rules is Baidu's bot. Last time I checked, they only support starts with rules at directory boundaries. For Baidubot, Disallow: /aa would prevent crawling of /aa, /aa/ and /aa/bb but not of /aardvark.

Another possible workaround would be to include <meta name="robots" content="noindex"> tag in every page you disallow. That way even if Google does end up crawling the pages, at least Google wouldn't index them.