That is the longest robots.txt file I've ever seen. It is 88K. Google officially has a limit of 500K, but I suspect the length of the file has something to do with it.
I don't see any other obvious problems with your robots.txt:
- The file appears well formed.
- It doesn't have extra new lines
- It has only one section
User-agent: * which should apply to all robots
- It only has
Disallow: directives and a
Sitemap: directive, which should be fine.
- The robots.txt file is in the right place at the root directory of your domain
- The robots.txt file is returning
200 OK HTTP status
- The robots.txt file has an appropriate
Content-Type: text/plain; charset=utf-8
Disallow: /samer.kassem-saad/* rule should match your example URL of
- You say that your robots.txt file has been in place more than 24 hours, and I believe you.
One thing I notice is that all your rules have a wildcard
* in them, even when they don't need to. These two rules should be equivelant:
They are the same because without wildcards, robots.txt rules are all "starts with" rules. A star at the end of the rule shouldn't ever be necessary. While Googlebot (and other search engine bots) should understand the wildcard syntax, most robots do not. Removing the trailing star would make your robots.txt file understood by far more robot user agents. Your only rule that actually needs the wildcards is
To try to make your robots.txt shorter, you could try combining some rules. A single
Disallow: /werner. rule could replace nine other rules, assuming that there isn't any
/werner* pages you do want to have crawled. Searching Google for
site:www.signal-iduna.de inurl:/werner doesn't show any URLs that aren't disallowed, so I think you do have some room to combine a lot of your rules.
The only crawler I know of that would have a problem with combined rules is Baidu's bot. Last time I checked, they only support starts with rules at directory boundaries. For Baidubot,
Disallow: /aa would prevent crawling of
/aa/bb but not of
Another possible workaround would be to include
<meta name="robots" content="noindex"> tag in every page you disallow. That way even if Google does end up crawling the pages, at least Google wouldn't index them.