4

Two months ago i disallowed some directories from crawling with robots.txt. Since then i monitor logfiles everyday and realize, that Googlebot absolutely ignores robots.txt.

Really, it crawls every single url, it was crawling before directories were disallowed for crawling.

Search console test displays, as all urls from disallowed directories would be allowed! Only live tests of search console display urls as disallowed - it says, disallow rules are correct and working.

According to disallowed crawling urls from disallowed directories should appear in index without snippets. But, urls from disallowed directories appear with snippets, despite of their caching date is from this week.

All main rules i know about Google and websites aren't working there.

Any ideas, what could be happen here? Example url.

  • I wonder if this has to do with apex vs www domain, maybe it's checking against one and not the other, and redirects are not set up. – Maximillian Laumeister Feb 7 at 17:01
  • @MaximillianLaumeister no there are no these or similar issues - i work with the site since some years and know it well as my own vest pocket. Correct setup is validate though live test through GSC - it confirms correct disallowing. – Evgeniy Feb 7 at 21:58
  • How are your redirects and canonicals worked out? Are these pages getting indexed in some other way than how you intend? – Greg Nickoloff Feb 11 at 18:19
  • @GregNickoloff why? what do you mean? Don't follow to your question's relation to the subject. – Evgeniy Feb 11 at 20:09
1

That is the longest robots.txt file I've ever seen. It is 88K. Google officially has a limit of 500K, but I suspect the length of the file has something to do with it.

I don't see any other obvious problems with your robots.txt:

  • The file appears well formed.
    • It doesn't have extra new lines
    • It has only one section User-agent: * which should apply to all robots
    • It only has Disallow: directives and a Sitemap: directive, which should be fine.
  • The robots.txt file is in the right place at the root directory of your domain
  • The robots.txt file is returning 200 OK HTTP status
  • The robots.txt file has an appropriate Content-Type: text/plain; charset=utf-8
  • Your Disallow: /samer.kassem-saad/* rule should match your example URL of /samer.kassem-saad/kfz-versicherung.php
  • You say that your robots.txt file has been in place more than 24 hours, and I believe you.

One thing I notice is that all your rules have a wildcard * in them, even when they don't need to. These two rules should be equivelant:

  • Disallow: /samer.kassem-saad/*
  • Disallow: /samer.kassem-saad/

They are the same because without wildcards, robots.txt rules are all "starts with" rules. A star at the end of the rule shouldn't ever be necessary. While Googlebot (and other search engine bots) should understand the wildcard syntax, most robots do not. Removing the trailing star would make your robots.txt file understood by far more robot user agents. Your only rule that actually needs the wildcards is Disallow: *widgets/adp-suche*/*.

To try to make your robots.txt shorter, you could try combining some rules. A single Disallow: /werner. rule could replace nine other rules, assuming that there isn't any /werner* pages you do want to have crawled. Searching Google for site:www.signal-iduna.de inurl:/werner doesn't show any URLs that aren't disallowed, so I think you do have some room to combine a lot of your rules.

The only crawler I know of that would have a problem with combined rules is Baidu's bot. Last time I checked, they only support starts with rules at directory boundaries. For Baidubot, Disallow: /aa would prevent crawling of /aa, /aa/ and /aa/bb but not of /aardvark.

Another possible workaround would be to include <meta name="robots" content="noindex"> tag in every page you disallow. That way even if Google does end up crawling the pages, at least Google wouldn't index them.

| improve this answer | |
  • About your recommended workaround: afaik this workaround leads to situation, where Googlebot can't get noindex rule applied as meta robots, becasue of disallowed crawling. This could be the cause of appearence of an url in SERP without snippet, but with the notice "A description for this result is not available because of this site's robots.txt – learn more." – Evgeniy Feb 8 at 15:44
  • 1
    normally that would be the case. If Google can't crawl it it's not going to see the noindex rule. However in this case it appears that Google is crawling them anyway so putting that rule in at the least is not going to hurt. If you want to do you just use the no index rule then put the no indexes in and remove all your disallows. – Stephen Ostermiller Feb 8 at 17:47
  • I agree with stephen, john also said same thing, twitter.com/IMgoyllo/status/903897084377554945 twitter.com/IMgoyllo/status/902120475605688320 – Goyllo Mar 9 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.