1

We have recently integrated with google webmaster tools.

Now, on one of the pages, google complains: Indexed, though blocked by robots.txt

Our robots.txt looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php


Sitemap: http://<domain>/sitemap.xml.gz

So, if I understand how it works, all the rest of the URL (except /wp-admin/) should not be blocked.

Why am I getting this warning?

Improve this question
9
  • Nothing you have said so far looks out of place. Have you updated robots.txt recently? Is the page on a subdomain? Does http: vs https: match what you would expect?
    – Stephen Ostermiller ♦
    Feb 18, 2019 at 19:49
  • Well, we just published it several days ago, and it only started indexing
    – Shurik Agulyansky
    Feb 18, 2019 at 19:50
  • And we still don't have ssl installed, so that part should be fine
    – Shurik Agulyansky
    Feb 18, 2019 at 20:42
  • What was on the domain before this site you published a few days ago?
    – Stephen Ostermiller ♦
    Feb 18, 2019 at 20:56
  • @StephenOstermiller Another wordpress installation with robots.txt blocking everything. But, we never added it to webmaster tools before.
    – Shurik Agulyansky
    Feb 18, 2019 at 20:58

1 Answer 1

Reset to default
2

The reason is that you recently updated your robots.txt file. It may take several months for Google to crawl all your pages again and realize that they are no longer blocked by robots.txt.

Even though you had never signed up for Google Search Console with your old site, Google was still collecting data about it. Now that you have signed up for GSC, Google is showing you everything that you would have seen had you signed up previously. Your new site data will be mixed in with your old site data.

You may see other remnants of your old site as well:

  • Links in the external links report that were created for your old site
  • Crawl errors from URLs that used to be on your old site
  • Search analytics data from before you signed up for Google Search Console
Improve this answer
3
  • 1
    I also had this problem. Took me 47 days to get it sorted in the Google Search Console. OP, if you are facing this issue in the GSC, do the fixes and give time to the crawl bot. That's all.
    – idk
    Feb 19, 2019 at 9:27
  • 1
    "...several months for Google to crawl all your pages again" - this also implies the URL existed on both the old site ("another WordPress installation") and the new one. (Assuming this URL does exist on the new site and it's not simply a hangover from the old site?!)
    – MrWhite
    Feb 19, 2019 at 11:48
  • If the page is not on the new site it would change from showing as "indexed, blocked by robots.txt" to "not indexed - 404 not found crawl error."
    – Stephen Ostermiller ♦
    Feb 19, 2019 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.