I just discovered that there was a huge decrease in the number of pages that are blocked by robots in my search console. It is an online shop so i blocked many product pages and filters to avoid duplicateenter image description here content a while back. I wonder why this change has occured and I can't really nail it down.

Can this affect my SEO score? I've seen the AHREFS visibility index go down a little bit. What can I do to clean it up?


When Google says "blocked by robots.txt" they actually mean "URLs we have wanted to crawl recently, but couldn't because they were blocked by robots.txt".

The decrease could be because they already know about those URLs and know they shouldn't be crawled. The decrease could be because they are not discovering new URLs.

There is also a "Note" on the graph right when the decline starts to happen. Google puts in these notes when they change algorithms that can change how data on the graph looks going forward. It also may be that the algorithm change prevents them from wanting to crawl many of the pages you have blocked by robots.txt

| improve this answer | |

Google Search Console lags quite a bit in reporting. I have over 1000 errors that have been there for over 6 months but when I test the robots.txt in GSC everything is normal. Have you tried looking at the Index Status report? Do you see the same decline in indexed pages during the same time frame?

enter image description here

| improve this answer | |
  • Thanks for your quick reply! as you can see in the screenshot we made some changes to the structure of the site on 6/11/2016 after which the blocked pages declined gradually until today but we don't understand why this happens. The indexed pages seem fine... where can we see the index status report? – Mtm Project Feb 24 '17 at 17:31
  • @MtmProject I added the image for Index Status report. As far as to why...check your Index Status first. If everything there is ok then I would not worry about it. As I have said, GSC is highly inaccurate on many reports. – dasickle Feb 24 '17 at 19:01
  • thank you so much for your help. I'll keep this thread updated :) – Mtm Project Feb 24 '17 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.