3

Three weeks ago I add this in my robots.txt to disallow pagination so it will not appear in the Google index:

User-Agent: *
Disallow: /blog/1
Disallow: /blog/2
Disallow: /blog/3
Disallow: /blog/4
Disallow: /blog/5
Disallow: /blog/6
Disallow: /blog/7
Disallow: /blog/8
Disallow: /blog/9

When Google was allowed to crawl these pagination pages, they gave result like this when I searched a keyword in my blog:

pagination got crawled and indexed (not the article)

Is my robots.txt line above I intend to disallow only the pagination, but not the articles. The crawler should be able to still index the article.

Today Ahrefs reported around 173K uncrawled pages. Is it caused by my new robots.txt disallows? I suspect it is either that or because after migration there things still need time to settle.

There is this message too:

The website wasn't fully crawled

The crawl has reached the maximum depth level from the seed, and the website has not been crawled completely.

To crawl deeper levels of your site, increase the "Max depth level from seed" in the project settings and start a new crawl.

You might also want to check why some of the URLs on your website are considerably distant from the seed.

Can someone explain it to me about this? I'm confused since it is the first time I've stumbled upon this problem.

0
4

Robots.txt Misinformation

There is a common misconception that disallowing URLs in a robots.txt record is a good method to de-index content.

This is misguided. It is not your fault for thinking to do this, it is an extremely common tactic.

I am hired to recover websites from exactly this at least once every three months.

The proper way to keep new content out of Google's index, or remove existing content from it, is via meta robots no-index:

<meta name="robots" content="noindex">

OR

Password protect the pages you are working on.

To prevent further issues, please revert your robots.txt disallows ASAP and apply robots meta no-index tags instead.

Whether or not they are ranking on the 1st page or the 100th page, most of your pages are likely receiving impressions for some search queries. When Google crawls your website, most often it is to check existing content for changes, not look for new content. For example, in Search Console under the sitemaps tab that one of your sitemaps was "last read" today, but a piece of content you published a few days ago was not included.

When content is all of a sudden is blocked from Google (they respect robots.txt) they make assumptions.

Saying you can't see this content vs please don't index this content send different signals.

To Fix Your "Max Depth" Issue

In Ahrefs go to your site audit settings and increase the maximum crawl depth that is applied. The default should be something like 5 if I am not mistaken.

This means that your link depth exceeds 5 directories like so:

https://example.com/dir-1/dir-2/dir-3/dir-4/dir-5/

In any case, if you are experiencing this it concerns me a bit. That is highly excessive. Pages past dir-3 are likely starved for PageRank. We want to keep your URL structure as clean and concise as possible.

Please read up on internal linking and website hierarchy best practices.

I really like how Brian approaches internal linking. That article is better than most. The Databox article on structure is pretty new (haven't read it in full nor do I know of the author) but it looks very legitimate.

Identifying the Root Cause of You Problem

The reason that you might attempt to remove pages from Google's index is likely because you believe one of the following is hurting your SEO:

  • Index bloat
  • Crawl budget being wasted
  • Keyword cannibalization

Quality over quantity...Always

7
  • 1
    Another root cause could be content duplication with similar URLs showing the same content. Aug 10 at 8:43
  • 1
    If Google can't crawl it, Google USUALLY won't index it. It is fairly rare for Google to index pages blocked by robots.txt and show them in search results other than a site: search. They generally have to have numerous external links for that to happen. Aug 10 at 8:45
  • @Mike Ciffone thankyou for the answer, it really explaining what I want to know about this problem and its very thorough when explaining about my problem, thankyou, I'll try to apply it because as you said There is a lot of misconception like what I did where I think deindex is better than using <meta name="robots" content="noindex"> as for the page problem when I search specific keyword in Google, is it better to do optimization instead no index in the blog page it seems Aug 10 at 10:31
  • @StephenOstermiller so it really a rare case when Google index a page that already being dissalow from robots.txt, I thought my robots.txt script is failed but when I tried in google supports system for robots.txt simulator it run normally Aug 10 at 10:33
  • 1
    Mike's has good advice about preferring noindex over disallow when you don't want stuff indexed. In your case you want to remove things from the index. When removing noindex is a lot faster than robots.txt (weeks rather than months.) Aug 10 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.