3

Index coverage says "Indexed, though blocked by robots.txt". Those pages are no longer active and return 404 status. So that they don't get crawled and indexed, our SEO guys have been adding them to robots.txt.

Personally I believe we shouldn't do so because google will automatically drop "not found" pages from their index. There for we should allow bot to access the page instead of blocking via robots.txt

Please advise me if I'm wrong.

Improve this question
3
  • "no more active" - To clarify, you mean they "don't exist"? So return a 404? – MrWhite Sep 25 '18 at 7:43
  • @MrWhite they mentioned "Not Found" both in the question title and in the second paragraph. Yes they mean 404. I've edited the question for clarity. – Stephen Ostermiller♦ Sep 25 '18 at 10:24
  • @StephenOstermiller Yes, it would seem obvious. Except that the only answer (at the time) appears to have read the opposite; that these pages still exist, but are simply "not active". And the ambigious part in the question is the "no longer active" part. (Why would you write "no longer active" if the page doesn't exist?) We are assuming these pages return a 404. – MrWhite Sep 25 '18 at 10:33
1

Indexing is not prevented by blocking crawling:

  • Blocking an URL from crawling doesn't mean the url will not be indexed - it could get indexed and it will, especially if it has internal links. The SERP snippet would then be a non-descriptive "URL is blocked by robots".
  • Preventing both indexing AND crawling will never ever work. Only one of the two.
  • It is absolutely correct, that your 404 appears in index - because it was blocked from crawling.
  • In case of 404 it worth to noindex it rather to close with robots.
Improve this answer
1
  • 1
    There is no need to add a NoIndex tag to 404 pages. Google won't index them anyway. – Stephen Ostermiller♦ Sep 25 '18 at 12:56
0

It depends on what your goal is.

If you want the pages to appear in search result pages, but not get crawled anymore because they are not active anymore, disallowing them via robots.txt is the correct way to do it.

If you want the pages to not appear in search result pages at all, you should add a <meta name="robots" content="noindex"> to the <head> section of each individual page you want search engines to stop indexing.

What you are doing right now, blocking crawling via robots.txt, prevents crawlers from accessing the pages, but the pages will continue to appear in search result pages.

Improve this answer
2
  • 1
    This answer seems off to me. They state that these pages are no longer on their site and are "not found". They don't want those pages in search results. Google will automatically stop indexing 404 pages, no need to add the meta tag. Of your four paragraphs, only the last one makes much sense for the question. – Stephen Ostermiller♦ Sep 25 '18 at 10:21
  • At the time of my answer the question was not worded like that. – Σπύρος Γούλας Sep 25 '18 at 14:40
0

You don't need to add 404 Error pages in robots.txt file of your website. In first stage, those pages were indexed by Google search engine but somehow the links disappeared or you changed the URL permalinks. The pages are still existing in the cache memory of the search engine but the links are broken now. That's why when someone click on your page/post through search engine it redirects to 404 Error page. In that case, you simply "Request to Remove" Those URLs.

To do that, Open Google Search Console, open your related property, then go to:

Crawl

i) Crawl Errors

ii) Copy The 404 Page URL

Google Index

iii) Remove URLs

iv) Click on Temporarily Hide

v) Paste The URL and Continue

Now choose the first option "Temporarily hide page from search results and remove from cache" Then Click "Submit Request"

Within 24 hours this problem will be solved.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.