Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I got many soft404 in Google webmaster Tools, and those webpages aren't existing any more. thus I am unable to insert <meta name="robots" content="noindex, nofollow"> into my pages, and I've been searching a while but didn't get some valuable clues.

There are about 100 URLs are soft 404, to redirect them all one by one is a bit silly as it would cost too much time for me.

If i just add those links into robot.txt like below

User-agent: *
Disallow: /mysite.asp
Disallow: /mysite-more.html

if this way will fix all soft404s solidly? or if there is a way to change all soft404 to hard404? Please give me some suggestions. Many thanks

share|improve this question

1 Answer 1

up vote 1 down vote accepted

Wow. I think I got what you are saying, but there are some missing pieces so please bare with me.

Soft 404 errors are often something really stupid like having the phrase not found in your content. That drives me nuts! It happens to me all the time. It cannot occur unless Google sees a page and (likely) is getting a page with not found but no actual 404 response header. Something is interrupting the normal 404 error process. This we know because you tell us that the pages do not exist.

I am not sure how a non-existing page can have soft 404 unless it is being redirected or triggering a custom 404 error that is not working properly. If this is the case, then remove the redirect or custom 404 and let a real 404 happen. It may take take about 30 days or so for Google to stop looking for these pages. But it will clear up in time.

share|improve this answer
    
Thanks very much for taking time to answer my question. Cannot find any thing redirected. odd thing! If possible to use robot.txt to ask google to ignore / crawl specific pages? –  olo Jun 10 at 3:21
1  
Yes you can. I did not address that part of your question because it appears that there is something else wrong. You can enter each page one per line (as you know) or if there is a whole directory then you can specify that. Check your ErrorDocument directive (if you can) in your .htaccess or specific site configuration file under /etc/apache2/sites-available/. I am making the assumption you are using Apache. Any other configuration I would not know. You are making sure that it does not have something funky, wonky, googlie, whacky, or just plain weirded out. –  closetnoc Jun 10 at 3:31
    
Many thanks for your answer! I will try robot.txt –  olo Jun 10 at 3:56
    
Anytime! Keep an eye out for something that may be interfering with the normal 404 error process. That you can take some time on. The robots.txt thing will likely keep you busy enough for a while. –  closetnoc Jun 10 at 3:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.