Soft error 404 and robots txt error:-

we have recently submitted sitemap xml to Search console, but we got a warning stating 131 of the page links are softerror 404 and for the home site it says Submitted URL blocked by robots.txt

enter image description here

1)We crossed checked our HTTP Header ,we noticed that header name Vary has only value of accept encoding,so we append and made it as http header name to Vary:Accept-encoding,User-agent

enter image description here

2)While checking with google search console ,robots.txt file is allowing for all the pages. But while we try to index .the error says as Submitted URL blocked by robots.txt for home page i.e .. https://www.3esofttech.com

3)When I fetch as google and render the page it via this link :https://technicalseo.com/tools/robots-txt/ no errors are displaying and robots.txt results is allowed

enter image description here

4)We run screaming frog spider crawl,and tested our whole website and the result was no page is getting blocked by robots.txt and re-indexed our site. However it was again more trouble that we got 131 pages soft404 error, and our home page was blocked by robots.txt

5)We thought may be it is cloudflare issue. We haven't created any WAF (web application firewall) So we tried to purge the cloudflare cache's of robots.txt URL - http://3esofttech.com/robots.txt and re-submitted sitemap.xml.but still same error is coming

6)We temporarily emptied the robots.txt file and submitted it in search console . We uploaded sitemap.xml again and the same warnings came even though no robots.txt file was there

We referred to this article URL - https://www.searchenginejournal.com/fix-404-vs-soft-404-errors/208046/

(all these 131 pages have unique content and no thin content or duplicate content for softerror 404 ) .but still Google search console says it is Softerror 404

Any help would be appreciated !!

  • Issue is resolved – Moin97 May 10 at 5:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.