2

I want to remove multiple URL in my website (developed in ASP MVC). All these URL have the format www.example.com/planning/xxx

I have already changed the code to make these URLs 301 redirect. I expact that would indicate to robots to remove the page of their index.

Now, when I check the Google Search Console, I see many soft 404s which corresponds to these URLs.

Why did Google consider them as soft 404 whereas I do a permanent redirection so with the 301 status? Are these URL really removed from the Google index? (I don't think because many of them are still found in the Google results.) Did a better way exist to remove multiple URLs? Will modifying robots.txt with to add Disallow: /planning be useful or will it only prevent Google from indexing new URLs without removing the existing?

Thanks

Improve this question
2
  • 404 doesn't imply that your site is broken, it implies that the page no longer exists. If you didn't redirect when Google did a few crawls then it could take several weeks for the system to catch up. Also, you should investigate to ensure the header status is returning 301 and not 404... you can do this using CURL. If it does return a 301 then there's nothing to worry about and you should ignore WMT, just wait and have patience ~ WMT is weeks behind on most things.
    – Simon Hayter
    Oct 26, 2015 at 21:16
  • Your first mistake was turning what should have been a 404 into a 301. That keeps the URLs alive in the index. Allowing a 404 to happen will remove the URLs from the index. Your doing a 301 to another page, there are likely clues on that page that indicates a soft-404 such as page not found, not found, ooopppsss!!, etc. While soft-404s are good for recovery of user attention, generally, I agree with Google that the web is a bit better without them. A custom 404 is a whole other animal.
    – closetnoc
    Oct 26, 2015 at 22:40

1 Answer 1

Reset to default
1

I want to remove multiple URL in my website (developped in ASP MVC) All these URL have the format www.mysite.com/planning/xxx I have already adapted the code to make 301 redirect and indicate to robots to remove the page of their index...

A 301 is a mistake. You actually teased the robots because you made them think that good content to index is found at the URL defined by the location HTTP header (if you provided one)

Now, when I check the WebMasterTools, I see many soft 404 which corresponds to these URLs.

This is the robot's way of saying you teased them. You issued a redirect to URL which the robots think is an error page. (even though the HTTP status returned is 200).

So, I have some questions : Why did google consider them as soft 404 whereas I do a permanent redirection so with the 301 status ?

A redirection means change to a new URL. It's the resulting URL that matters, and in your case, its a URL resulting in a soft 404.

Soft 404's are pages having text that robots believe are actual errors, but they're "Soft" because the HTTP header doesn't return a 404 status. it instead returns a 200 (success) status.

Are these URL really removed in google index (I don't think because many of them are always find by google) ? Did it exist another way (a better way :) ) to remove multiple URLs ? May adapting the robots.txt with a Disallow: /planning be useful or it only prevents to index without removing from existing index ?

The best way to remove the URLs is to first make a list of all the URLs that google complains about in Webmaster tools that you never plan to turn into actual pages, then make a list of all the URLs that contain code to redirect to error pages.

Then when someone asks for a URL on the list, return a page with the 410 error code which represents GONE. This means the first line in the http headers must begin with HTTP/x.x 410 (where x.x is the version you use which is probably 1.1).

I could give you apache and PHP code if you need it, but I'm not sure if that works with your server setup.

Improve this answer
5
  • Or they can just let them 404 naturally and slowly. ;-)
    – closetnoc
    Oct 27, 2015 at 3:55
  • True but I'm not sure the OP wanted to go that route tho or he wouldn't have asked the question.
    – Mike -- No longer here
    Oct 27, 2015 at 4:01
  • @closetnoc: I have had URLs that have been 404ing for over a decade, and Google Webmaster Tools still shows them and without a referring URL.
    – winwaed
    Mar 13, 2018 at 17:37
  • @winwaed That is likely because there are links to that page. Alternatively, you can serve 410 instead.
    – closetnoc
    Mar 13, 2018 at 19:09
  • By "without a referring URL" I meant Google wasn't listing any - usually it lists the pages (or I assume a subset if a lot) that refer to it. Good way to find broken links.
    – winwaed
    Mar 14, 2018 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.