3

Here is my issue. I have a rewrite rule in my htaccess like this:

RewriteRule ^(news|review|feature|editorial|podcast)/([0-9]+)/(.*)$ /article.php?id=$2 [L]

Basically what it does is take in a URL like this:

http://www.example.com/feature/13889/negative-world-2017-goty-awards

And bring the user to this page:

http://www.example.com/article.php?id=13889

However, within the last year or so Google seemed to decide it doesn't like this anymore, and will return soft 404 errors when it crawls any of those original URLs. ("The target URL doesn't exist, but your server is not returning a 404 (file not found) error.")

Whether because of this or something else, a lot of my content is no longer indexed on Google, which led to a massive drop in hits for my site.

While reading up on this, I read something that said 301 redirects would solve this. So I switched the line to this in htaccess:

RewriteRule ^(news|review|feature|editorial|podcast)/([0-9]+)/(.*)$ /article.php?id=$2 [L,R=301]

Should that solve the issue, or is there more that I have to do?! I'm a little worried because when I do "fetch as Google" it just says "redirected".

I guess my other question would be, would this explain why this stuff isn't being indexed anymore, or is there probably some other issue I should be looking into?

1

I can't see how your original internal rewrite would have anything to do with the soft-404 reported by Google. Google is not aware of any rewrite - this is entirely hidden from users, bots and Google.

And changing this to a 301 redirect is only going to make matters a whole lot worse I suspect. Presumably, you are linking to the "pretty" URL (ie. /feature/13889/negative-world-2017-goty-awards)?

A soft-404 is generally reported when Google is served a page that looks like a "Not Found" error, or the page is very thin on content, and which responds with a 200 OK status. From the Google help docs on Soft 404 errors:

A soft 404 means that a URL on your site returns a page telling the user that the page does not exist and also a 200-level (success) code to the browser. (In some cases, instead of a "not found" page, it might be a page with little or no usable content--for example, a sparsely populated or empty page.)

So, whether you serve the page from the "pretty" URL /feature/13889/negative-world-2017-goty-awards or the "ugly" URL /article.php?id=13889 (via a 301 redirect or direct link) then it's still the same page and so still the same soft-404.

would this explain why this stuff isn't being indexed anymore

Google is unlikely going to index a page that it considers to be a (soft) 404.

I'm a little worried because when I do "fetch as Google" it just says "redirected".

I would remove the redirect and redo the "fetch as Google" check. What is Google seeing? Maybe it's getting a different response than what users are seeing?

  • 2
    Welp, I think you're right. I followed the fetch as Google through, and the pages were actually being redirected to my own error page. So I looked into the code and found a pretty nasty bug that basically redirects anyone who tries to access articles but is not logged in to an error page (they shouldn't be redirected at all, articles are public.) Must have been added a year or so ago and NO ONE NOTICED SOMEHOW. Anyway I switched it all back, fixed the bug, and fetch as Google is working now so... we'll see if I can salvage my rankings at this point. Thanks! – Andrew Whatever Jan 28 '18 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.