1

My site lists blogs like this

example.com/?status=blog&id=number

I only have 3 blogs now (id=1, id=2 and id=3). I checked my log files and saw that search bots are manipulating the URLs and crawling. Like

example.com/?status=blog&id=4

The above URL just shows a message "Not found".

What is the best practice from an SEO perspective to handle this?

Options I am considering in order of preference:

  1. Use a 404 status with a custom page
  2. 302 to latest blog available (eg id=3 in this case)
  3. Just keep showing the "Not found" message.
  4. Do nothing.

Please note that I may write another blog and id=4 may be available then. I am open to any other suggestions as well.

  • What would "Do nothing" mean? – Stephen Ostermiller Jan 30 '18 at 18:44
  • @StephenOstermiller Don't do anything. assuming bots will eventually figure it out themselves that these are pages with no content. – nandu kk Jan 30 '18 at 18:53
  • So the current behavior is to return "200 OK" status and a page with only "Not found" for content? – Stephen Ostermiller Jan 30 '18 at 19:07
  • @StephenOstermiller The blog contents are dynamically fetched from db based on the id. If no blog is found for the id then just "Not found is" displayed. So yes. – nandu kk Jan 30 '18 at 19:16
  • So, #3 and #4 are really the same? Are you finding that many URLs of this form a being crawled? eg. many URLs like /?status=blog&id=4&article=123 or id=5, id=6, etc. etc.? – DocRoot Jan 31 '18 at 11:20
1

My only concern would be how bots are finding or manipulating the URLs to find the not available URLs. To me, from an SEO perspective this is a crawling issue and potentially has an impact in performance and perceived website quality.

Had this issue been solved then we would not be having to take these considerations. If applicable, consider trying to fix the way bots are instructed to crawl your website and the way you are fetching the content from the database to avoid generating automated content or blogs without content.

After checking your code to understand how a bot is able to request such URLs, Try adding something like the following Allow rules before any Disallow to your robots.txt file:

.

.

Allow: /?status=blog&id=1/

Allow: /?status=blog&id=2/

Allow: /?status=blog&id=3/

Disallow: /*&id*

In the development stage of any website I always recommend people to retrieve some “coming soon” content, set up Google search console and even analytics, before publishing a new blog/site. This is to allow crawling and indexing the new website which is good for SEO purposes.

If for some reason you can not resolve the crawling issue, I would suggest doing nothing of the above solutions. By taking the first choice, you will potentially generate an almost infinite number of 404 errors pages and this, from the SEO perspective it is a sign of poor maintenance, bad UX and quality. 302 might be a good alternative but it will consume a lot of resources from you and from the bots’ servers (we don’t want them to get mad at us, remember Skynet). Since this is probably an auto generated content/URL I will suggest implementing noindex, nofollow meta tag approach and retrieving http 200 status response code. You can also add noindex, nofollow to the links pointing to the blogs. Well, you might say, “wait a minute, 200 ok header responses are still consuming resources”, true, but believe 3xx responses are more expensive.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.