1

In order to provide a better experience for users we have begun flushing our head section of our website well before we have accumulated and rendered the HTML to be sent to a client. This has many benefits, but one large downside: For what would normally be a 500 or 400 response, we are currently returning a 200.

we're not particularly worried about the 400s, we currently just toss a no-index on them and call it good, but...

500s are a nasty problem. They are transitory, as opposed to what is probably a more permanent state in a 404, and as such we do want to index the page, but not the current version of the page.

I've scoured the internet and I cannot find anyone who has way to tell search engines "Not right now, please try later?" Any Ideas?

Improve this question
4
  • Yes, we should fix our 500s... They very rarely happen, but we have resource contention with new features and it may be time consuming to hunt down their causes.
    – The Honorable Samuel Clemens
    Jul 18, 2017 at 18:43
  • How does that help user experience? A user sitting there with just the head and no content isn't getting a better experience.
    – Stephen Ostermiller ♦
    Jul 18, 2017 at 18:47
  • On our site it takes between 250ms and 500ms to load the full content and begin delivering it to the user, however we can deliver style and JavaScript in under 15ms. That means the browser has up to half a second to do work to prep the page for the rest of the content.
    – The Honorable Samuel Clemens
    Jul 18, 2017 at 18:54
  • We ended up removing the initial response flush in order to correctly inform Google et al of issues via the status code.
    – The Honorable Samuel Clemens
    Jul 19, 2017 at 23:13

2 Answers 2

Reset to default
2

In order to provide a better experience for users we have begun flushing our head section of our website well before we have accumulated and rendered the HTML to be sent to a client. This has many benefits...

The only thing that might appear to the user's screen instantly is a new title in the window bar but not many people would pay attention to just that title. You might want to include some HTML in the initial flushing so users can see content right away. In fact, try to make the main HTML code file size as small as possible, so that you can flush everything at once. This means using basic javascript if your HTML contains lots of repetitive code blocks. Or if you don't want javascript, and you have lots of text that fills several pages, consider creating categories for your content then have one HTML file in each category.

... For what would normally be a 500 or 400 response, we are currently returning a 200.

You'll need to setup your server so that on each request, it successfully passes all the checks required to deliver the 200 response BEFORE any output is sent to the client (a.k.a. your website visitor's browser) because in the HTTP 1.x specification, the first line of output (10th character I think) is the actual status code delivered to the client. Yes I understand this increases the time to first byte somewhat, but delivering the truth is well worth it for all your guests (including the google robot, googlebot).

The first line in a typical HTTP 1.x request goes along the lines of:

HTTP/1.1 200 Success

Then after other HTTP headers can be used to optimize the page. Consider client caching.

After the HTTP headers are sent to the client, The HTML then follows.

Improve this answer
2
  • If we want to send any content we'd have to make requests to the long service calls that necessitated investigating flushing early. If we wait to send a status code until we know if retrieval is successful then we'd have already waited the 250ms-500ms for the content and we could just send it all down the pipe w/ the correct status code.
    – The Honorable Samuel Clemens
    Jul 19, 2017 at 23:26
  • That means you need to reconfigure the background processes. If possible, consider caching these "long service calls". On my website, I have a background process that caches images of large file size only once into ram so that subsequent requests to them result in a few milliseconds loading time instead of hundreds of milliseconds since requesting from a physical disk takes longer than from ram.
    – Mike -- No longer here
    Jul 21, 2017 at 5:34
0

Mike defines very important area of your question. Just want to add few lines with mike that is add a test version of your site to GSC. Use some traffic bot and see what error response you're getting in GSC. It is way to tell search engine your problem.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.