8

With the rise of HTML Streaming (e.g. React 18 Streaming), I'm wondering how crawlers handle HTML streams.

Most importantly, do crawlers wait until the stream has ended?

How does the Google crawler handle this? Google does mention streaming here but it doesn't say how the Google crawler handles HTML streams.

How about other crawlers (DuckDuckGo, Bing, Facebook, Twitter, etc.)? I'd assume that crawlers would wait until the HTML streams end, am I safe with that assumption?

HTML streaming could lead to SEO optimized pages with a fast TTFB. That's a big deal.

Improve this question
1

1 Answer 1

Reset to default
9
+50

Currently, as of 2021, Streaming SSR combined with partial hydration is not fully materialized. So the exact impact is unknown. But in general, streaming HTML is not a new thing. It has been there since a long time. Streaming HTML is just being implemented with modern SSR using React, Solid.js, etc.

As far as the search engines are concerned, the reduction in TTFB definitely adds to improving the page/content ranking but it won't impact the ability to properly crawl the web page. It would behave like any other crawling. In fact, modern crawlers are smart enough to even crawl the CSR HTML.

Streaming is about getting that extra juice out of the door. Also, you are right that crawlers wait until the HTML stream ends with sensible timeouts. Long running stream would penalize the ranking.

Improve this answer
2
  • 2
    That makes a lot of sense. While I guess dynamic rendering (i.e. cloaking) to achieve the same result in regards to SEO today, I can see Google to eventually rank a fast TTFB with a long running stream significantly better than a slow non-streamed HTTP result - maybe that's actually already the case? Thanks for your wonderful answer.
    – brillout
    Jul 27, 2021 at 17:30
  • 1
    Actually, after reading the discussion at React 18 New Suspense SSR Architecture - SEO / Status Codes, I'm not sure whether crawlers grab the content of only the very first stream flush?
    – brillout
    Jul 29, 2021 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.