0

If we use static site generators (SSG) and edit some block duplicated on many pages e.g. header or footer, we need to regenerate all the HTML pages using this block. Regardless of the speed of our SSG, the process takes a time. And the more pages we have, the more time we need to wait.

But what if we store the code of duplicated block e.g. header or footer only in one place and load their HTML codes to the pages on the fly by JavaScript?

I mean if the source code of HTML page just has unchanged <head> with link(s) to CSS and <body> with unique main content for the page like

<!DOCTYPE html>
<head>
<meta charset="utf-8">
<title>Page Title</title>
<link rel="stylesheet" type="text/css" href="styles.css">
</head>
<body>

  <header></header>
  <script>
  var headerReq = new XMLHttpRequest();
  headerReq.open('GET', 'header.html', false);
  headerReq.send();
  document.getElementsByTagName('header')[0].innerHTML = headerReq.responseText;
  </script>

  <article>
  <h1>Page Heading</h1>
  <p>
  Main content
  </p>
  </article>

  <footer></footer>
  <script>
  var footerReq = new XMLHttpRequest();
  footerReq.open('GET', 'footer.html', false);
  footerReq.send();
  document.getElementsByTagName('footer')[0].innerHTML = footerReq.responseText;
  </script>

</body>
</html>

And we load header and footer (as well as other blocks duplicated on many pages) to the DOM by our JS as codes before and after the main content.

If we change the main content, we may only regenerate the single post page with the content and the category page containing the intro text of the post i.e. regenerate just two HTML pages.

And if we change header or footer, there is no need to regenerate HTML pages at all. We just change such block in one place and JS refreshes it on all the pages when we open any of them.

But will there be the issue of mismatching the source code (we see after right-click and "View page source") and the final DOM code (we see after right-click and "Inspect Element")? The issue described by the link https://www.polemicdigital.com/view-source-quickly-compare-raw-html-rendered-dom

To avoid the issue in our JS we can check user agent and load the header and the footer only for users and not for the bots.

What are the downside(s) of the approach?

ps. I know, in such case if we inline critical CSS and later want to change it, we still need to regenerate all the pages with such critical CSS. And with HTTP/2 there is probably no need in critical CSS at all.


UPDATED

The main question is: to load duplicated parts for both users and search bots OR only for users?

If we load for both, there are at least two issues: 1) the SEO issue of mismatching the source code and the final DOM code and 2) the SEO issue of a delay as search bots need more time to render and index a page with code loaded by JS than a page with only normal source code.

If we load duplicated parts only for users and NOT for the bots, there may be the issue of cloaking or/and hidden texts. *However, I can't understand why as the bots don't see the duplicated parts and can't compare the code they get with the code users get.

2
  • How many pages are on your site? I used a static site generator for a site until it grew to about 30,000 pages at which point that got unwieldy. But for me even tens of thousands of pages only took 5 minutes to generate. Oct 24, 2018 at 14:27
  • It's not about a specific website, it's a general question
    – stckvrw
    Oct 24, 2018 at 15:01

1 Answer 1

1

will there be the issue of mismatching the source code

I think, for sure. Before rendering there will be just a javascript, after rendering there will be javascript + new content, pulled by javascript. But this shouldn't cause any negative effect.

and JS refreshes it on all the pages when we open any of them

I suspect here some cache issues - not critical, but think about it.

render-blocking issue

If you don't block javascript by robots, there shouldn't be any.

In general, this is not a kind of new or special approach. It isn't critical for indexing too. You should just think about some things in advance and there will be nothing critical. I.e. Googlebot is waiting about 6 seconds after onLoad for arriving of new content. Make sure, your javascript is fast enough and you are safe.

7
  • As I mentioned, I would like to load header and footer by JS only for live users and not for the bots. So the bots will not see header and footer at all. I don't see a reason to index them as they don't have valuable contents. The home page may have header including nav menu with links to inner pages and footer in its source code. But if we create inner pages, they just have its main contents. And if we open any of them, JS gets the header and the footer from the index.html and load them to the DOM of the inner page. As I can understand, it also resolves the cache issues(?).
    – stckvrw
    Oct 25, 2018 at 10:05
  • 1
    and not for the bots - doing so you are at risc to be penalized for cloacking. How do you want to decide, who is the bot, and who is a human? Deciding on user agent and delivering different content is against Google's ToS. Don't risc it - deliver everything for all, but use for non-meaningful parts a kind of lazy loading.
    – Evgeniy
    Oct 25, 2018 at 14:08
  • As I know, it's normal to prevent a piece of code be indexed by search engines. The cloaking is "presenting different content... to human users and search engines". But we're talking not about content, but about header and footer. And what do you mean under the lazy loading?
    – stckvrw
    Oct 25, 2018 at 22:03
  • to prevent a piece of code be indexed by search engines - afaik there is no legit technics to do so, beside of loading or not loading, depending on user agent. What you name header+footer is for bots content. Whether it is chrome, is a decision, done by heuristics after crawling. But ok, lets suppose, we deliver different chrome for different mobile agents - then it is ok to deliver different page versions. With lazy loading i mean a kind of defering, i.e. you load footer only if user scrolls down to the end of page.
    – Evgeniy
    Oct 26, 2018 at 8:05
  • there is no legit technics to do so by the link there is a mention about iframe and other references. Do you mean Google views such non-crawlable parts of page including iframes as hidden texts and/or cloaking?
    – stckvrw
    Oct 26, 2018 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.