2

I have a Nuxt app which utilizes a headless CMS and static site generation to generate the landing page and all article pages at build time. On my landing page I'm displaying the latest articles which can be statically generated with no problem, I simply have to rebuild the landing page everytime a new article gets added.

Now to view all available articles the user (and search engine bot) has to navigate to the "search" page which lists all articles by default (when no search query was entered). When a search query is entered or the search URL is opened with query parameters directly (e.g. https://domain.example/search?param=hello) a HTTP request is sent to the CMS and the listed articles get refreshed.

Now my question is, in order to make the search engine bot find and index every individual article, should I statically generate the default search page with the list of all articles and mark any other search URL which the bot might find (through internal or external links) as canonical to the default search page? Or should I use an entirely different approach?

Improve this question
3
  • Googlebot can't tell the difference between static content and server side generated dynamic content. There is no way to know just by viewing a page how the page was constructed or stored on the server.
    – Stephen Ostermiller ♦
    Dec 16, 2021 at 11:10
  • @StephenOstermiller I think you got that wrong. There is no SSR involved with my application, everything is either static or client side rendered. Currently the search result page is entirely client side meaning all results get fetched via AJAX. What I'm asking is if I should statically generate that AJAX request in order to make all articles available without any client side work.
    – DaDo
    Dec 17, 2021 at 15:30
  • 1
    Yeah, dynamic client side stuff could benefit from static prerendering. I updated your question and tags for clarity.
    – Stephen Ostermiller ♦
    Dec 17, 2021 at 15:37

1 Answer 1

Reset to default
3

I haven't seen a search page like this that doubles as a full article listing, but in general you should be preventing bots from crawling infinite spaces like search results by using robots.txt. In this case, you may consider letting bots crawl the default page, but not any queries:

User-agent: *
Disallow: /search? # this allows the base "/search" while disallowing "/search?[query]"

Whether to statically generate it is a different question. Google is very good nowadays at parsing JavaScript, but there are some caveats. See this Q&A.

Improve this answer
2
  • Is a robots.txt reliable enough to prevent unnecessary search query indexing? From Google Search docs: "Pages blocked in robots.txt won't be crawled, but still might be indexed if linked to by another page. Google can infer the content of the page by a link pointing to it, and index the page without parsing its contents."
    – DaDo
    Dec 21, 2021 at 17:03
  • 1
    @DaDo robots.txt is designed to tell bots what they can crawl, not to tell search engines what they can index. Google asks you to prevent them from crawling infinite spaces, but they don't care if you explicitly prevent indexing of URLs within them. Restricting what search engines can index is typically done with noindex, but is not necessary for the use case in your question. If you need to restrict what search engines are allowed to index, please ask a new question and link to this one if it's useful for context. Thanks!
    – Maximillian Laumeister ♦
    Dec 21, 2021 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.