I use simple javascript to generate pages based on thousands of products stored in a JSON object. The page URLs will be like /productList?keyword=XXX&page=x. The keyowrd is what uses input through the search bar. Single product pages would be /product?id=XXXX. For both of the product list pages and product pages, the product info is retrieved from the JSON API using javascript.

My questions are:

  1. There will be countless combinations of keyword and page, how would search engine bots crawl and index them?
  2. If users search a product's name, what result would search engine give him? My concern is the products won't be searchable. But I read an article saying major search engines are able to execute javascript. So in my case, would search engine recognize my pages and products?

2 Answers 2

  1. Search engines do not like to index so-called "infinite spaces". You should use robots.txt to block search engines from crawling your internal search results pages. You can block them by using a robots.txt directive like this:


    or this, if appropriate:


    Make sure you let Google index (don't block) your single product pages, at:

  2. You should test whether Googlebot can properly execute your JavaScript. To do this, use the inspect URL feature of Google Search Console. You'll want to choose "Test Live URL" when you get the option to, then select View tested page. Scroll through the HTML view, which shows the final HTML that Googlebot was able to render from your JavaScript, and along with the screenshot you will be able to get an idea for what Googlebot is able to see. Bing may have a similar tool, I am not sure.

  • Thanks for the advice. I am still not very clear about how I can make my products searchable in my case?
    – Wen Shenk
    Oct 14, 2021 at 14:05
  • You'll want to make each product page accessible by exactly one canonical URL. Let Google index the product pages (finite), block it from indexing the internal search result pages (infinite). Oct 14, 2021 at 16:35
  • Thanks. Having thought about this, I set single product's pages as ` /product?id=XXXX`. Is this good?
    – Wen Shenk
    Oct 14, 2021 at 16:40
  • @WenShenk That'll work! Oct 14, 2021 at 16:43
  • Today I tested my product list page, product page and also normal page( not built with javascipt) using Search Console's test live url function. It seems Google bot won't load all of the js files and every time what JS files will be loaded is different: webmasters.stackexchange.com/questions/137607/… This surely is a big headache for me. Any advice?
    – Wen Shenk
    Nov 22, 2021 at 18:05

Search engines do not like "search result pages" in their search results. They tend to push better pages out of the top 10 results, and create a bad user experience overall, since the user is NOT taken to the item they want, but must now scroll down through another search results page to find the item they want.

As such, it is best to exclude search result pages from search engines altogether.

The search result page should have

 <meta name="robots" content="noindex,nofollow">

And the robots.txt should be set as Maximilian describes above, for all crawlers (*), not just googlebot.

The latter will prevent crawling. The former will prevent the page from appearing in search results. Not quite the same thing; a robots.txt blocked page with good link relevance will get a "shadow page" created for it in search results based on that linktext data. If MLB.com/yankees blocked crawling with robots.txt, they would still be #1 for "yankees". So sometimes you have to let the search engine crawl the page so it can see the "meta noindex".

  • Thanks for the advice. I am still not very clear about how I can make my products searchable in my case?
    – Wen Shenk
    Oct 14, 2021 at 14:05
  • @wen The search result page should have <meta name="robots" content="noindex,nofollow"> Oct 14, 2021 at 17:54
  • @ReinstateMonica I'm totally confused here. Isn't noindex,nofollow telling search engine to skip this page?
    – Wen Shenk
    Oct 14, 2021 at 20:58
  • 1
    @WenShenk i am saying do that for your search result pages, not your product pages. And actually "nofollow" is not needed. Oct 14, 2021 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.