Will bots crawl my page generated by javascript

Question

I use simple javascript to generate pages based on thousands of products stored in a JSON object. The page URLs will be like /productList?keyword=XXX&page=x. The keyowrd is what uses input through the search bar. Single product pages would be /product?id=XXXX. For both of the product list pages and product pages, the product info is retrieved from the JSON API using javascript.

My questions are:

1. There will be countless combinations of keyword and page, how would search engine bots crawl and index them?
2. If users search a product's name, what result would search engine give him? My concern is the products won't be searchable. But I read an article saying major search engines are able to execute javascript. So in my case, would search engine recognize my pages and products?

Answer 1

1. Search engines do not like to index so-called "infinite spaces". You should use robots.txt to block search engines from crawling your internal search results pages. You can block them by using a robots.txt directive like this:

   /productList?keyword=
   

   or this, if appropriate:

   /productList?
   

   Make sure you let Google index (don't block) your single product pages, at:

    /product?id=XXXX
   

2. You should test whether Googlebot can properly execute your JavaScript. To do this, use the inspect URL feature of Google Search Console. You'll want to choose "Test Live URL" when you get the option to, then select View tested page. Scroll through the HTML view, which shows the final HTML that Googlebot was able to render from your JavaScript, and along with the screenshot you will be able to get an idea for what Googlebot is able to see. Bing may have a similar tool, I am not sure.

Answer 2

Search engines do not like "search result pages" in their search results. They tend to push better pages out of the top 10 results, and create a bad user experience overall, since the user is NOT taken to the item they want, but must now scroll down through another search results page to find the item they want.

As such, it is best to exclude search result pages from search engines altogether.

The search result page should have

 <meta name="robots" content="noindex,nofollow">

And the robots.txt should be set as Maximilian describes above, for all crawlers (*), not just googlebot.

The latter will prevent crawling. The former will prevent the page from appearing in search results. Not quite the same thing; a robots.txt blocked page with good link relevance will get a "shadow page" created for it in search results based on that linktext data. If MLB.com/yankees blocked crawling with robots.txt, they would still be #1 for "yankees". So sometimes you have to let the search engine crawl the page so it can see the "meta noindex".