1

Might crawlers visit a page if there is no link referencing this page anywhere but a URL to this page is generated client side with JavaScript ?

Context:

Let's say I have a SPA with Server Side Rendering. Some pages show a list of items and offer a filtering facility. When the user selects some options or fills in some input field to filter the list, I'd like to embed this information in the URL (eg. /items?sort=price&order=desc&q=something) via the history API (client side routing). Behind the scene, an API call is made to get the results.

Since I do SSR, the server will also be able to understand these URLs and render these pages (hence the user can bookmark the page or share it). But nowhere in the HTML pages these URLs will appear, there are only generated client side in response to user events.

In this context, I think crawlers won't know these pages exist, and so, they should have no impact on SEO. Even if crawlers are now able to run JavaScript, they don't use it to simulate user events.

Am I wrong ?

(I guess if someone shares publicly that kind of URL, it could suffice to make this page crawled ? In any case, what I'm worried about is the cost on the crawling budget if all these pages are visited, but I'm ok with a few pages being crawled, they could be marked as "noindex" for instance).

Improve this question

2 Answers 2

Reset to default
2
+100

You don't need to worry about crawl budget. Your site should have plenty of crawl budget even if Googlebot does start crawling those pages. Pages that have just one way to get to them are going to be very low priority. Google would still crawl the main pages of your site well and far more often.

Googlebot will crawl pages that you don't link to in some cases. see How can I figure out how a search engine is finding hidden pages?

  • Googlebot can process JavaScript and it has heuristics to pick things that look like URLs out of JavaScript code. If you have JS code like var url = "/items?sort=price&order=desc&q=something" that would trigger Google to crawl that page. However if the URL is built from smaller pieces, Googlebot shouldn't try to piece together the URL.
  • Googlebot executes JavaScript that happens on page load. If that JavaScript creates HTML <a hrefs in the DOM, Googlebot will crawl those URLs. It is worth noting that Googlebot doesn't simulate user actions such as scrolling or clicking when it executes JavaScript. Googlebot only executes code that runs when the page loads and then analyzes the output of that.
  • Googlbot will try to fill in forms by picking and choosing combinations of the various inputs. Google announced that here. Google says that they limit this type of crawling to "a small amount" per site.
  • If your pages have external links, the pages you link to will get notified of your URL in the HTTP Referrer header. Many sites publish inbound links with features like WordPress trackbacks or analytics data that isn't password protected. Googlebot often finds out about pages through their external links because of this.
  • Users may create links or bookmarks to pages that Google finds (as you already noted.)

Google could penalize your site if you allow site search to be crawled. Your q=something parameter is worrying. Google doesn't want to index anything on your site that could look like search results because it is bad user experience for a user to click from Google search results onto other search results. See https://www.mattcutts.com/blog/search-results-in-search-results/

Having lots of crawlable pages based on permutations of parameters can lead to lots of duplicate content and low quality pages. It is usually best to ensure that crawlers see content at only one preferred URL. See What is duplicate content and how can I avoid being penalized for it on my site?

I would recommend disallowing these URLs in robots.txt. Something like this would work. It would allow crawling of /items but disallow crawling of the same URL with parameters.

User-agent: *
Disallow: /items?

If you want to disallow Googlebot from URLs with just a specific parameter (like q=) you could use the following. It useds wildcard rules in Disallow: directives which Googlebot understands, but most robots do not.

User-agent: Googlebot
Disallow: /items?*&q=
Disallow: /items?q=

When pages are blocked by robots.txt, Googlebot won't crawl them, but Google still may show them in the search results based on external links. See Why do Google search results include pages disallowed in robots.txt?

Improve this answer
2

I don't see any reason for this to impact the SEO of your site. Google won't know what to fill in and I doubt seriously that Google would try each of the possible combinations of your dropdowns/radio buttons or similar options. Based on your description, I'm also assuming the Javascript you are concerned about triggers an onClick event that executes your JavaScript?

Google has some good documentation as to how they process JavaScript applications here. https://developers.google.com/search/docs/guides/javascript-seo-basics

Per this document,

When Googlebot fetches a URL from the crawling queue by making an HTTP request it first checks if you allow crawling. Googlebot reads the robots.txt file. If it marks the URL as disallowed, then Googlebot skips making an HTTP request to this URL and skips the URL.

So if your major concern is indexing pages your don't want crawled, per Google, set them to disallowed in your robots.txt file.

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.