Will Googlebot click a button/link to reveal AJAX content?

Question

my client has to inform it's customers about some new regulations that the Googlebot should NOT crawl. It is not possible to place this information on a separate page and disallow Google to crawl it. So the idea is to place a button/link on the page, that will AJAX-load the corresponding information only when the user clicks it. My assumption is, that Google is unable to click the link and crawl that specific AJAX content.

Am I right? And if yes, is there an official documentation that proofs my point on this?

Answer 1

Googlebot does not interact with the page like a user. It does not click on anything. It does not scroll. If content is loaded into the page when users click, Google is not going to index it as part of that page.

However, Googlebot still may find AJAX content to which users have to click. That is because Googlebot scans the page source, the rendered document object model, and loaded JavaScript files for things that look like links. Googlebot is likely to discover the URL for the AJAX that way and then Googlebot may crawl it.

The only reliable way to keep Googlebot from crawling something is to use robots.txt. You could put the AJAX URL itself into robots.txt. Googlebot would still be allowed to crawl the page, but would not be allowed to load the AJAX. So you could use Disallow: /regulations.json in robots.txt but still have Googlebot crawl and index /mypage.html which calls regulations.json via AJAX.

You can prevent Googlebot from seeing portions of the page like this. It doesn't matter if the content is loaded via AJAX on click, or on document load. If Googlebot isn't allowed to crawl the AJAX URL, Google won't see the content. See Preventing robots from crawling specific part of a page for full details.