Does blocking api URL hurt rendering by googlebot

Question

In my GSC crawl stats most of the 404s and URLs using page resource load are:

1. API URLs for a JavaScript part of our content
2. Cloudflare's managed challenge

If I add an entry to robots.txt for 1, will it break how Google renders portions of our site?

And for two, will it break anything if I stop Googlebot crawl requests?

Answer 1

Robots.txt file will not stop a page from being indexed (or attempted to be indexed) if

1. You have internal links to those pages
2. There are external links to those pages.

Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results.

If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.

From Introduction to robots.txt

Google is already getting a 404, so putting it in robots.txt is not going to make it worse. Equally, it probably will not stop the errors.

I would suggest that the fix is

• to remove links to them from other pages
• or let them return something innocuous
• or use a meta tag of noindex on those calls, see below

noindex is a rule set with either a <meta> tag or HTTP response header and is used to prevent indexing content by search engines that support the noindex rule, such as Google. When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.

Important: For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.

From Block Search indexing with noindex