1

I have 1 own website and 1 CDN server. I have CSS, JS and images on my CDN server. All the CSS, JS and images on my own website come from the CDN server.

If I block my CDN server with a robots.txt file, will images on my own website but whose URLs are on the CDN be indexed? Also, does Google Bot ignore CSS and JS files?

3

The robots.txt standard is per domain/subdomain. So the file at cdn.example/robots.txt will control access to resources on cdn.example, and the file at yoursite.example/robots.txt will control access to resources on yoursite.example.

What this means is that for example if cdn.example/robots.txt has a User-agent: * ⏎ Disallow: / directive, all resources on cdn.example including images will be blocked. So in that case if you have a web page on yoursite.example that tries to load those CDN images as part of the page, those images will be blocked from being accessed by the crawler, so the crawler will render your page without the images.

Google bot uses CSS and JS files to render and rank your page properly. It will complain if you don't allow it access, and it may demote your site somewhat due to not being able to check it for mobile friendliness, page load time (part of "page experience"), etc. Definitely do not block CSS or JS files from being crawled by Google.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.