-3

Hi in my Drupal website site I have a few directories like:

/sites/modules/

Now I only want googlebot or other bots to index or allow say file user.css

I don't want Googlebot to index the file lightbox.css in the same directory.

How do I edit robots.txt to achieve the above? There would be other files in the same directory.

  • 1
    Search engines do not index CSS files. – closetnoc Aug 7 '17 at 4:34
  • Furthermore Google has requested that it be allowed to crawl all your CSS and JS so that it can evaluate your site's layout: seochat.com/c/a/google-optimization-help/… Preventing Googlebot from crawling some CSS may cause Google to assume your site is not well laid out and cause your rankings to get worse. – Stephen Ostermiller Aug 7 '17 at 8:45
  • I'm voting to close this question as off-topic because it discusses a problem which does not occur and cannot be reproduced. – Rob Aug 7 '17 at 10:06
1

Now I only want googlebot or other bots to index or allow say file user.css

a CSS file? it doesn't offer value to an end user.

I don't want Googlebot to index the file lightbox.css in the same directory. How do I edit robots.txt to achieve the above?

Do nothing.

Search engines are only interested in high-quality content which means text. Such text is generally located in html files (or even processed by accessing script files such as php files).

I'm assuming the CSS files are required to make your page function properly. These need to be accessible to the public so that search engines can access the files and render the pages correctly to better understand them. These days, page formatting is often done in CSS and if the CSS file isn't accessible then the page will look almost like a mess at best.

To prove my theory, register an account with Google search console and use the fetch as google tool to try to load the HTML file that requires the CSS files that you are "trying to not make google index" and you can see that if the files aren't accessible then the page can't load as intended.

-2

You can use following to disallow in robots.txt:

disallow: */lightbox.css
  • While this looks like it might work, such a short answer is not high quality. It would be better to explain what it does and link to documentation. In this case you are using a wildcard. You should explain how wildcards are special in robots.txt and which robots can use them. – Stephen Ostermiller Aug 7 '17 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.