0

This will be my first website, so I am completely new to all of this.

I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden files like /.htaccess?

I have read elsewhere on ProWebmasters that we should not disallow: /*.js$ or /*.css$, but those answers were almost a decade old!

And what about stuff like /includes or /includes/connect_to_database.php?

Even Google's own document regarding robots.txt do not seem to cover this stuff. Is anybody aware of informative resources somewhere on the web relating to this stuff?

Thanks

1

Do you want to deny access to those files for robots or humans ?

I have read elsewhere on ProWebmasters that we should not disallow: /.js$ or /.css$, but those answers were almost a decade old!

Never disable js and css files because google crawls them to render the urls.

I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden files like /.htaccess?

You can disallow access to theses files, example in your .htaccess file to prevent access to the .htaccess file itself :

# Deny access to .htaccess
<Files .htaccess>
Order allow,deny
Deny from all
</Files>

For /includes or /includes/connect_to_database.php etc ... you can also manipulate .htaccess :

If you choose to modify your httpd.conf file to solve this and you have multiple Options directives, then you must add a - or a + before each directive. Example:

Options -Indexes +FollowSymLinks
  • i thought the whole point of robots.txt is to determine what is indexed, not what is crawled? – PrimitiveNom Aug 5 at 18:39
  • 1
    Google will still crawl and sometimes index urls even if there's instructions not to in robots.txt. The only way to prevent indexing is either .htaccess / htpsswd / meta robots noindex etc... – pjmg Aug 5 at 18:47
  • ok, but you didnt really address my point. isnt the purpose of robots.txt indexing and not crawling? – PrimitiveNom Aug 5 at 19:25
  • Robots.txt is not a file aimed at indexing. It's a set of rules for robots/crawlers. For indexing you can submit url to the search console or use a sitemap. – pjmg Aug 5 at 20:34
  • or <meta name='robots' content='index'> right? – PrimitiveNom Aug 5 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.