1

Can spiders crawl the sites if there are no asterisks used on the robots.txt?

For example:

User-agent: *
Disallow: .*/ping/.*

then

User-agent: *
Disallow: /ping/

Please can someone confirm this, many thanks

1

As @w3dk said you can't use dot asterisk (.*) in robots.txt with directory, it will probably gives you an error or you will not going to accomplish that you want.

.dot prefix used to block specific kind of file, like this.

User-agent: *
Disallow: *.pdf

This will block all pdf files to crawl in your website. No matter it is in root level or in sub folder.

Your second robots.txt clarification.

User-agent: *
Disallow: /ping/

Here all polite bots including Googlebot, MSNBot, Yandexbot allowed to crawl your all directory except /ping/ directory, but your /ping/ directory should be placed in root level, otherwise it won't work.

First example - www.example.com/ping/ 
Second example (This won't work) - www.example.com/sub/ping/
  • 1
    Note that the pattern the OP used is .* (regex style, dot-asterisk) - this is probably in error. The dot here is a literal dot, not any character. (Also note that robots.txt is prefix matching, so any wildcard * at the end of the URL-path is superfluous.) – MrWhite Oct 13 '16 at 23:50
  • 1
    Thanks w3dk for clarification. I am going to change my answer. – Goyllo Oct 14 '16 at 4:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.