0

I want to tell Baidu not to crawl. One of their crawlers is Baidu Spider.

Should that be: User-agent: Baidu%20Spider or User-agent: Baidu Spider

Improve this question
2
  • I can't find anything that says you can use %20, so I wouldn't. Checking my own robot.text it contains: User-agent: Baiduspider. Google says: Spaces are optional. In reality, I cannot find anything definite on how you should do it.
    – KIKO Software
    Commented Jul 8 at 20:13
  • Yeah information is woefully sparse even on supposedly authoritative sites on the space issue. FWIW there are sites I've seen that list known spiders and they include Baidu Spider, with a space. My concern is mostly having robots.txt not read by, say, Google because of a syntax error (it's happened before albeit with an obvious 'whoops' mistake.
    – SKidd
    Commented Jul 9 at 8:17

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.