Timeline for Do two word bots need escaping with %20 or are spaces Ok in robots.txt
Current License: CC BY-SA 4.0
3 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 9 at 8:17 | comment | added | SKidd | Yeah information is woefully sparse even on supposedly authoritative sites on the space issue. FWIW there are sites I've seen that list known spiders and they include Baidu Spider, with a space. My concern is mostly having robots.txt not read by, say, Google because of a syntax error (it's happened before albeit with an obvious 'whoops' mistake. | |
| Jul 8 at 20:13 | comment | added | KIKO Software |
I can't find anything that says you can use %20, so I wouldn't. Checking my own robot.text it contains: User-agent: Baiduspider. Google says: Spaces are optional. In reality, I cannot find anything definite on how you should do it.
|
|
| Jul 8 at 12:49 | history | asked | SKidd | CC BY-SA 4.0 |