Tagged Questions
2
votes
2answers
41 views
How can an exception be created for Facebook on robots.txt?
I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
0
votes
0answers
65 views
Facebook crawling [closed]
Does a normal person is allowed to crawl all the publically visible links of facebook for research purpose with the help of automated bots?
And whether there is any API provided by ...
1
vote
1answer
285 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
4
votes
1answer
332 views
Should I block search engines with robots.txt on my Facebook app?
There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
