A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.).
3
votes
1answer
81 views
Yandex frequently replaces page names with ampersands
The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is:
/mypage.aspx?param=value
then it ...
2
votes
1answer
32 views
Which tools can help limit maximum page view per ip to limit scrapers and bots?
I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I Am thinking of going with Fail2ban and limiting hits from an ip maybe at around 1000 per day. ...
1
vote
1answer
33 views
Different Blocked URLS Stats in Google Webmaster
Just want to know i got different blocked urls stats at webmaster.
I blocked entire site for revamp and latest builds implementation. Here are different stats
Under Health tab> Blocked URls>2,580
...
1
vote
1answer
169 views
Using rel=canonical and noindex in a 1-n partners enviroment
We sell a whole site (domain, etc) to partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original, but the one that is ...
1
vote
1answer
284 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
0
votes
0answers
140 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
