A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.).
2
votes
1answer
398 views
Is it possible to block an entire site to Google then list exceptions?
I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
2
votes
2answers
44 views
What is this robots.txt disallowing?
Robots.txt – It is currently disallowing “/” – GoogleBot image.
Just curious what that is disallowing and if it’s important for search engines to index? You don’t want to restrict any search robots ...
2
votes
4answers
3k views
What is the best way to exclude bots from view counts?
My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
2
votes
1answer
131 views
How can I let robots access my site?
I have a webpage on my university's server, which means I cannot access the root directory and change files there.
Is there some other way to give access to robots to my page?
Thanks.
1
vote
1answer
93 views
What is user agent Issbot
I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
2
votes
3answers
738 views
What user agent should I set?
There is Ask bot, which sets this header:
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Considering this, I have the following questions:
If I'm writing a web crawler named Goofy, what user agent ...
2
votes
1answer
109 views
Is there a good way to filter all the robots and domains they are on?
I have been working more on the bot filter for my website, but by no means is it complete.
So far I have the main ones:
Google,
Yahoo,
MSN,
Baidu,
Amazon,
and a few others...
Right now, I am ...
2
votes
2answers
911 views
How to detect search engine bots reliably within asp.net website?
What is the best way to detect bot generated activity (clicks/page visits) etc on a asp.net website? We have a website where we track leads generated to external websites, we track the IP of the user ...
6
votes
7answers
1k views
Is there an index of the IP addresses used by indexing bots?
I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list.
Is ...
