A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.).
1
vote
1answer
93 views
What is user agent Issbot
I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
3
votes
4answers
317 views
Google Bots Crawling my Site daily is this good or a bad thing?
Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
2answers
249 views
Menu hidden links and bots!
I want to make a menu with a list with hidden links and when the user points the menu with he mouse the links will appear, my problem is that the bots may think this as a deceiving tactic and lower my ...
2
votes
3answers
738 views
What user agent should I set?
There is Ask bot, which sets this header:
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Considering this, I have the following questions:
If I'm writing a web crawler named Goofy, what user agent ...
6
votes
2answers
197 views
Does the adsense bot ever get bored?
Each time someone clicks on a page like
http://www.example.com/crazy.cgi?input=silly+banana
where the "silly banana" bit represents some arbitrary user input which changes each time, I get yet ...
2
votes
2answers
4k views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
2
votes
1answer
109 views
Is there a good way to filter all the robots and domains they are on?
I have been working more on the bot filter for my website, but by no means is it complete.
So far I have the main ones:
Google,
Yahoo,
MSN,
Baidu,
Amazon,
and a few others...
Right now, I am ...
2
votes
2answers
913 views
How to detect search engine bots reliably within asp.net website?
What is the best way to detect bot generated activity (clicks/page visits) etc on a asp.net website? We have a website where we track leads generated to external websites, we track the IP of the user ...
6
votes
7answers
1k views
Is there an index of the IP addresses used by indexing bots?
I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list.
Is ...
