A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.).
8
votes
3answers
291 views
Tactics for dealing with misbehaving robots
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.
Obviously we've had ...
7
votes
2answers
1k views
Ask.com Sitemap Crawler Down (for good)?
I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
6
votes
7answers
1k views
Is there an index of the IP addresses used by indexing bots?
I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list.
Is ...
6
votes
4answers
3k views
Adsense click bot is click bombing my site
I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then ...
6
votes
2answers
197 views
Does the adsense bot ever get bored?
Each time someone clicks on a page like
http://www.example.com/crazy.cgi?input=silly+banana
where the "silly banana" bit represents some arbitrary user input which changes each time, I get yet ...
5
votes
2answers
415 views
Should I block bots from my site and why?
My logs are full of bot visitors, often from Eastern Europe and China.
The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on.
Should I block these bots from my site and ...
5
votes
2answers
140 views
Which meta “robots” tag gets preference?
My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
5
votes
3answers
537 views
Disadvantages of a fake phpMyAdmin honeypot that causes ip blacklisting and robots.txt disallow/exclusion of the honeypot?
I'm trying to figure out whether I should set up a honeypot system with a fake phpMyAdmin (site gets hits all the time with people spidering for insecurities with that app).
My thought was to create ...
4
votes
2answers
174 views
Number of page requests by any Bot in 5 secs
I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?
4
votes
1answer
2k views
Google Webmaster Central tells me that robots is blocking access to the sitemap
This is my robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.mydomain.org/sitemap.xml.gz
But Google Webmaster Central tells me that robots is blocking ...
4
votes
1answer
238 views
What is the notified-CompliancePage page?
I sometimes have bots (I assume) hitting pages such as:
/notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw==
...
3
votes
4answers
1k views
Why deny access to website for msnbot/bingbot?
I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
3
votes
2answers
258 views
How often are sitemap.xml checked for updates by crawlers?
How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
3
votes
4answers
318 views
Google Bots Crawling my Site daily is this good or a bad thing?
Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
2answers
140 views
How can I prevent Googlebot from indexing web service URL's?
I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
3
votes
1answer
290 views
How To Slow Down A Generic Bot?
There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a well-known search-engine, I'd rather slow it ...
3
votes
1answer
529 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
1answer
481 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
3
votes
1answer
480 views
How To Track Down and Stop Rogue Bots?
Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says:
Unknown robot (identified by 'bot*') consumed 164 GB this month.
By comparison, Googlebot ...
3
votes
1answer
83 views
Yandex frequently replaces page names with ampersands
The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is:
/mypage.aspx?param=value
then it ...
2
votes
4answers
3k views
What is the best way to exclude bots from view counts?
My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
2
votes
3answers
745 views
What user agent should I set?
There is Ask bot, which sets this header:
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Considering this, I have the following questions:
If I'm writing a web crawler named Goofy, what user agent ...
2
votes
3answers
309 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
2
votes
1answer
205 views
Stop bots from crawling old links with extensions
I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist.
So I'm trying to find out if there is a ...
2
votes
2answers
230 views
All the IPs of search engine bots?
if I want to filter out all the search engine bots for example for a script that counts users that are online is there website or a list which is allways up to date and includes the ip addresses of ...
2
votes
1answer
131 views
How to hide download file from bots? [duplicate]
Possible Duplicate:
How to restrict the download of all files in a folder?
I want to make a private file available for download but not use username/password protection. I want to put the ...
2
votes
2answers
121 views
Does Yahoo Slurp crawl hammer a site?
A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
2
votes
1answer
413 views
Is it possible to block an entire site to Google then list exceptions?
I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
2
votes
2answers
44 views
What is this robots.txt disallowing?
Robots.txt – It is currently disallowing “/” – GoogleBot image.
Just curious what that is disallowing and if it’s important for search engines to index? You don’t want to restrict any search robots ...
2
votes
2answers
913 views
How to detect search engine bots reliably within asp.net website?
What is the best way to detect bot generated activity (clicks/page visits) etc on a asp.net website? We have a website where we track leads generated to external websites, we track the IP of the user ...
2
votes
1answer
39 views
What does “all” mean in meta robots tag?
Several websites have the following meta tag aimed at search engines like Google, Bing, etc.:
<meta name="robots" content="all" />
What does it do?
2
votes
1answer
138 views
Is browser and bot whitelisting a practical approach?
With blacklisting it takes plenty of time to monitor events to uncover undesirable behavior and then taking corrective action. I would like to avoid that daily drudgery if possible. I'm thinking ...
2
votes
1answer
131 views
How can I let robots access my site?
I have a webpage on my university's server, which means I cannot access the root directory and change files there.
Is there some other way to give access to robots to my page?
Thanks.
2
votes
2answers
4k views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
2
votes
1answer
110 views
Is there a good way to filter all the robots and domains they are on?
I have been working more on the bot filter for my website, but by no means is it complete.
So far I have the main ones:
Google,
Yahoo,
MSN,
Baidu,
Amazon,
and a few others...
Right now, I am ...
2
votes
1answer
34 views
Which tools can help limit maximum page view per ip to limit scrapers and bots?
I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I Am thinking of going with Fail2ban and limiting hits from an ip maybe at around 1000 per day. ...
1
vote
1answer
110 views
Do or can robots cause considerable performance issues?
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Our setup:
Lets say I have site ...
1
vote
1answer
421 views
Question about robots.txt Google shows urls are blocked
I am working on this site dealsin.us and the Google webmasters tool shows that there are about 9800+ urls blocked by robots.txt. You can view the robots.txt here I have blocked some directories which ...
1
vote
1answer
78 views
Do these IP addresses belong to robots?
When i check the traffic on my website, the top IP's have have statistics like this.
> IP Page Hits
> 91.212.x.x 7123 7123
> 91.x.x.x 249 249
These ...
1
vote
2answers
128 views
Robots.txt and “Bad” Robots
I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
1
vote
2answers
840 views
Does searchbots index content in iframe [duplicate]
Possible Duplicate:
Do we still need to avoid using frame/iframe for good SEO?
I have a html page that has a map of a country with lots of dots indicating location of companies through out ...
1
vote
1answer
90 views
In User Agent strings, what does “AskTbTRL” denote?
Here are a bunch of examples:
http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl
I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
1
vote
1answer
501 views
Why are nonexistant files listed in the TRANSFER log?
This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose.
The most puzzling thing to me is that the GET address doesn't correspond ...
1
vote
2answers
50 views
Can Google show some pages deleted six months back and blocked using Robot.txt?
If I have deleted some pages on my website six months back and blocked these pages using Robots.txt, can that non-existent page show up on Google?
1
vote
1answer
49 views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
1
vote
1answer
83 views
Set initial robots.txt restrictions so high, Googlebot won't even try to access the updated less strict file.
When I initially setup my blog I put the restrictions on robots.txt way too high. They were so stick that googlebot can't even access the udated less strict robots file. I've been at this problem for ...
1
vote
2answers
66 views
SEO: How to stop certain web-pages from “having importance” [duplicate]
Possible Duplicate:
SEO - Reducing significance of irrelevant kewords.
I wrote "having importance" since I do not really know the right way to say this.
I have certain pages in my site ...
1
vote
1answer
93 views
What is user agent Issbot
I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
1
vote
1answer
34 views
Different Blocked URLS Stats in Google Webmaster
Just want to know i got different blocked urls stats at webmaster.
I blocked entire site for revamp and latest builds implementation. Here are different stats
Under Health tab> Blocked URls>2,580
...
1
vote
0answers
79 views
Methods for keeping no-good scrapers off the site? [duplicate]
Possible Duplicate:
Tactics for dealing with misbehaving robots
I want to keep no-good scrapers (aka. bad bots) that steal content and consume bandwidth off my site. At the same time, I do ...
