A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.).
2
votes
1answer
39 views
What does “all” mean in meta robots tag?
Several websites have the following meta tag aimed at search engines like Google, Bing, etc.:
<meta name="robots" content="all" />
What does it do?
2
votes
1answer
34 views
Which tools can help limit maximum page view per ip to limit scrapers and bots?
I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I Am thinking of going with Fail2ban and limiting hits from an ip maybe at around 1000 per day. ...
5
votes
2answers
132 views
Which meta “robots” tag gets preference?
My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
1
vote
1answer
34 views
Different Blocked URLS Stats in Google Webmaster
Just want to know i got different blocked urls stats at webmaster.
I blocked entire site for revamp and latest builds implementation. Here are different stats
Under Health tab> Blocked URls>2,580
...
1
vote
2answers
50 views
Can Google show some pages deleted six months back and blocked using Robot.txt?
If I have deleted some pages on my website six months back and blocked these pages using Robots.txt, can that non-existent page show up on Google?
3
votes
2answers
138 views
How can I prevent Googlebot from indexing web service URL's?
I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
5
votes
2answers
403 views
Should I block bots from my site and why?
My logs are full of bot visitors, often from Eastern Europe and China.
The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on.
Should I block these bots from my site and ...
0
votes
0answers
143 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
1
vote
0answers
79 views
Methods for keeping no-good scrapers off the site? [duplicate]
Possible Duplicate:
Tactics for dealing with misbehaving robots
I want to keep no-good scrapers (aka. bad bots) that steal content and consume bandwidth off my site. At the same time, I do ...
1
vote
1answer
49 views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
0
votes
1answer
116 views
mod_rewrite and SEO friendliness
My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes.
I have a unique, large PHP script, and I use the Apache ...
4
votes
2answers
174 views
Number of page requests by any Bot in 5 secs
I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?
0
votes
0answers
65 views
Facebook crawling [closed]
Does a normal person is allowed to crawl all the publically visible links of facebook for research purpose with the help of automated bots?
And whether there is any API provided by ...
2
votes
1answer
131 views
How to hide download file from bots? [duplicate]
Possible Duplicate:
How to restrict the download of all files in a folder?
I want to make a private file available for download but not use username/password protection. I want to put the ...
1
vote
1answer
83 views
Set initial robots.txt restrictions so high, Googlebot won't even try to access the updated less strict file.
When I initially setup my blog I put the restrictions on robots.txt way too high. They were so stick that googlebot can't even access the udated less strict robots file. I've been at this problem for ...
4
votes
1answer
2k views
Google Webmaster Central tells me that robots is blocking access to the sitemap
This is my robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.mydomain.org/sitemap.xml.gz
But Google Webmaster Central tells me that robots is blocking ...
2
votes
3answers
309 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
3
votes
1answer
83 views
Yandex frequently replaces page names with ampersands
The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is:
/mypage.aspx?param=value
then it ...
0
votes
0answers
181 views
List of most common bot user-agents? [closed]
I'm trying to find a list of the most common bot user-agents (say 10 or so) but I can't find much of anything; all I'm finding is lists that contains thousands of user-agents.
All I need is the most ...
3
votes
1answer
526 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
4answers
1k views
Why deny access to website for msnbot/bingbot?
I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
1
vote
1answer
110 views
Do or can robots cause considerable performance issues?
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Our setup:
Lets say I have site ...
6
votes
4answers
3k views
Adsense click bot is click bombing my site
I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then ...
1
vote
2answers
128 views
Robots.txt and “Bad” Robots
I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
2
votes
1answer
136 views
Is browser and bot whitelisting a practical approach?
With blacklisting it takes plenty of time to monitor events to uncover undesirable behavior and then taking corrective action. I would like to avoid that daily drudgery if possible. I'm thinking ...
7
votes
2answers
1k views
Ask.com Sitemap Crawler Down (for good)?
I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
1
vote
1answer
172 views
Using rel=canonical and noindex in a 1-n partners enviroment
We sell a whole site (domain, etc) to partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original, but the one that is ...
0
votes
1answer
244 views
Apache real time robot detection
Background
Our website is being
a. scraped - complete pages downloaded
b. spidered - complete pages and sub links followed
c. mined - specific files downloaded at high speed
by hundreds of ...
1
vote
2answers
834 views
Does searchbots index content in iframe [duplicate]
Possible Duplicate:
Do we still need to avoid using frame/iframe for good SEO?
I have a html page that has a map of a country with lots of dots indicating location of companies through out ...
3
votes
1answer
481 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
1
vote
1answer
420 views
Question about robots.txt Google shows urls are blocked
I am working on this site dealsin.us and the Google webmasters tool shows that there are about 9800+ urls blocked by robots.txt. You can view the robots.txt here I have blocked some directories which ...
3
votes
2answers
256 views
How often are sitemap.xml checked for updates by crawlers?
How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
2
votes
1answer
205 views
Stop bots from crawling old links with extensions
I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist.
So I'm trying to find out if there is a ...
0
votes
1answer
781 views
BingPreview user agent
I'm getting several invalid requests from a "BingPreview" user agent. The complete user agent header is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
The ...
4
votes
1answer
235 views
What is the notified-CompliancePage page?
I sometimes have bots (I assume) hitting pages such as:
/notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw==
...
2
votes
2answers
121 views
Does Yahoo Slurp crawl hammer a site?
A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
1
vote
1answer
90 views
In User Agent strings, what does “AskTbTRL” denote?
Here are a bunch of examples:
http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl
I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
1
vote
1answer
285 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
1
vote
1answer
501 views
Why are nonexistant files listed in the TRANSFER log?
This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose.
The most puzzling thing to me is that the GET address doesn't correspond ...
3
votes
1answer
289 views
How To Slow Down A Generic Bot?
There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a well-known search-engine, I'd rather slow it ...
3
votes
1answer
478 views
How To Track Down and Stop Rogue Bots?
Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says:
Unknown robot (identified by 'bot*') consumed 164 GB this month.
By comparison, Googlebot ...
8
votes
3answers
291 views
Tactics for dealing with misbehaving robots
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.
Obviously we've had ...
1
vote
1answer
78 views
Do these IP addresses belong to robots?
When i check the traffic on my website, the top IP's have have statistics like this.
> IP Page Hits
> 91.212.x.x 7123 7123
> 91.x.x.x 249 249
These ...
1
vote
2answers
66 views
SEO: How to stop certain web-pages from “having importance” [duplicate]
Possible Duplicate:
SEO - Reducing significance of irrelevant kewords.
I wrote "having importance" since I do not really know the right way to say this.
I have certain pages in my site ...
2
votes
1answer
406 views
Is it possible to block an entire site to Google then list exceptions?
I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
2
votes
2answers
44 views
What is this robots.txt disallowing?
Robots.txt – It is currently disallowing “/” – GoogleBot image.
Just curious what that is disallowing and if it’s important for search engines to index? You don’t want to restrict any search robots ...
5
votes
3answers
536 views
Disadvantages of a fake phpMyAdmin honeypot that causes ip blacklisting and robots.txt disallow/exclusion of the honeypot?
I'm trying to figure out whether I should set up a honeypot system with a fake phpMyAdmin (site gets hits all the time with people spidering for insecurities with that app).
My thought was to create ...
2
votes
1answer
131 views
How can I let robots access my site?
I have a webpage on my university's server, which means I cannot access the root directory and change files there.
Is there some other way to give access to robots to my page?
Thanks.
2
votes
4answers
3k views
What is the best way to exclude bots from view counts?
My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
2
votes
2answers
229 views
All the IPs of search engine bots?
if I want to filter out all the search engine bots for example for a script that counts users that are online is there website or a list which is allways up to date and includes the ip addresses of ...
