The spider tag has no wiki summary.
3
votes
2answers
53 views
Am I being hacked or harvested? [closed]
I just did some log analysis of Q1 for one of my client's sites.
There were about 30,000 sessions created for this site for each of the five separate IP addresses starting with 79.171.81.xxx. 150,000 ...
0
votes
1answer
74 views
Spiders crawling broken links
Tl:Dr
Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin.
The Problem
It seems at least once a week we're getting a big ...
2
votes
1answer
66 views
Is there a spider / link checker that can start deep inside a login-protected site
We use vendor hosted Blackboard for our distance education courses, but host course multimedia on our own servers. The multimedia server has been moved and the domain has changed. Blackboard DBAs have ...
1
vote
2answers
82 views
Why is Verisign spidering my website?
In my logs, I keep noticing that 69.58.178.57, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
2
votes
3answers
303 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
0
votes
3answers
2k views
How to block baidu spiders
Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
3
votes
1answer
474 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
7
votes
2answers
494 views
@font-face, google fonts and SEO
Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
3
votes
1answer
267 views
Is there any reason to allow Yahoo! Slurp to crawl my site?
I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
8
votes
4answers
599 views
Preventing robots from crawling specific part of a page
As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
2
votes
3answers
63 views
Anyone use Spiders apart from search engines and spammers?
Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would they ever use them?
Thanks
5
votes
1answer
528 views
Am I covering the major search engine spiders in my anti-crawl protection white list?
I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
1
vote
1answer
428 views
How does bing-bot( is that the right spider-name? ) and googlebot interpret 301 redirect?
I've been looking for documentation on how the Microsoft and Google bots interpret 301 redirects.
It seems that google-bot stores documents on a url based index system. But I haven't been able to ...
2
votes
1answer
218 views
Should I strategically use NoFollow on internal links to make our indexing crawl more efficient?
We have a well designed internal linking system that makes certain no page is more than four pages deep, and the indexing process has gone well for us.
However, in addition to the hierarchical page ...
2
votes
3answers
738 views
What user agent should I set?
There is Ask bot, which sets this header:
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Considering this, I have the following questions:
If I'm writing a web crawler named Goofy, what user agent ...
10
votes
3answers
2k views
Does the Google spider render JavaScript?
Does the Google spider knows how to render JavaScript, or only HTML?
2
votes
1answer
109 views
Is there a good way to filter all the robots and domains they are on?
I have been working more on the bot filter for my website, but by no means is it complete.
So far I have the main ones:
Google,
Yahoo,
MSN,
Baidu,
Amazon,
and a few others...
Right now, I am ...
1
vote
1answer
105 views
What is USH.HTML and why do some robot try to access it?
I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is.
Is this a type of robots.txt file or something specific ...
20
votes
4answers
750 views
Is Yahoo Search the same as Bing Search now?
We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
