Hot answers tagged spider
13
Apparently it does, some, according to Matt Cutts:
"For a while, we were scanning within JavaScript, and we were looking for links. Google has gotten smarter about JavaScript and can execute some JavaScript. I wouldn't say that we execute all JavaScript, so there are some conditions in which we don't execute JavaScript. Certainly there are some common, ...
11
Yes and no, he's conflating two different things.
In late 90's/early 2000's, when designers needed to use fonts for design reasons, they embedded graphics files in HTML pages. Search engines cannot understand these, nor can the screen readers used by blind and partially sighted people. This is because the only HTML code in the page for a menu link, for ...
8
I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back.
From the document Joel linked to there is another document ...
7
It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.
4
Another solution is to wrap the sig in a span or div with style set to display:none and then use Javascript to take that away so the text displays for browsers with Javascript on. Search engines know it's not going to be displayed so shouldn't index it.
This bit of HTML, CSS and javascript should do it:
HTML:
<span class="sig">signature goes ...
4
I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness.
First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...
4
Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser.
It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...
4
I raised a search enquiry with Yahoo on Monday - the acknowledgement included the following note on the transition to Bing which you might find relevant:
Yahoo! Search is excited to reach a new stage in the Yahoo! and
Microsoft Search Alliance; the integration of Microsoft algorithmic
results is near complete for the U.S. and Canada marketplace. ...
3
Is it definitely a bot? Do you have a user-agent? And is there a referer [sic] header for those requests?
A Google search doesn't bring up anything remotely significant so I don't think it's related to robots.txt, nor anything like a bot polling for security flaws.
It is likely to be something simple like a mistyped link to your site - for example you have ...
3
PageRank shaping is a perfectly fine thing to do.
Bots don't get distracted by multiple links to the same page so I wouldn't even continue that train of thought. I also wouldn't throw away PR to your internal pages for any reason (nofollow links still send PR but it essentially thrown away as it does not get counted for the receiving page). If you're ever ...
3
Some reasons a bot may be used that come to mind for me are:
Some directories will crawl a website for its meta information (usually after a site has been submitted to the directory)
Contextual ad services like Adsense use bots to figure out what a page is about so relevant ads can be displayed
Webmasters can use bots to find potential linking partners. ...
2
I wish I could I find the article I read about 3 months ago. However, the short version is that Yahoo is using Bing's index (entirely) and algorithm (mostly). Yahoo is still crawling with their own bot to make small amendments to the search results and for other projects.
In short, banning the bot could drop you entirely out of the Yahoo SERPs (but would ...
2
There are many, many potential uses for web crawlers - a review of Wikipedia's list of Open Source Web Crawlers or examples of Spidering Hacks provided by O'Reilly may help illustrate the numerous ways webmasters and programmers utilize others' services and information.
In addition to "friendly" spiders, you can also expect to see plenty of spambot ...
2
This is the wrong strategy. Also, headers are trivial to spoof.
Honestly, anti-crawler protections are very fragile and generally unwise. You may end up blocking legitimate users (who'll be annoyed), or your code might end up being forgotten about, become stale, and block crawlers you want to allow.
You can verify whether a bot belongs to Google, however - ...
2
Never redirect like this using JavaScript. It's bad for a variety of reasons. Stick to the standards and send a 301 redirect via an HTTP header. Otherwise you risk sending the wrong message, or no message, to the search engine and then you run the risk of being de-indexed, penalized, or somehow losing value that you have rightly accrued (e.g. link ...
2
It does, partly, as a quick Google search reveals:
http://blogs.forbes.com/velocity/2010/06/25/google-isnt-just-reading-your-links-its-now-running-your-code/
Still Google is relatively quiet on what the bot exactly understands.
2
You mean interpret JavaScript? It does request the JavaScript files.
And I'm pretty sure it does, to some degree.
T.J. Crowder's answer confirms this.
2
Have you looked at the Bing Webmaster Tools at all?
You can sign up for them and the process of claiming your domains is the same as for Google Webmasters Tools.
These can then give you a full list of crawl stats including links leading to 404s.
Note that if you've removed content you'll see 0 links becuase the not is requesting pages it knew about ...
2
It's highly unlikely you're going to find some completely universal list of UserAgents, in part because they can just be made up. Before even getting to that, though, it'd be a ridiculous amount of work. You just need to compile a few resources and then do some further searching for anything else you don't recognize. (Surprisingly, I can't find a Wikipedia ...
1
You can block by IP address using the ngx_http_access_module of nginx. To block a single IP you can add a line to the conf file like
deny 12.34.567.1;
To block a range, use CIDR notation, like 12.34.567.1/24 for the 24-bit subnet block (of 256 IP addresses) which includes the 12.34.567.1 IP address. For more details see, for instance, this page.
1
First, inform your client and ask what they would like to do - it's
still their site - and it could be a major customer's traffic or another source you're not privvy to.
If client knows nothing of the traffic source, suggest blocking the IP range & monitor load. Notify the client as soon as the block has been implemented.
If the traffic jumps to ...
1
Ideally you'd set a 301 Redirect from /foo/bar/item/ to either your homepage or to the main list of products (like a category-type page). This means:
Any robots will be automatically taken to a valid page
Any users will be automatically taken to a usable page
Your error log should be much cleaner
Search engines will stop picking up broken pages
If you ...
1
Yes, there are crawlers that can crawl a site which requires login. This requires that you log into your site using your web browser and export your cookies. Then you start the crawler with the cookies that you had used to log in and the the crawler crawls the site as your logged in user.
To export your cookies, use Firefox with the Export Cookies ...
1
It could be related to the daily malware scan[CNet, PDF @Verisign] they run automatically as a service to their ssl certificate customers. You could always opt out of the service temporarily in order to see if the crawls from that IP cease.
1
This week our company (Incapsula) launched Botopedia.org - a Community-Sourced bot directory. It's 100% free and open for all and you can use it to find a complete user-agent list for all bots you`ll want to look up.
As for indentification methods, I want to refer you to this discussion in Security.Stackexchange which covers different methods of bot ...
1
I think the following links can help:
http://www.user-agents.org/
http://en.wikipedia.org/wiki/User_agent#Format
http://tools.ietf.org/html/rfc1945#section-10.15
1
Are there any other good techniques to
detect if the hit is coming from a bot
or not?
It depends upon which variety of bot you're hunting - here are a few tips for isolating malicious bots:
Look for hits on non-existent (or
restricted access) administrative
scripts, e-mail scripts, et cetera
Look for nearly-instantaneous
retrieval of your site ...
Only top voted, non community-wiki answers of a minimum length are eligible
