Hot answers tagged web-crawlers
Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x
Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...
Most bots don't accept cookies (including Googlebot), however, some bots do. You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. ...
I think that you may be interested in this article: Design AJAX-powered sites for accessibility. It's a very basic article, but it should give you some pointers about what to do and what not to do. AJAX should be used to improve the experience on specific pages and elements of the site, but not to build a whole site. For the user perspective, it's a bad ...
There is no difference to search engines between those two methods. You can generate it like you want. Moreover, you need to know blocking a page with robots.txt is not the perfect way to prevent indexing of pages. You can read this question for more information.
There is some argument about how bounce rate is measured. Google has even changed how it looks at bounce rates by adjusting how it views a visit through the years. In short, Google says that the bounce rate is the rate where a user has left your site from the entrance page. This page seems to explain some of what Google sees as a visitor: ...
Specifying a canonical on the filter pages to the parent (non-filter) page is perfectly fine but you can also serve a meta noindex, nofollow on these filter pages as well and instruct Google what to do with these filter URL's (if they contain parameters) from the section of Google WMT you've already mentioned. Disallowing them in robots.txt is also another ...
Every time I read questions like this I think of Kevin Spacey's character in Henry and June. The fellow who was always writing his greatest Novel, but was so worried about someone stealing his ideas he kept it locked away in a briefcase, carried close to his chest... Every linux user is a "legit" wget user. I use it often for grabbing debs, videos, ...
wget is often used for scraping. It's a command-line tool to download webpages and their assets. If your website isn't being publicized, you can almost be sure that it's a bot doing scraping. So yes, you could block it, but also be aware you may need to do something more sophisticated than blocking it with robots.txt since wget can easily be told to ignore ...
If a page has content="nofollow" , google will not index the page and links inside that page also . so your iframe page also will not be indexed . Unless the iframe link is there from some other page also which doesnt have a no follow tag . or create a sitemap with the pages in that iframe and submit to google for indexing.
Disallow: /article/ will block the URL (or page) article/ and all subfolders or subpages. That's why, if the robots.txt of externaldomain.com applies this directive for the user agent is googlebot, don't worry. Here are Google instructions on this.
Because it could then be abused to index a single page 100 times per minute, which would waste Google's server power and bandwidth. You may look at Google as a machine with limitless resources but it is not so, every byte and cpu cycle counts :) As for the 500 and 10 links submission limit per week - it is set reasonably high, most smaller sites do not ...
Only top voted, non community-wiki answers of a minimum length are eligible