Hot answers tagged web-crawlers
10
This is Vincent from Bing Webmaster Tools.
This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details:
Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center
Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog
You ...
6
This is no doubt dependent on your remaining 200+ lines, but what you have posted so far would seem to be reducible to just 2 lines:
RewriteRule ^patients/billing/(FAQ_billing|getintouch).html$ $1.php [L,NC]
RewriteRule ^patients/findadoctor/([a-z]).html$ findadoctor.php?id=$1 [L,NC]
This shouldn't be reducing your site to a crawl.
As mentioned in ...
5
I don't think people should ban bing bot.
Bing has an equivalent Bing Webmaster tools at http://www.bing.com/toolbox/webmaster/
where they also have 'Crawl Settings' where you can adjust the crawl rate as seen in this video:
http://www.bing.com/videos/watch/video/bing-webmaster-tools-crawl-rate-settings/1ii1ej9jz
Googlebot is just as notorious in excessive ...
4
Downloading another site's code, removing their branding, and re-uploading as your own site will get you in trouble sooner or later. Don't do that.
Besides, it won't work. Complex sites have more going on than just the final HTML output. There's a ton of moving parts and server-side code just to get to the output. You are strongly encouraged to build ...
3
msnbot is quite prolific when it comes to spidering servers and if you have a lot of pages to index it can quite easily cripple your server. As traffic from MSN is considerably less than what Google can give it's quite common just to deny the msnbot via .htaccess, iptables or robots.txt. With Googlebot you can limit the speed quite easily in ...
2
Have you looked at the Bing Webmaster Tools at all?
You can sign up for them and the process of claiming your domains is the same as for Google Webmasters Tools.
These can then give you a full list of crawl stats including links leading to 404s.
Note that if you've removed content you'll see 0 links becuase the not is requesting pages it knew about ...
2
It's highly unlikely you're going to find some completely universal list of UserAgents, in part because they can just be made up. Before even getting to that, though, it'd be a ridiculous amount of work. You just need to compile a few resources and then do some further searching for anything else you don't recognize. (Surprisingly, I can't find a Wikipedia ...
2
Modern inbound marketing does not just rely on getting indexed by Google's spiders, or even just Google and Bing/Yahoo. As SEO and SMM become more and more intertwined, more and more social media and social sharing services come into play. As such, you'll see crawlers that aren't just search spiders.
When you post a link on Twitter and it gets shortened by ...
1
The htaccess does, by definition, slow down your site because the server must check every folder for the file - e.g. in your case the folders /patients/findadoctor/, /patients/ and the root folder. It is more efficient to write those rules in the server config itself, then set AllowOverride None, which will stop Apache looking for htaccess files. If you are ...
1
I use the following Wordpress plugins which might help you:
True Google 404 - Keeps track of 404 errors and tries to resolve the ones that it can through site search
Custom Permalinks - Lets you choose the URLs for your pages (you could rename the URLs to what they used to be under Joomla)
Quick Page/Post Redirect Plugin - Allows you specify other ...
1
Yes, there are crawlers that can crawl a site which requires login. This requires that you log into your site using your web browser and export your cookies. Then you start the crawler with the cookies that you had used to log in and the the crawler crawls the site as your logged in user.
To export your cookies, use Firefox with the Export Cookies ...
1
This week our company (Incapsula) launched Botopedia.org - a Community-Sourced bot directory. It's 100% free and open for all and you can use it to find a complete user-agent list for all bots you`ll want to look up.
As for indentification methods, I want to refer you to this discussion in Security.Stackexchange which covers different methods of bot ...
1
One of my clients was doing $10,000 monthly from Bing shopping alone. Organics from Bing was even more. Banning them would cause a big loss of revenue. Anyone suggesting it must have their own personal reasons. Bing generates visits so if you want to decrease your traffic go a head and ban Bing. Otherwise like Anthony said you can work with their Webmaster ...
1
If the links still exist on other sites then Google will continue to look for them on your site, there isn't much you can do to stop it if you can't remove the links.
The only thing that you can do is add the code below to your htaccess file, this 410 gone message will let Google know the content is gone for good. Google may not stop looking but at least ...
1
Yes 1, 2, and 3 will be counted as a link, because it's a link. Google doesn't look at a social bookmark and give it any lesser value because it's a social bookmark. Just as they don't look at a link on a .edu and give it any more credit based on the TLD.
Google is taking into account hundreds of signals. To Google a link is a link now what else can they ...
1
Google decides what to show 1st based on how its algorithm matches the words that user is currently searching. So if pages www.example.com and www.example.com/index.php?a=1&b=2 show different contents (i.e. the 2nd one shows a 'product X') it might be that Google decided the 2nd one best fit a certain search (i.e. user searched exactly for the words ...
Only top voted, non community-wiki answers of a minimum length are eligible
