Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...
Update: As of Jan 30, 2015, Googlebot supports TLS 1.2 . See Post by Google:
Googlebot now supports TLS version 1.2, in addition to previous
versions: TLS 1.0, TLS 1.1 and SSLv3.
As of September 14th, 2014, the Googlebot does NOT support TLS 1.2. You get notified about 100% inaccessibility and thats it. I am affected by this directly. Using TLS 1.2 ...
Most bots don't accept cookies (including Googlebot), however, some bots do.
You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. ...
503 Unavailable is intended for all users (Downtime for XYZ).
429 Is intended for users and machines that request too many threads (Requests).
You should use status 429 as this implies that it is intended for that user and other users may not be affected. Furthermore the better status would be 403 Forbidden if your blocking on IP addresses as this implies ...
This is Vincent from Bing Webmaster Tools and I noticed your post.
First of all, I'm sorry to hear about the problem you are having with our crawler's crawl activity across your subdomains. I am sure we can do better.
Couple of things:
I noticed you mentioned crawl pattern setting in Webmaster Tools wasn't working for. The reason is that when using a ...
You should encode the image name before giving it to crawl for any bots, i.e. the image name should be as %2F%D0%B7%D1%83%D1%80%D0%B0%D0%B3.jpg in your page source code.
All modern browser decode it and for users will show the UTF-8 characters in a human-friendly way.
So, do server-side encoding and encode all these characters before including in your ...
Maybe that user has a toolbar from Bing installed. This toolbar might submit visited URLs to Bing, and the Bingbot might (try to) crawl these URLs then.
From the searchengineland.com article How The Bing Toolbar May Accidentally Submit Private Pages & Ads:
Microsoft has confirmed that they do discover and index URLs that they find through users surfing ...
No, the last two lines of your robots.txt file take precedence over the first four as the syntax of those first four lines is incorrect. As a result Google is blocked from viewing your website.
To allow Google and Bing you must specifically and individually allow each crawler:
These are Google IP addresses as you stated. However, this does not mean that it is part of the search engine. Google has expanded it's business lately and not all of what is happening using a Google IP address has lived up to the standards we have all grown accustomed to. Unfortunately.
There are no reverse PTR records for these IP addresses. The ...
Msnbot is just one of the user agents that Bing users, and it isn't even the main one. According to https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
Bingbot - Main bing crawler
Msnbot - Old crawler, still used some
MSNBot-Media - Image and video crawler
AdIdxBot - Ads crawler
BingPreview - Preview crawler
To allow Bing to index ...
We found out that not only CloudFlare, but also other services on CloudFlare were blocking BingBot.
Finally we allowed Bing user-agent Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) in Firewall settings and also added another rule to by-pass other default rules in WAF when the user-agent is Bingbot.
It may vary as per individual ...
Google should be able to handle all versions of SSL/TLS.
The best way to see how Google handles something is via Google Webmaster Tools.
Note: You have to have your site setup in Webmaster Tools for this to work. It's something you should consider anyway.
Go to Webmaster Tools
Select your site
Select Fetch as Google
I suspect you are right. I would say that if Google and/or Bind are paying attention to the site, it would be better to just wait. First it would take time to spider the entire site, then it would take time to refresh the index and even more time to correct any penalty that may be applied. Expect as much as 60 days before the SERPs trend toward any ...
It is normal to see abnormal bot behavior, in the sense that numbers do not seem to make sense. The boards are full of questions addressing many variations of similar situations.
2 things to do if you have not done one, the other or both.
Sign up with Bing webmaster tools: http://www.bing.com/toolbox/webmaster
Control how Bing crawls your site: https://...
Looking at Microsoft's public IP address space, it looks to me like Microsoft only owns 18.104.22.168/11, 22.214.171.124/13, and 126.96.36.199/14 - they don't own the entire /8 (13.X.X.X).
The 13.X.X.X block is administered by ARIN, and my guess is that it is divided up among other corporations.
So unless I am mistaken, I don't think Microsoft actually owns the IP ...
There are 2 separate things being looked at here -
the IP address - which from your advice is not tracking back to Microsoft. IP address is a strong predictor of prominence as it is difficult to forge (it's certainly possible, but not something your average user can do as it requires elevated access to networks which is not something most people have - and ...
The reason for this is simple.
There are two major schools of thought regarding indexing the web.
One is to index everything and then decide what to keep. The advantage of this is that all metrics regarding sites and pages are available to make decisions.
The second is to selectively index pages making any decision before indexing the page. The ...
Bing has now deleted its malware warnings from Bing Webmaster Tools. I think at this point it must have been a false positive.
I never did find any way to dig into what the issue was. Bing does not appear to publish any in depth analysis of what malware it actually found or what triggered their report.
According to Yahoo’s documentation for webmasters, yes, they use Bing’s index for Yahoo Search, and they don’t mention any geographic restriction:
Are you a webmaster or website owner looking to remove your webpage(s) from Yahoo Search?
Search results on Yahoo Search are now powered by Bing. […]
In the footer of their SERPs, it says "Powered by ...
It seems so. I did some research and I am finding that Yahoo! will use Bing for search period. It seems that Yahoo! is giving up running a search engine despite the fact that they own Inktomi which was one of the worlds larges indexes at one time even rivaling Google.
It is seen as a way to drastically cut costs by removing the most expensive part of search ...
Do you happen to send some of these urls by mail? In that case a nosniff header could help to prevent mail clients from sniffing the page.
Hotmail would use Bing-bot, Gmail would use the Google crawler and so on.
Here is a nosniff snippet for htccess:
# Add headers to all responses.
# Disable content sniffing, since it's ...
I actually experienced this at one point on my site.
What you can do to solve the problem is to make the links the search engine robots are not supposed to access as POST based links.
Here's how you do it perfectly with good compatibility for just about every browser invented:
Create a form with a method post. For example:
You could tell Bing, and other webcrawlers, what to spider and what to ignore using a file called robots.txt in the root of your website.
You can tell specific or all crawlers to ignore specific urls.
in your case
you might need to make small changes in the Disallow line depending on the parameters used ...
Based on the contents of your robots.txt file and your needs, you're better off making that file entirely empty or even better, maybe with a link to the sitemap files so search engines can find your URLs.
I suggest this because crawlers normally think they are allowed to access anything. By specifically using Disallow, you're trying to tell at least one ...
As others mentioned, you can verify real Googlebots, and this isn't a Googlebot IP address.
I double-checked with the team about these requests, and they appear to be for the PageSpeed service, which can act as a cache/proxy for websites. If search engines - like Bing or Google - crawl URLs like that, the service will forward those requests to your website ...
You can verify all Google crawlers by using PTR records.
See: Verifying Googlebot
I find this very accurate. Google now also offers hosting services as well as many other services, so there can easily be requests from Google assigned IP ranges that are not Google's search agents.
There is also a good list of Google User Agents.
EDIT: As Martin points out, Bing can still index the site with SSL links in SERPS even though it errors on SNI -- the error occurs before the following results. The following is arbirtary for the question.
Bing does support SNI [Edit: as in it can still successfully index SSL links]. Im looking at a multidomain multiSSL via SNI setup right now, clicking the ...