Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [bingbot]

The tag has no usage guidance.

1
vote
0answers
20 views

How to fix BingWebmaster Sitemap: Failed Status?

In Bing Webmaster, I recently resubmitted my sitemap after doing my regular changes. Every time, I try to resubmit, it shows "Failed" status as shown below. Previously this used to not happen and the ...
1
vote
1answer
25 views

Clients from Microsoft IP address ranges seem to be spoofing Bingbot

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block ...
2
votes
0answers
49 views

User agent of BingWeb - bot or mobile agent?

When I scan my logs for user agents containing Bing (capitalized), I find two types of user agents: those containing BingPreview and those containing BingWeb. Bing lists BingPreview as one of their ...
0
votes
0answers
75 views

Seo Report Result Problems

This is my first seo project.I am new at SEO.Have some problems, Info : Webform ASP.NET 4.5 1-) "Accesible Index Page Configured wrong" error is my biggest problem.My site have 4 URL for homepage . ...
4
votes
1answer
311 views

Bing stopped indexing the site after disallowing all crawlers except Googlebot, Yahoo-slurp, and Msnbot

Bing removed my top pages from results, 3 days ago they were showing at first page. Maybe i updated robots.txt file few weeks ago. Can someone check below template if there is something wrong that is ...
7
votes
2answers
260 views

Bingbot is encoding non-ASCII characters in URLs with backslash x rather than percent

We have hashbang urls with extended utf-8 characters. For example: https://example.com/#!каталог Bing bot fetches the html snapshot https://example.com?_escaped_fragment_=каталог encoded URL as \...
1
vote
1answer
191 views

Bing not indexing URLs From Sitemap [duplicate]

I have a site that Google has indexed 19k pages, in a fairly short amount of time. Bing on the other hand is lagging pretty far behind...51 pages after approximately 3 months of waiting. Many of the ...
0
votes
1answer
93 views

How do you tell what Bing is detecting as malware on a page?

Bing Webmaster Tools started flagging some of my pages as infected with malware. Despite my best efforts, I have been unable to figure out what the problem might be. The "help" message about the ...
1
vote
1answer
245 views

Bingbot is creating more sessions on my website than any other IP address

I ran some aggregation on my site's session data, mostly out of curiosity. I discovered that certain IP addresses had thousands of sessions, and as far as I can tell all of them were "open once, don'...
0
votes
0answers
14 views

Google indexing non-existing pages [duplicate]

Google and Bing are crawling non-existing pages and even though they don't exist, the links get directed to index page. Google even indexed one of these links causing duplicate content. Link format: ...
1
vote
1answer
192 views

Bingbot in requesting stange, invalid URLs

I have strange URLs that Bing is trying to crawl. The Umbraco CMS throws exception when Bingbot request these URLs. Bing appears to think that these are valid URLs, but the URLs do not exist. The ...
7
votes
3answers
647 views

Bing bot following user on password protected pages?

I'm seeing some traffic on our classic ASP website from a verified Bing bot that seems to be following a current user of mine. For instance, if this user adds an item to their shopping cart, 2 to 24 ...
0
votes
0answers
30 views

Bingbot access urls from paypal transactions (Solved) [duplicate]

I log all transactions in the process of a paypal payment. If a user comes back from paypal, a unique token and PayerID is attached as GET query parameter by paypal. Today I noticed that bingbot ...
3
votes
1answer
41 views

Prevent bing from crawling thousands of essentially identical pages?

I have a web page with a dozen tables of data on it, each with half a dozen columns. Every table can be sorted by a column by clicking on the relevant header, and these get appended to the ...
0
votes
2answers
972 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
1
vote
2answers
174 views

Does Yahoo use Bing outside the US/Canada, too?

Yahoo uses the Bingbot at least in the US and Canada to power its search results. Does anyone have some sources on which bot Yahoo uses in other countries?
1
vote
2answers
396 views

BingBot hitting multiple subdomains all at the same time, causing panic

I have a site with multiple subdomains. On certain hours of the day, Bingbots would gather at my site and do a massive scan like this: 01:23:11 a.example.com GET /index HTTP/1.1 200 Bot.A 01:23:11 b....
4
votes
3answers
2k views

Bingbot requests from Google IP address

We have some suspicious requests to our server, 74.125.186.46 - - [24/Aug/2014:23:24:11 -0500] "GET <url> HTTP/1.1" 200 16912 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/...
7
votes
1answer
619 views

Will Bing bot index pages with invalid SSL certificates?

Bingbot and Yahoo Slurp do not support SNI (Server Name Indication) when using SSL. Ignoring other workarounds (multi domain certificates, non-SSL content etc.). Will Bingbot index pages that have an ...
4
votes
1answer
2k views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
11
votes
2answers
984 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
394 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
12
votes
4answers
734 views

Does Googlebot support TLS v1.2? If not, what will my TLS v1.2-only site look like to it?

I currently redirect all non-HTTPS connections to HTTPS, and support TLS v1, TLS v1.1, and TLS v1.2. I reconfigured my testing server to accept only a very limited and secure cipher suite and TLSv1.2 ...
1
vote
1answer
27 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
6
votes
1answer
7k views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record (...
2
votes
2answers
221 views

Generating snapshots of Ajax content for BingBot vs GoogleBot

We have a website whose content is generated dynamically via JavaScript. For the purpose of search engine indexing we also provide snapshots of the pages. Here are some examples: https://www.recreyo....
1
vote
1answer
125 views

Has Microsoft introduced a bug in Bingbot such that it crawls non-existent images?

I've started receiving requests for paths such as /scale/jackets/00000086/80x80_P2_G1715325W.JPG several times a day from the Bingbot. This started a couple of days ago. Anybody know why this is?
1
vote
1answer
2k views

Do web crawlers/spiders index azure web sites?

For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is ...
1
vote
1answer
161 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...