The tag has no usage guidance.

learn more… | top users | synonyms

4
votes
3answers
138 views

Bing bot following user on password protected pages?

I'm seeing some traffic on our classic ASP website from a verified Bing bot that seems to be following a current user of mine. For instance, if this user adds an item to their shopping cart, 2 to 24 ...
0
votes
0answers
9 views

Bingbot access urls from paypal transactions (Solved) [duplicate]

I log all transactions in the process of a paypal payment. If a user comes back from paypal, a unique token and PayerID is attached as GET query parameter by paypal. Today I noticed that bingbot ...
3
votes
1answer
27 views

Prevent bing from crawling thousands of essentially identical pages?

I have a web page with a dozen tables of data on it, each with half a dozen columns. Every table can be sorted by a column by clicking on the relevant header, and these get appended to the ...
0
votes
2answers
165 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
1
vote
2answers
137 views

Does Yahoo use Bing outside the US/Canada, too?

Yahoo uses the Bingbot at least in the US and Canada to power its search results. Does anyone have some sources on which bot Yahoo uses in other countries?
1
vote
2answers
245 views

BingBot hitting multiple subdomains all at the same time, causing panic

I have a site with multiple subdomains. On certain hours of the day, Bingbots would gather at my site and do a massive scan like this: 01:23:11 a.example.com GET /index HTTP/1.1 200 Bot.A 01:23:11 b....
4
votes
3answers
1k views

Bingbot requests from Google IP address

We have some suspicious requests to our server, 74.125.186.46 - - [24/Aug/2014:23:24:11 -0500] "GET <url> HTTP/1.1" 200 16912 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/...
7
votes
1answer
497 views

Will Bing bot index pages with invalid SSL certificates?

Bingbot and Yahoo Slurp do not support SNI (Server Name Indication) when using SSL. Ignoring other workarounds (multi domain certificates, non-SSL content etc.). Will Bingbot index pages that have an ...
3
votes
1answer
454 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
11
votes
2answers
904 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
186 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
8
votes
4answers
522 views

Does Googlebot support TLS v1.2? If not, what will my TLS v1.2-only site look like to it?

I currently redirect all non-HTTPS connections to HTTPS, and support TLS v1, TLS v1.1, and TLS v1.2. I reconfigured my testing server to accept only a very limited and secure cipher suite and TLSv1.2 ...
1
vote
1answer
23 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
6
votes
1answer
3k views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record (...
2
votes
2answers
147 views

Generating snapshots of Ajax content for BingBot vs GoogleBot

We have a website whose content is generated dynamically via JavaScript. For the purpose of search engine indexing we also provide snapshots of the pages. Here are some examples: https://www.recreyo....
1
vote
1answer
113 views

Has Microsoft introduced a bug in Bingbot such that it crawls non-existent images?

I've started receiving requests for paths such as /scale/jackets/00000086/80x80_P2_G1715325W.JPG several times a day from the Bingbot. This started a couple of days ago. Anybody know why this is?
1
vote
1answer
1k views

Do web crawlers/spiders index azure web sites?

For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is ...
1
vote
1answer
143 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...