Hot answers tagged

6 votes
Accepted

How long does Bingbot typically take to process a sitemap submission?

This post is 4 years old, hope the status in not Pending anymore. But this helped me. After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call ...
langeleppel's user avatar
4 votes

Why are Bing and SEMRushBot ignoring crawl-delay in my robots.txt file?

Your robots.txt file does not do what (I think) you intend it to do because you do not use blank lines between the sections. The robots.txt standard says (emphasis added): The format logically ...
Stephen Ostermiller's user avatar
4 votes

Is it possible to exclude just OpenAI - ChatGPT from scraping my website?

Yes, it is possible both through a robots.txt declaration and to be restricted at the IP address level. If you want to restrict it via robots.txt, you need the following declarations: User-agent: ...
Maniac's user avatar
  • 324
3 votes

Does Bing execute Javascript when crawling and indexing web pages (like Google)?

As of 2018, the Bing Team clarified that bingbot is generally able to render JavaScript. They do recommend detecting the bingbot user agent and prerendering the content on the server side and ...
Scott Willeke's user avatar
3 votes

BingPreview user agent

I realize that this is an old post but after researching this I came upon an article explaining the purpose of this user agent. In this article they explain that this is a user agent string used to ...
MddHtt13's user avatar
3 votes

Inconsistent site: search results on bing and yahoo

Firstly on yahoo if I use site:https://www.example.com/ using https Yahoo declares 121 results. But site:http://www.example.com/ or site:example.com declares 1300 results. Why would this be? ...
DocRoot's user avatar
  • 4,297
3 votes

Bing stopped indexing the site after disallowing all crawlers except Googlebot, Yahoo-slurp, and Msnbot

Msnbot is just one of the user agents that Bing users, and it isn't even the main one. According to https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0 Bingbot - Main bing ...
Stephen Ostermiller's user avatar
3 votes

Why are Bing and SEMRushBot ignoring crawl-delay in my robots.txt file?

Thanks for the question and welcome to this great community of like-minded individuals. You're query is perplexing however it is solvable. I see a couple of things that can be improved/changed in your ...
Nina Dubovich's user avatar
3 votes

Bing refuses to index my site at all

First up, Webmaster Tools is pretty inaccurate. We have a page on there that has been on Bing since the beginning. It claims it was discovered in 2018 (which isn’t true). The error message is ...
Jack Yan's user avatar
3 votes

Is it possible to exclude just OpenAI - ChatGPT from scraping my website?

they have a special useragent for their scraper now, it’s GPTBot https://platform.openai.com/docs/gptbot i would also recomment blocking Googlebot, since they declared to use everything they can ...
magnus's user avatar
  • 31
3 votes
Accepted

Is it possible to make your website to be searchable with specific keywords?

NO. Webmasters can't definitively do anything to force Googles search to behave a certain way. The best they can do is follow best practices for the moment and provide data optimized in a way that ...
davidgo's user avatar
  • 7,904
2 votes

Do all search engines obey nofollow rel attribute on hyperlinks

From my experience, BingBot does follow links despite them having a nofollow attribute. I can see it currently in our logs: thousands of requests per day for pages that are linked using nofollow ...
Effing's user avatar
  • 151
2 votes

Google SERP penalty for Multiple Business websites on one server?

absolutely no penalty for "multiple websites, all related to the same business-type, on one server and all of them sharing the same IP address". Different if these multiple website are low quality ...
Massimiliano Rubino's user avatar
2 votes
Accepted

Bing + React: Inconsistent indexing

No. You can't really encourage Bing to index JS properly. It seems like unless you use the isomorphic / universal style of coding javascript apps then you're just out of luck. It's a roll of the ...
Hayk Saakian's user avatar
2 votes

Bing webmaster indicating crawl errors on http to https 301 redirect

...my question is why is it looking at http pages when I have https pages in sitemap... This is the key. The answer is simple. Sitemaps are not the authority by which search engines submits pages to ...
closetnoc's user avatar
  • 32.8k
2 votes

How to remove deleted webpage returning 200 from indexers?

At this point, your best bet is probably re-purchasing the domain and removing all DNS records. This way search engines (and users) won't be able to access anything; it will be as if nothing exists. ...
Andrew Lott's user avatar
  • 5,854
2 votes
Accepted

Return message from Bing search engine in English with curl php

By default Bing will reply in the native language that it thinks is correct based on the location of the user. This is why you are receiving a response in the language of the region your server is ...
Analog's user avatar
  • 2,375
2 votes

Bing returns garbed text for PDFs

Looks like you have not optimised the PDF for search engines as there is no meta information on the document. I have added a screenshot of what search engines will be seeing. This can be updated ...
Jason's user avatar
  • 686
2 votes

Remove site from search engines for specific keyword

I dont know any way to unrank a page for spesific keyword, the only think that would work is removing from your page (H1, Title, Descriptions, p) which are related to the keyword. This is what John ...
John Could's user avatar
2 votes

Website deindexed by Bing.com but they won't tell me why

In my opinion, you should not do anything with it. Go back to it next quarter and see if if hasn't straightened itself out. You seem more informed than 75% of websites. I wouldn't worry about this. ...
Mike Ciffone's user avatar
  • 6,360
2 votes

Bing Caching - 125 KB Limit

I was stumbling above the same SEO warning of Bing's webmaster tools. The description you linked (which is still there at time of writing) indicates that it's about the search crawler internal cache, ...
MichaIng's user avatar
  • 121
2 votes

How to inspect and fix Bing's "Crawl errors"

Uncheck the options aside from the errors, and then click the SEO Reports section to see the errors. They should be listed individually, and a lot of them, (missing alt tags, multiple h1, description ...
keepkalm's user avatar
  • 917
2 votes

NextJS site hosted on Heroku not indexed by Bing after 2 months

Bingbot does not execute JavaScript, so any content on your pages that requires JavaScript to execute on the client isn't going to get indexed in Bing. Bingbot effectively sees a nearly blank home ...
Stephen Ostermiller's user avatar
2 votes

Does Bing penalised sites for too many H1 Tags?

Anecdotally? Only Bing can directly answer your question -- but I believe I can indirectly answer it. I think your experience also answers it, although indirectly. Another way to ask that question is ...
Wayne Smith's user avatar
  • 2,636
2 votes
Accepted

Bing Search Performance Keyword and Page Exports don't add up, should they?

I am able to replicate this on my end, the data doesn't add up. In addition, the data appears to come back in different orders depending on what page I click export from in the report's pagination. ...
Mike Ciffone's user avatar
  • 6,360
2 votes

My website's favicon is not displayed in Bing search results

This is a caching issue. It could take awhile to resolve. Be patient. Note that Bing is insistent on requiring the .ico file extension. While Google will use whatever image you've setup even if it is ...
Justin's user avatar
  • 84
1 vote

Calculating requests per second within bing

Aside from setting the "Crawl Control" settings within Bing Webmaster tools you could add a crawl-delay: directive to your robots.txt file. Bing supports whole number values ranging from 1 – 20. ...
Analog's user avatar
  • 2,375
1 vote

How do you filter results by site using Bing?

Bing supports the use of the site: filter in the search bar. I just performed a test using Bing and doing a search for a government department in Australia to filter just on departments website for ...
Chris Rutherfurd's user avatar
1 vote

How I know my site is indexed on AOL or not

AOL search is powered by bing. When you're on AOL.com and search for something, you can use the parameter site:example.com. It should show you if your site is indexed or not. You can submit your site ...
Michael d's user avatar
  • 4,788
1 vote

Bing not indexing URLs From Sitemap

The reason for this is simple. There are two major schools of thought regarding indexing the web. One is to index everything and then decide what to keep. The advantage of this is that all metrics ...
closetnoc's user avatar
  • 32.8k

Only top scored, non community-wiki answers of a minimum length are eligible