4

I have been trying for months now to get Bing to index my personal website, to no avail. I'm not talking about ranking, as I expect it to rank quite low. It is invisible. I don't care about Bing in general but as it's the upstream index for DuckDuckGo this causes discoverability issues. Google has no issue at all with my site, 10+ years running.

You can check this for yourself:

W3C Validator reveals no issues with the markup, its entirely valid and minimal. The site is fast, contains no scripts or external dependencies and exclusively original content written by me. I do not sell anything or use this for any kind of direct marketing. It's my personal website and technical blog. robots.txt of course allows crawling by Bing, I have manually verified this. Sitemaps exist and are submitted.

For most pages I tried using their URL Inspection in Bing webmaster Tools, I see this:

Bing rejects

Note the date of first discovery. The URL for the screenshot is https://herrbischoff.com/2019/11/advids-spam-email-and-what-to-do-about-it/, which is the #1 Google result for the search term "advids spam" and has been at least since July 2020.

Sure enough, using the "Live URL" tab, Bing recognizes none of the issues it apparently found at some unspecified time earlier:

Bing agrees

I have submitted the page and site a dozen times over the course of the last couple of months. I have been in contact with Microsoft support which manually reviewed my site, found no issues, escalated the issue and weeks later ultimately responded with a generic "please follow the Bing Webmaster Guidelines". No further explanation, no comment on whether it's a technical issue or content-related, nothing.

Some pages are even actively blocked by Bing, for no discernible reason. Again an example (sadly not the only one) of a page that is the #1 Google search result when searching for its headline contents "mutt How to Open HTML Email in Safari":

Bing blocks

This crosses the border towards the ridiculous for me. At this point I feel I'm being penalized for something I do not understand and have no chance to. To the degree possible for me to grasp, I do follow the guidelines, by choice do not employ any SEO tactics or shady practices at all and run a clean, rather low traffic site.

If anyone has any substantial idea what's going on here, I'd be very thankful for sharing that arcane knowledge.

Update

  • Lighthouse score: 100 / 94 / 100 / 91 / -
  • Ahrefs health score: 100

Update 2

Improve this question
11
  • In my experience, when it says there are some issues, there ARE some issues. It is also hard sometimes to figure it out. Have you tried using a website checker site? There are many free ones out there. The problem isn't always incorrect html or css.
    – Rohit Gupta
    Jul 24, 2022 at 2:56
  • 1
    @RohitGupta, if there are issues so serious they prevent indexation entirely, Microsoft should be upfront about what they are. Especially after a human review who apparently confirmed my site not to be a questionable source. My hunch is there are certain opaque, unexplained "quality requirements" where my site does not check their hidden boxes. Which I find unacceptable but have very little recourse to change.
    – herrbischoff
    Jul 24, 2022 at 13:43
  • Well that's how it is, it took me 4 months before I could get all my pages indexed with Bing. What I found helpful was Yandex, they are even stricter. But they tell you what's wrong and they will respond to emails for clarification. I can't remember details on which had what criteria. But one of them, for instance wanted a large favicon as the first entry in the meta tags. One was also very particular about the length of description and title.
    – Rohit Gupta
    Jul 24, 2022 at 15:16
  • You didn't answer if you had used website checkers - they can find an awful lot of criteria that the search engines are particular about. I mainly use ahrefs (its free for small websites) and a few others.
    – Rohit Gupta
    Jul 24, 2022 at 15:16
  • I have updated my question with more information.
    – herrbischoff
    Jul 24, 2022 at 16:56

3 Answers 3

Reset to default
3

First up, Webmaster Tools is pretty inaccurate. We have a page on there that has been on Bing since the beginning. It claims it was discovered in 2018 (which isn’t true). The error message is identical.

Bing claims that there are description issues and missing H1 tags, yet the page they rank highest for our trade mark (also one of ours) has an obscenely long description and no H1 tags.

Yours is even cleaner than mine and your HTML is so simple, I wish more would do it your way. I wouldn't take Bing Webmaster Tools as definitive.

I have been looking into this for the last two weeks and there are webmasters all with the same complaint, including escalations that result in nothing. (They can be found on Microsoft’s own site, among others.)

They typically refer you to their guidelines and every webmaster swears they are being followed.

The most likely explanation to me—and this is not what others who voted down and deleted my last reply wanted to hear—is that there is something wrong with Bing itself. It’s why your escalations are getting dropped.

SInce I was accused of link spamming, there is a Dutch website by an academic that gives the size of the web statistically. I have sent you the link privately. It estimates that Bing’s index is currently at around one milliard pages—that’s tiny, as search engines in the 2000s exceeded this.

If you do site: searches for any website, Bing is running at a tiny fraction of Google’s index. Even Mojeek’s is bigger and they’re a newer player. These are existing sites, so I question if newer ones will even get indexed.

Here are some to which I have no connection, and I presume they have their HTML and robots.txt all correct, and that they follow the guidelines.

Die Zeit
Google: 2,600,000
Mojeek: 4,796 (0·18 per cent of Google’s total)
Bing: 3,770 (0·15 per cent of Google’s total)

The New York Times
Google: 36,200,000
Mojeek: 2,823,329 (7·80%)
Bing: 1,190,000 (3·29%)

The Rake
Google: 11,500
Mojeek: 1,443 (12·55%)
Bing: 49 (0·43%)
 
Travel & Leisure
Google: 28,100
Mojeek: 9,750 (34·70%)
Bing: 220 (0·78%)

Annabelle (Switzerland)
Google: 11,700
Mojeek: 405 (3·46%)
Bing: 105 (0·90%)

With your own site:
Google: 132
Mojeek: 1 (0·76%)
Bing: 0

I noticed that many independent publications tend to sit at around the 50-page mark, so The Rake is pretty typical.

Of our pages, the majority that Bing shows in its index are HTML and not PHP, but some PHP ones have made it.

The only other strange observation is that I was at one point using Cloudflare's IndexNow, which Bing indexes. Cloudflare claims to send new pages to Bing, yet I never saw that. The pages that were being sent were all old (1999 and early 2000s) plus some PDFs. Those would not be pages people would be reading in 2022. Some of them did indeed wind up on Bing: there is a correlation between the Cloudflare list and what appears in Bing’s index.

If you can find a service that does IndexNow, it might be an unexplored route—though why they would work and others don’t, I have no idea.

Improve this answer
1
  • 2
    On its face, this is quite interesting (and irritating) information. The website mentioned is worldwidewebsize.com, for reference. I cannot immediately judge how reliable this information is but is does appear to show a rather telling pattern.
    – herrbischoff
    Jul 28, 2022 at 10:11
1

I think you should re-evaluate your robots.txt. You have it set up to block everything, and then selectively allow certain crawlers. While you do allow Bingbot, you also block Bing ads and Bing preview further down which could cause issues.

There's really no reason to block everything by default. Firstly, it prevents you from being indexed in any other search engine that you don't know about yet, as well as social media (for things like link previews).

If you have serious problems with a certain crawler then go ahead and block them (although most problematic crawlers aren't going to obey robots.txt anyway so you'd need to IP block). But I'd start with this:

User-agent: *
Disallow:

And only add bots if/when they are a problem.

Another thing to look at would be getting more organic links to your site. A quick site: query shows a few but all the ones I saw were "nofollow" (they have the rel="nofollow" attribute which prevents crawling). There must be some organic links out there since Google found and crawled the site, but a few more would signal to Bing that your site is worth looking at. You could try getting someone to write about your site/business, or you could add a link to a client site (with permission of course).

Improve this answer
1
  • 2
    While you certainly raise valid points from a SEO perspective, I don't see why blocking all but the ones I want should cause issues with Bing. Except if Bing penalized excluding their other crawlers and bots. Their Webmaster tools show all URLs to be allowed and accessible by Bingbot. My experience with other sites show this to be unlikely as well. I am responsible for several sites that use identical directives and are indexed by Bing just fine. So there must be something else going on.
    – herrbischoff
    Jul 26, 2022 at 10:26
-1

I tested your page on a few selected websites.

  1. Many timed out
  2. 2 of them said - blocked by robots.txt
  3. And you have a problem with your security headers, summary below
Results for herrbischoff.com
2022-07-28 01:18:34 Etc/UTC
HTTPS by default:  Yes
Content Security Policy:  Implemented, but has problems
Referrer Policy:  Referrers leaked
Cookies: 0
Third-party requests: 0
IP address: 88.99.124.138Look up

Link to Full Results

As I have suggested in the comment, you need to test you website with as many sites that you are able to. They all have something to add.

Note - I am not affiliated with this or any other website checking sites.

Improve this answer
3
  • 1
    I have updated my question with more information. The results this "website checker" complains about are irrelevant. They are entirely optional settings (nice to have) but not required at all for proper operation. Compare results for The New York Times, Google front page and Bing front page. The fact that you have encountered many blocks exposes the quality of those tools.
    – herrbischoff
    Jul 28, 2022 at 9:13
  • @herrbischoff - All my pages for all 6 of my websites are on Bing now, except for 2 which will be soon. It is up to you what you want to do. As I said, every time a try a new website checker, I find something else that it's not happy about, so I tweak it and resubmit to Bing, Yandex and Google. And if you are sure you have done everything correctly, then maybe you can explain why Bing is not indexing some of your pages.
    – Rohit Gupta
    Aug 1, 2022 at 23:04
  • Bing is not indexing "some" pages, it is not indexing any, which is the entire point of my question.
    – herrbischoff
    Aug 1, 2022 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.