Wordpress site submitted to Baidu doesn't get crawled

Question

I've got a website (Wordpress blog, custom .net domain, hosted in Germany) that I've submitted via the Baidu webmaster tools, via all of the available tools, that is, direct URL submission, sitemaps and the JavaScript snippet that auto-submits new links. For all of the manual submissions there have been no errors and nothing indicates any problems - except, however, that it simply doesn't get crawled at all. It all works fine on both Google and Bing though.

The page is using HTTPS (Let's Encrypt) and there's an automatic redirect to HTTPS in place, however as far as I understood the documentation that should be okay, and again, the report on HTTPS also indicated no problems with the certificate. Also, the page is accessible without a VPN, so I didn't suspect anything wrong on that front.

Are there any good pointers on how to even start debugging this?

---

robots.txt:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Answer 1

Whoa boy, this approach and Baidu won't play nice at all....

1) Check to see if the domain is blocked by the "Great Firewall" with https://en.greatfire.org/analyzer. If so, sorry, there's something that's "objectionable" and you likely will get your site blocked again no matter what you do to try and get around the censors. But if not:

2) See if the domain is accessible via China-situated clients with http://www.webkaka.com/webCheck.aspx. If not, try and mirror the site on a Chinese CDN and use the China ccTLD while you're at it. If you get a bunch of 200's, then:

3) Make sure the crawler can render your page correctly by masquerading as Baidubot. A virtually blank page because of poor rendering will be stopped before being indexed even it passes the tools' reports somehow. Use a Chrome extension or https://technicalseo.com/tools/fetch-render/. If it looks really funky, then remove as much Javascript, Ajax, whatever "rich" experience web tech you're using. If it passes, then:

4) Try pushing your URLs to Baidu via their API. This is not easy.

• Follow the steps here to get an API key; it involves gaining access to a Chinese phone number!: https://stackoverflow.com/a/29469842/11985090
• Update your WP functions.php with the code from https://topic.alibabacloud.com/a/wordpress-real-time-push-new-link-to-baidu-to-achieve-a-rapid-collection-of-methods_8_8_20029891.html and remember to add the API key in the appropriate place.
• Slowly update pages over time to trigger the API pushes via the new php.

Answer 2

I've had the same problem while launching a site for a client there. When submitting any site to Baidu, it expects a fully chinese site hosted in China. If you can't provide a fully translated Mandarin version for your Wordpress site, try adding this to your html:

<META HTTP-EQUIV="Content-Language" CONTENT="ZH">

Place the above meta tag on atleast one of your html/php pages, it lets Baidu know you have a chinese site which is a requirement according to their community guidelines and resubmit your site again.

Good luck!

Answer 3

I haven't specifically dealt with this problem so I'm "thinking out loud" as I write this:

I've never been especially concerned with Baidu but my understanding is that it is a Chinese social media site so:

• Do they allow users to create subdomains (like Google Sites, Blogger, WordPress.com etc)?
• Do they allow users to create maps listings?
• Is there a Chinese equivalent to Craig's List where you could post an ad with a backlink?
• Do they allow any type of paid ads (that would effectively alert them to a new advertisers website)?

If none of the above seem viable, I have to ask a few other follow up questions:

• Is the content in any way objectionable to the Chinese government?
• Is the content in any way likely to be flagged programmatically (skin tones, like porn)?
• Is the content in Chinese and have you implemented hreflang tags?

If contemplating the above doesn't suit your needs, have you looked into other means of promotion (programmatic and manual) such as:

• Expanding the list of services to "ping" whenever you update your content (in the WordPress admin area) with particular attention to servers likely to be relevant to the Chinese market?
• Doing outreach via social media to people in China?
• Commenting on high profile blogs in your industry that are likely to be spidered by Baidu?
• Submitting your website to Chinese directories, bookmarking sites, URL shorteners, etc?

Last but not least:

• (and this builds upon the last question) Have you tried building links from sites that Baidu is likely to spider, and "pinging" those links (i.e. tiered link building)?

Edit Your Robots.txt file

It's also probably not a bad idea to remove any references to WordPress such as wp-admin from your robots.txt file. I believe WordPress automatically tells search engines not to crawl wp-admin with a "noindex" directive (which might be an HTTP header, rather than a meta tag).

And even if it doesn't, do you really want your robots.txt file to announce to the world that you have a website running wordpress?

Don't get me wrong, I love WordPress and most of my websites utilize it, but it's popularity as a CMS also amounts to publicly announcing it's vulnerabilities from a security standpoint.

Try robots.txt

User-agent: *
Allow: /