1

My website, let's say www.example.com ranks fairly well ranking 1-5 for 6 keywords. On the other hand, my blog on the same website keeps on getting de-indexed.

The Website is HTML (www.example.com), and the blog (www.example.com/blog) is hosted on WordPress.

Some information about the blog: It has extremely good quality original content, got ~700 views organically for each individual blog post. So there's no chance of a copied content or a Google penalty. Google webmasters still shows no message.

There are no server crashes. As hosting is same for both website and blog, we can rule out the possibility of de-indexing because of server crashes.

There's no .htaccess file.

I checked Robots.txt for possible mistake that might have been blocking the search engines. I even cross checked the url using GWT's testing tool that says www.domain.com/blog is not blocked by robots.txt

I have a sitemap.xml.

Although, I am yet to set the preference for www or non www for my domain in GWT. But this isn't something that should have resulted in de-indexing.

I am using https:// for the entire site including blog. While all page appear to be secure, blog does have some sources which are not secure. But, the blog has been that way since a year and nothing happened.

Also, when I add url to fetch as Google, the submit to index option doesn't shows up.

NOTE: My Wordpress settings mistakenly were set to stop search engines from indexing the blog. I changed that a month ago, the blog was indexed, but the pages were not. Shortly, the blog got de-indexed again, I re-indexed it, and got de-indexed again, while other pages never got indexed.

Now the blog is not indexed, however, one of the blog post is indexed. But a google search display the meta description as "A description for this result is not available because of this site's robots.txt – learn more."

What should I do now? I have tried everything. Nothing seems to work.

Improve this question
12
  • Have you checked the Source? Since WordPress doesn't NORMALLY block using robots.txt for deindexing, it uses the source with a noindex attribute. If you share the URL in question it will be much easier for the community to help.
    – Simon Hayter
    May 18, 2015 at 14:03
  • Hi @bybe the domain is www dot simform dot com and the blog is www dot simform dot com slash blog
    – ssp
    May 19, 2015 at 6:31
  • Did you recently add <meta name="robots" content="index, follow">? because that does not look like your loading that via wp_head but rather a manual entrie.
    – Simon Hayter
    May 19, 2015 at 9:03
  • 1
    You shouldn't need to use that all... Did you find nofollow in the head and then used this one?
    – Simon Hayter
    May 19, 2015 at 9:08
  • 1
    If you want faster indexes and updates you need to regularly add pages and freshen pages.
    – Simon Hayter
    May 27, 2015 at 14:36

1 Answer 1

Reset to default
1

Is only your blog in Wordpress?

Anyway: it can happen that there's a conflicting setup in robots.txt, due to plugins (SEO plugins and plugins dedicated to robots.txt) and Wordpress settings mismatch. Try access your robots.txt file from the main folder

www.example.com/robots.txt

and even your blog folder

www.example.com/blog/robots.txt

Then, from Google Webmaster Tools go to robots.txt tester. Click on the option to look at robots.txt on the website, so you can see which robots.txt is GWT really looking at.

I hope you are just blocking /wp-content/plugins/ folder from access, and nothing else.

What about your sitemap? Do you have only one of it in main folder (or at least only one index)? Is it properly linking your blog resources? Use Yoast plugin for sitemap, is very well done.

Also, you should find wordpress .htaccess in its main folder. So, if only the blog is hosted by Wordpress, look with FTP client like FileZilla (not cPanel file manager) in /blog/ folder.

Try to crawl your website using other online tester or software like Screaming Frog.

Improve this answer
6
  • Hi @lucgenti I checked my website with Screaming Frog, no issues came up. I have provided my website link in a comment above. Can you suggest something based upon it? There's no robots.txt for my blog, GWT is currently looking at www.domain.com/robots.txt. For the sitemap.xml, I generated it online, and currently not using yoast. Will not using yoast cause any issue?
    – ssp
    May 19, 2015 at 7:32
  • Did you set up Screaming Frog as Googlebot? If you look at your sitemap, it's listing only main directories, not every webpage. Use Yoast. Its sitemap is good and you have more control in setting index and noindex. Also, if your actual robots.txt is maybe causing trouble, just modify it. Remove everything, except resources you want to keep "secret" like pdfs accessible only through member access (but for that, is better to create a single folder, put your files in it and noindex the folder). Don't worry about /category/ or /trackback/ There's also stuff I don't understand.
    – lucgenti
    May 19, 2015 at 9:08
  • I don't have the licensed version of screaming frog, however the default settings are of Googlebot I guess. But does a bad sitemap.xml actually causes this much damage? I mean, if my pages are already indexed, then will a bad sitemap result in de-indexing?
    – ssp
    May 19, 2015 at 9:21
  • No, your sitemap doesn't cause much damage. Absolutely. But a better sitemap can help bots index your content better. And because we are not sure about the origin of that problem, we need to isolate variables. Tell me when you also "clean" your robots.txt
    – lucgenti
    May 19, 2015 at 11:14
  • Hi @lucgenti I cleaned my robots.txt and the result is still the same. Can you suggest something?
    – ssp
    May 27, 2015 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.