3

I migrated my wordpress site from godaddy to AWS and converted to a php site with wordpress into a sub directory, blog. My main site has few pages. When I look at content keywords after few weeks in google webmaster , to my surprise instead of keywords which my site was ranking, I see these below

google webmaster content keywords

If I click on any of these I can see that the crawler is going to these sub directories in wordpress blog and counting image extension .jpg in keywords.

enter image description here

To avoid this situation, I added a robots.txt in blog sub directory to avoid this happening

user-agent: *
Disallow: /blog/wp-admin/
Disallow: /blog/wp-includes/
Disallow: /blog/wp-content/
Disallow: /blog/comments/feed/

It has been more than a month and I still see the content keywords to be same. I wanted to review my robots.txt and few sites like yoast suggested that the robots.txt should allow /blog/wp-content/

I am kind of confused as to what should go in robots.txt for wordpress. If I allow /blog/wp-content/ , then I end up situation like above.

My question is if this is the right approach and is my robots.txt for wordpress correct ?

Improve this question
3
  • 2
    This list from Google Search Console is junk and does not hardly ever change much. Instead, I would pay attention to how your site is actually found. You will likely find that none of these terms are used to find your site. I have been critical of this page/feature from Google for a long time. It is not only misleading, it is almost always completely wrong.
    – closetnoc
    Commented Apr 11, 2016 at 2:05
  • 1
    It looks like you have directory indexes enabled? What do you get when you access /wp-content/uploads/2011/03/? If you see a generated directory index then that's your problem - where the keywords are coming from. You probably want to return a 403 forbidden (the default when directory indexes are disabled). But also, the Google report does not reference the /blog subdirectory that's included in your robots.txt file - why is that?
    – MrWhite
    Commented Apr 11, 2016 at 8:11
  • ah that is very good point.
    – vsingh
    Commented Apr 11, 2016 at 13:51

3 Answers 3

Reset to default
2

Most posts about WordPress robots.txt are totally wrong.

You should not block this in robots.txt

I suggest to use only this robots.txt for WordPress blog:

User-agent: *
Disallow: /blog/?s=*

If you block WordPress directories like admin and includes and content, then you're going to block CSS, JavaScript and other assets for Googlebot, and that will result in it not rendering correctly.

By the way, directories like wp-admin will not be crawled because it is password-protected, you can read my answer on other question in regards to this.

Content keywords which are displayed in Google's Search Console indicates that you're using these keywords the most in your website, but it does not have any relationship with keyword ranking or traffic. In general I only checkout that data to see whether my site contains any incorrect keywords because that can happen when your site is hacked and someone inserts other keyword stuff, but if you're using this for ranking analytics, then you're probably using it incorrectly.

Improve this answer
1
  • I agree with rankings. My rankings have not been affected.
    – vsingh
    Commented Apr 11, 2016 at 13:52
1

From http://www.wpbeginner.com/wp-tutorials/how-to-optimize-your-wordpress-robots-txt-for-seo/ 

User-Agent: *
Allow: /?display=wide
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /readme.html
Disallow: /refer/

Sitemap: http://domain.com/post-sitemap.xml
Sitemap: http://domain.com/page-sitemap.xml
Sitemap: http://domain.com/deals-sitemap.xml
Sitemap: http://domain.com/hosting-sitemap.xml
Improve this answer
5
  • I saw this post from wp-beginner, but my point is that if I allow /wp-content/uploads/ then the keywords density gets affected
    – vsingh
    Commented Apr 11, 2016 at 1:21
  • In wp-content/uploads there isn't even any content is there? It is just images. So how does that affect SEO?
    – Steve
    Commented Apr 11, 2016 at 1:30
  • We ll I think that is what google is crawling and considering them in keyword density. I have attached the screen shot from google webmaster above.
    – vsingh
    Commented Apr 11, 2016 at 1:33
  • Ahhh...I see what you mean. That is indeed odd. Stupid question, but do the images have standard format names e.g. imagename.jpg and not something like imagename.jpg.jpg.
    – Steve
    Commented Apr 11, 2016 at 1:54
  • 1
    @vsingh There is no such ting as keyword density. It is made up SEO B.S. It is beyond a myth. Google does not make keyword matches. It never has. Google began as a semantic search engine. Granted, it was not very sophisticated in the beginning, but it most certainly is now.
    – closetnoc
    Commented Apr 11, 2016 at 2:32
0

Try below robots.txt

User-agent:  *
Disallow: /blog/cgi-bin/
Disallow: /blog/wp-admin/
Disallow: /blog/archives/
Disallow: /blog/*?*
Disallow: /blog/*?replytocom
Disallow: /blog/comments/feed/
Disallow: /blog/*/trackback/*
Disallow: /blog/*/xmlrpc.php
Disallow: /blog/wp-content/uploads/*

sitemap: http://www.yourdomain.com/sitemap.xml
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.