I migrated my wordpress site from godaddy to AWS and converted to a php site with wordpress into a sub directory, blog. My main site has few pages. When I look at content keywords after few weeks in google webmaster , to my surprise instead of keywords which my site was ranking, I see these below
If I click on any of these I can see that the crawler is going to these sub directories in wordpress blog and counting image extension .jpg in keywords.
To avoid this situation, I added a robots.txt in blog sub directory to avoid this happening
user-agent: * Disallow: /blog/wp-admin/ Disallow: /blog/wp-includes/ Disallow: /blog/wp-content/ Disallow: /blog/comments/feed/
It has been more than a month and I still see the content keywords to be same. I wanted to review my robots.txt and few sites like yoast suggested that the robots.txt should allow /blog/wp-content/
I am kind of confused as to what should go in robots.txt for wordpress. If I allow /blog/wp-content/ , then I end up situation like above.
My question is if this is the right approach and is my robots.txt for wordpress correct ?