1
User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

I use this command in my robots.txt file. But i do not think this is right. What should be the right command? Some article i have found where told not to do this in robots.txt

#Code to not allow any search engines!
User-agent: *
Disallow: /

And also found that we should disallow Googlebot except the js and css file.

User-agent: Googlebot
Allow: /*.js*
Allow: /*.css*
Allow: /google/

So what should be the right way to do?

  • What is your goal? – unor Oct 25 '17 at 0:41
1

I was quite confused with the question. If I were you, I think it's better to use a single user-agent, let's say assigning * to denote all bots. Then you can use the disallow function to the directories that aren't supposed to be visited by these bots.

User-agent: *
Disallow: /folder1/
Disallow: /folder2/

Please also take note that blocking the CSS and some scripts will also affect how Google sees your site - if there's a CSS for the responsive theme, your site might lose the mobile-friendly factor Google is looking after. Also, try running a search on your domain on the Google search bar and see if their bot can see it.

0

Googlebot should understand your Allow: directive, but that is not the standard way to to allow crawling. The standard way to allow crawling is to disallow nothing. I'd use:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

This is documented in the "To allow all robots complete access" example on the official robots.txt site: http://www.robotstxt.org/robotstxt.html

  • Thank you for Answer. I want to sure about one thing. Use Wordfence Security plugin for my Wordpress Website. Here live traffic, coming in my website shows in for Categories(Human, Bot, Warning, Blocked). where some of them not appears in Analytic. those traffic indicates as Bot traffic and also blocked by the plugin also. So i am confuse which kind of Bot this is? generating this type of traffic. The traffic also increase the CPU usage high. So i assuming to block Google bot as well all bot to see the result. What you think behind this type of Bot generate traffic... – Mourin Oct 25 '17 at 11:30
  • You don't want to block Googlebot if you want your site indexed in the Google search engine. Instead of trying to block bot traffic you should improve the performance of your site. I'v found that WordPress caching plugins such as W3 Total Cache do wonders at making WP run better. – Stephen Ostermiller Oct 25 '17 at 11:37
  • Thank you. But it's being a very big headache, CPU usage being higher within seconds. And the server community deactivate my site for this reason. My site SEO all good maintaining search engine rules and regulation,update plugin, update wp version, minimum number of plugin. So why this type of traffic occurring. Search here and there but no exact solution. How can i fix it? and i use wp super cache plug in for caching problem. – Mourin Oct 25 '17 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.