User-agent: * Disallow: / User-agent: Googlebot Allow: /
I use this command in my robots.txt file. But i do not think this is right. What should be the right command? Some article i have found where told not to do this in robots.txt
#Code to not allow any search engines! User-agent: * Disallow: /
And also found that we should disallow Googlebot except the js and css file.
User-agent: Googlebot Allow: /*.js* Allow: /*.css* Allow: /google/
So what should be the right way to do?