in detail I just want my website to get ranked in google but i don't want third-party bots to crawl websites. I am a beginner in SEO so any help is appreciated. thanks in advance.

New contributor
Hari Prasad is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • This is really quite a broadly-scoped topic that depends on your server environment. It would also be a really time-consuming activity to do since there are countless bots besides Google. It's best just to use robots.txt directives to try to limit the ones that support that. There may be other modules or plugins that you can use as well, though we're not able to recommend them here.
    – dan
    Nov 22 at 9:44
  • @dan I think that a simple robots that text answer would suffice for this question. Given that robots that text is available I'm not sure the answers would need to go into any server side modules. Nov 22 at 12:15
  • This can be done in the general case but if a bot wants to it can defeat the crawling by faking its browser identification string and/or ignoring robots.txt.
    – davidgo
    Nov 23 at 23:11

Browse other questions tagged or ask your own question.