If you want a website to appear in DuckDuckGo for example and not Google, is this possible? Can you specify which search engines for a website appear in?
To block a specific bot you use something like this within the robots.txt file:
To block a specific bot using HTML then include this meta name within the
You can find the user agents listed on Google's Webmaster Help, below you can find all types of bot user agents that Google uses. Note that Adsbot, and Media partners may not crawl your site nor index.
You can also find Microsoft's Bing crawler user agents on their website
DuckDuckGo bot uses the user agent
You can pretty much find any user agent using any good search engine, but to save your time its useful to bookmark this website that lists the majority of crawler user agents.
You can block all search engines by using