Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

If you want a website to appear in DuckDuckGo for example and not Google, is this possible? Can you specify which search engines for a website appear in?

share|improve this question

1 Answer 1

up vote 6 down vote accepted

You can block Google and other search engines by using a robots.txt file and meta name no index. You can use one or the other but Google recommends that you use both robots and meta.

To block a specific bot you use something like this within the robots.txt file:

User-agent: bot-name
Disallow: /

To block a specific bot using HTML then include this meta name within the <head></head>

<meta name="bot-name" content="noindex, nofollow">

You can find the user agents listed on Google's Webmaster Help, below you can find all types of bot user agents that Google uses. Note that Adsbot, and Media partners may not crawl your site nor index.

  • Googlebot
  • Googlebot-News
  • Googlebot-Image
  • Googlebot-Video
  • Googlebot-Mobile
  • Mediapartners-Google
  • Mediapartners
  • AdsBot-Google

You can also find Microsoft's Bing crawler user agents on their website

  • Bingbot
  • Adidxbot
  • MSNBot
  • BingPreview

DuckDuckGo bot uses the user agent duckduckbot and even lists the ip addresses it uses.

  • duckduckbot

You can pretty much find any user agent using any good search engine, but to save your time its useful to bookmark this website that lists the majority of crawler user agents.

You can block all search engines by using noindex or user-agent:* i.e

Robots.txt

User-agent: *
Disallow: /

Meta Name

`<meta name="robots" content="noindex, nofollow">
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.