Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want google to index my index page, but do not index any other pages.

User-agent: *
Disallow: /

So far I have this. However...when I do this, Google does not index my index page. When I search my name in Google, all it shows is "www.mydomain.com" as the first result with no description.

Instead, I want Google to index my main index page, WITH the meta description tag, too.

share|improve this question

migrated from stackoverflow.com Jun 29 '12 at 14:11

This question came from our site for professional and enthusiast programmers.

For Google, in particular, the following rules will do the trick:

User-Agent: *
Allow: /$
Disallow: /

For details, see Google's documentation of their supported robots.txt syntax. However, note that the middle line is non-standard for two reasons: first, it's an Allow directive (the basic robots.txt standard only supports Disallow) and second, it uses the non-standard end-of-URL anchor $. That said, several other major search engines, including Bing, do support more or less the same syntax.

share|improve this answer

Google and other search engines support the Allow: statement now in addition to Disallow:... though they may behave differently on different search engine spiders, and may not be supported or enforced by other types of site scrapers.

Documentation here

share|improve this answer

Why would you want to disallow Google your pages?

Anyway yo do this you could do:

Disallow: /
Allow: /index.html
share|improve this answer
1  
That won't match the usual root which is '/' – Bryce Dec 21 '13 at 7:31

I use like this:

User-agent: Yandex
Allow: /index.html
Disallow: /

User-agent: Googlebot
Allow: /index.html
Disallow: /
share|improve this answer

protected by John Conde Mar 24 '13 at 15:46

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?