Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I want google to index my index page, but do not index any other pages.

User-agent: *
Disallow: /

So far I have this. However...when I do this, Google does not index my index page. When I search my name in Google, all it shows is "www.mydomain.com" as the first result with no description.

Instead, I want Google to index my main index page, WITH the meta description tag, too.

share|improve this question

migrated from stackoverflow.com Jun 29 '12 at 14:11

This question came from our site for professional and enthusiast programmers.

4 Answers 4

For Google, in particular, the following rules will do the trick:

User-Agent: *
Allow: /$
Disallow: /

For details, see Google's documentation of their supported robots.txt syntax. However, note that the middle line is non-standard for two reasons: first, it's an Allow directive (the basic robots.txt standard only supports Disallow) and second, it uses the non-standard end-of-URL anchor $. That said, several other major search engines, including Bing, do support more or less the same syntax.

share|improve this answer

Google and other search engines support the Allow: statement now in addition to Disallow:... though they may behave differently on different search engine spiders, and may not be supported or enforced by other types of site scrapers.

Documentation here

share|improve this answer

Why would you want to disallow Google your pages?

Anyway yo do this you could do:

Disallow: /
Allow: /index.html
share|improve this answer
    
That won't match the usual root which is '/' –  Bryce Dec 21 '13 at 7:31

I use like this:

User-agent: Yandex
Allow: /index.html
Disallow: /

User-agent: Googlebot
Allow: /index.html
Disallow: /
share|improve this answer

protected by John Conde Mar 24 '13 at 15:46

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?