Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I want google to index my index page, but do not index any other pages.

User-agent: *
Disallow: /

So far I have this. However...when I do this, Google does not index my index page. When I search my name in Google, all it shows is "www.mydomain.com" as the first result with no description.

Instead, I want Google to index my main index page, WITH the meta description tag, too.

share|improve this question

migrated from stackoverflow.com Jun 29 '12 at 14:11

3 Answers

Google and other search engines support the Allow: statement now in addition to Disallow:... though they may behave differently on different search engine spiders, and may not be supported or enforced by other types of site scrapers.

Documentation here

share|improve this answer

Why would you want to disallow Google your pages?

Anyway yo do this you could do:

Disallow: /
Allow: /index.html
share|improve this answer

For Google, in particular, the following rules will do the trick:

User-Agent: *
Allow: /$
Disallow: /

For details, see Google's documentation of their supported robots.txt syntax. However, note that the middle line is non-standard for two reasons: first, it's an Allow directive (the basic robots.txt standard only supports Disallow) and second, it uses the non-standard end-of-URL anchor $. That said, several other major search engines, including Bing, do support more or less the same syntax.

share|improve this answer

protected by John Conde Mar 24 at 15:46

This question is protected to prevent "thanks!", "me too!", or spam answers by new users. To answer it, you must have earned at least 10 reputation on this site.