8

I want google to index my index page, but do not index any other pages.

User-agent: *
Disallow: /

So far I have this. However...when I do this, Google does not index my index page. When I search my name in Google, all it shows is "www.mydomain.com" as the first result with no description.

Instead, I want Google to index my main index page, WITH the meta description tag, too.

4 Answers 4

8

For Google, in particular, the following rules will do the trick:

User-Agent: *
Allow: /$
Disallow: /

For details, see Google's documentation of their supported robots.txt syntax. However, note that the middle line is non-standard for two reasons: first, it's an Allow directive (the basic robots.txt standard only supports Disallow) and second, it uses the non-standard end-of-URL anchor $. That said, several other major search engines, including Bing, do support more or less the same syntax.

3

Google and other search engines support the Allow: statement now in addition to Disallow:... though they may behave differently on different search engine spiders, and may not be supported or enforced by other types of site scrapers.

Documentation here

0

Why would you want to disallow Google your pages?

Anyway yo do this you could do:

Disallow: /
Allow: /index.html
1
  • 1
    That won't match the usual root which is '/'
    – Bryce
    Commented Dec 21, 2013 at 7:31
0

I use like this:

User-agent: Yandex
Allow: /index.html
Disallow: /

User-agent: Googlebot
Allow: /index.html
Disallow: /