Is it possible to get Google to stop crawling certain pages without using robots.txt?
My understanding is robots.txt can block Google from accsesing a page, but if using noindex (meta or header) Google may still crawl the page.
As I have millions of pages I want to block Google from ever accessing certain pages, to focus it's indexing on my more important pages.
My site had over 11 million pages in English. Many of these contained statistical information, for comparisson consider a site listing demographic information by postal code/ZIP. Due to requests I translated one type of page which accounts for most of the pages into several languages. This gives me a total of about 100 million pages. Around 250,000 of them have paragraph text on them - the rest just have statistical content.
According to Google Webmaster Tools, Google has indexed around 30 million pages. Many of the pages may be indexed in German or French, but not English. Since English is my main language I want to instead block all the non-English pages from Google expect non-English pages that have paragraph text; which I think will leave Google to largely index my new update of close to 30 million pages in English.
I believe the only way to do this would be to structure the site something like this:
domain.tld/en/ <- index everything in here domain.tld/de/ <- index everything in here domain.tld/dex/ <- block with robots.txt
So my pages with German paragraph text would be available via urls beginning /de/ and those without /dex/.