A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

New answers tagged

2

To allow a more specific URL you can use the Allow directive (supported by all major search engine bots). For example: User-agent: * Allow: /v6.0/i-want-this-page-indexable Disallow: /v6.0 Disallow: /v5.8 Disallow: /v5.7 Disallow: /v5.5 Disallow: /v5.1 Disallow: /v5.0 The order of these directives doesn't strictly matter for most search engine bots (...


4

No. Google would not take the right to do that (I don't think any browser has done such). Also many "hidden" pages would anyway not be accessible; i.e. I may have many pages on my computers that are 100% private; also many websites have pages that are not accessible unless you first create an account and log in. However, would Google be able to eventually ...


1

This line: Page fetch: Failed: Crawl anomaly Indexing looks like the underlying problem. For some unknown reason (a hiccough in the line at your end or theirs), Google wasn't able to fetch this file. The values in the other lines are consistent with this. Google will eventually search it again, but you can request that it happen sooner (within a few ...


5

Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site. An old site These URLs could be from a previous owner of your domain name, or from a previous version of the site that you have since taken down. Once bots discover URLs, they ...


Top 50 recent answers are included