5
Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site.
An old site
These URLs could be from a previous owner of your domain name, or from a previous version of the site that you have since taken down. Once bots discover URLs, they ...
4
No. Google would not take the right to do that (I don't think any browser has done such). Also many "hidden" pages would anyway not be accessible; i.e. I may have many pages on my computers that are 100% private; also many websites have pages that are not accessible unless you first create an account and log in.
However, would Google be able to eventually ...
2
To allow a more specific URL you can use the Allow directive (supported by all major search engine bots). For example:
User-agent: *
Allow: /v6.0/i-want-this-page-indexable
Disallow: /v6.0
Disallow: /v5.8
Disallow: /v5.7
Disallow: /v5.5
Disallow: /v5.1
Disallow: /v5.0
The order of these directives doesn't strictly matter for most search engine bots (...
1
Google is slowly moving all sites to mobile first indexing. i.e. Smartphone crawler.
I believe the decision to switch a site to mobile first is based on their analysis of the mobile v desktop versions. If they look similar the switch will probably be quick. If it looks like the site will lose out due to the switch, they may wait more time.
1
This line:
Page fetch: Failed: Crawl anomaly Indexing
looks like the underlying problem.
For some unknown reason (a hiccough in the line at your end or theirs), Google wasn't able to fetch this file.
The values in the other lines are consistent with this.
Google will eventually search it again, but you can request that it happen sooner (within a few ...
Only top voted, non community-wiki answers of a minimum length are eligible
