New answers tagged robots.txt
2
Disallow: / will prevent robots from crawling your entire site. Robots.txt rules are "starts with" rules. Google and other search engines allow some extensions to robots.txt rules such as wild cards and ends with operators. For the robots (including Google) that follow those extensions you could use:
User-agent: *
Disallow: /$
Most robots will still ...
1
To allow a more specific URL you can use the Allow directive (supported by all major search engine bots). For example:
User-agent: *
Allow: /v6.0/i-want-this-page-indexable
Disallow: /v6.0
Disallow: /v5.8
Disallow: /v5.7
Disallow: /v5.5
Disallow: /v5.1
Disallow: /v5.0
The order of these directives doesn't strictly matter for most search engine bots (...
0
Since it's only temporary, you should use 302 redirect instead of 301. But my advice is don't use auto-redirect, but inform the visitor in existing/old website to try a new platform with a popup every visitor visit it or something else.
My worry is that all new content will be indexed against a subdomain
Why are you worrying about it? I don't see any ...
5
Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site.
An old site
These URLs could be from a previous owner of your domain name, or from a previous version of the site that you have since taken down. Once bots discover URLs, they ...
Top 50 recent answers are included
