Tag Info

New answers tagged

3

First of all /de is not a sub-domain- it is a sub-directory. A sub-domain would be something like de.example.com. And yes. You will need /de/media/sales/ and /de/media/tmp/. You will always be working from the root directory. Lastly, the robots.txt file must be in the root directory. You cannot create a robots.txt in the /de directory. It will never be ...


1

Robots.txt does not block search engines from indexing a link, only the content of that link. So what is likely happening is that Googlebot is visiting your page even though it knows it can't index the contents of that page, only the url. From Google Webmasters: Blocking Google from crawling a page is likely to decrease that page's ranking or cause it ...


3

This is because... User-agent: * Sitemap: https://m.mysite.com.au/sitemap.xml ...would be incorrect. The correct syntax for the robots.txt is: To disallow all: User-agent: * Disallow: / To allow all: User-agent: * Disallow: To disallow specific directories: User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/ Examples taken from ...


0

I would try adding the Allows before the Disallows. Google actually state the order is not important, only the specificity of the rule based on the length of the entry. Order of precedence for group-member records At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will ...



Top 50 recent answers are included