3 votes

Can I have both "disallow nothing" and "disallow something" statements in my robots.txt?

You can, but the second statement renders the first obsolete. Essentially, in human-readable terms, you're saying: All bots Disallow nothing - so crawl anything you want But Disallow the "terms" ...
user avatar
3 votes

Allow all bots except Googlebot which should only be able to crawl the home page

It is already answered in Stackoverflow, you just need to define the useragent name there. Here we only specify Googlebot, so that rule is not applied to other search engines. user-agent: Googlebot ...
user avatar
  • 5,487
3 votes
Accepted

Google indexing Wordpress in subdirectory disallowed in robots.txt

Using Disallow in robots.txt doesn't prevent search engines from indexing pages, only from crawling them - this is a common SEO myth. To answer your questions: Googlebot finds pages not only through ...
user avatar
3 votes

Disallowing path with parameter

All right, I found an answer in docs for how to use robots.txt: URL matching based on path values Robots.txt Specifications This should be fine: Disallow: /Home/SomeBlockedPath
user avatar
  • 207
3 votes
Accepted

Should I add links to the Sitemap that are disallowed in robots.txt?

You should not add URLs to your sitemap unless you want them crawled and indexed. If a URL is disallowed, non-canonical, or redirecting, it doesn't belong in your XML sitemap. Google uses your XML ...
user avatar
2 votes

Indexed, though blocked by robots.txt

Google can still index based on links to these pages. Below is a passage from Google support: (https://support.google.com/webmasters/answer/7440203?hl=en) The page was indexed, despite being blocked ...
user avatar
  • 2,233
1 vote

Do I get back all the search engine rankings I had after having blocked a domain using robots.txt?

Whether you will take a permanent hit or not can't be said with 100% guarantee since Google does what Google wants. However, from what you've described, I don't see a reason they would permanently ...
user avatar
  • 386
1 vote

Block URLs that begin and end in common prefixes and suffixes using robots.txt

So is the variant Disallow: /category1/ */-more$ correct Almost, but you seem to have introduced an erroneous space and slash that is not present in your example, so this won't block the crawling of ...
user avatar
  • 41.2k
1 vote

Can I use robots.txt to allow Google to access some folders, but disallow everything else?

I generally focus on what I don't want them to crawl, but I think your sample seems on target. I would, however, remove the space between the first line and the second: User-agent: * Allow: /index*....
user avatar
  • 811
1 vote

Google can't access my page due to robots.txt but the testing tool says it's OK

When I test it with Google tool everything is ok. The tool is simply telling you that the file is syntactically valid. Whether it blocks (or allows) traffic is another matter. But in Google Search ...
user avatar
  • 41.2k
1 vote

How do I disallow some but not all subdirectories of a folder using robots.txt?

Disallow: /forntend/css/, Disallow:/forntend/js/, Disallow: /forntend/font/ There should be no comma (,) at the end of the path argument. (This isn't an array of elements.) The EOL separates each ...
user avatar
  • 41.2k

Only top scored, non community-wiki answers of a minimum length are eligible