3

It is already answered in Stackoverflow, you just need to define the useragent name there. Here we only specify Googlebot, so that rule is not applied to other search engines. user-agent: Googlebot Allow: /$ Disallow: / As Google said (at the end of page). Use $ when you want to Match URLs that end with a specific string. Here /$ will say allowed to index ...


3

Using Disallow in robots.txt doesn't prevent search engines from indexing pages, only from crawling them - this is a common SEO myth. To answer your questions: Googlebot finds pages not only through links and the sitemap, but also by reading the page's source code for URLs in JavaScript code, etc. It's quite clever. I wouldn't worry too much about API ...


3

All right, I found an answer in docs for how to use robots.txt: URL matching based on path values Robots.txt Specifications This should be fine: Disallow: /Home/SomeBlockedPath


3

You can, but the second statement renders the first obsolete. Essentially, in human-readable terms, you're saying: All bots Disallow nothing - so crawl anything you want But Disallow the "terms" directory - don't crawl that Sitemap here Above, you can take out the second line, to tell the bots to not crawl the "terms" directory. Thus, this would work: ...


2

Google can still index based on links to these pages. Below is a passage from Google support: (https://support.google.com/webmasters/answer/7440203?hl=en) The page was indexed, despite being blocked by your website's robots.txt file. Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page. Google ...


1

Whether you will take a permanent hit or not can't be said with 100% guarantee since Google does what Google wants. However, from what you've described, I don't see a reason they would permanently penalize you (this is my professional opinion, not a promise nor based on Google's input). As for the time it would take to recover there can't be guarantees ...


1

So is the variant Disallow: /category1/ */-more$ correct Almost, but you seem to have introduced an erroneous space and slash that is not present in your example, so this won't block the crawling of these URLs. To disallow all the urls in the /category1/ section and ending in -more? The * "designates 0 or more instances of any valid character", ...


1

I generally focus on what I don't want them to crawl, but I think your sample seems on target. I would, however, remove the space between the first line and the second: User-agent: * Allow: /index*.php Allow: /folder1/ Mozilla explains it in more detail


1

When I test it with Google tool everything is ok. The tool is simply telling you that the file is syntactically valid. Whether it blocks (or allows) traffic is another matter. But in Google Search Console I get an error that my site can't be accessed. User-agent: * Disallow: / As already mentioned in comments, this blocks everything and allows nothing! To ...


1

Disallow: /forntend/css/, Disallow:/forntend/js/, Disallow: /forntend/font/ There should be no comma (,) at the end of the path argument. (This isn't an array of elements.) The EOL separates each directive. The directive is prefix-matching, so any trailing , (comma) will match a literal comma. It's not required, but you should leave a space after the ...


Only top voted, non community-wiki answers of a minimum length are eligible