New answers tagged

1

I've noticed that Google does the same thing to some of my sites, but they don't actually index the content they've crawled. Read here: https://support.google.com/webmasters/answer/35308?hl=en According to Google, using that in the robots.txt prevents them from indexing the content, it doesn't say that it will stop crawling it entirely.


1

Allow: /services/online.html Disallow: /*?dir= The most specific rule (based on the length of the path argument) wins when resolving Allow: / Disallow: conflicts - regardless of the order of the directives in the file. So, for the given URL, the first rule wins because it is the most specific path that matches the requested URL. To resolve this you can ...


0

It appears that you are looking forward to block URL with appearing with sorting parameters. The best thing to do is add canonical tags. It the recommended method Coming to the above, The robots test in webmaster often takes time to update and display the new data add in live robots.txt page. Please confirm that it has been updated.


0

Crawling for the Internet Archive is done both by Alex and by the Internet Archive's own crawlers. Support for the Crawl-Delay directive in the robots.txt file is vairly hap-hazard between the two due to the directive not being part of the official robots.txt standard. In addition the way both companies treat the Crawl-Delay directive when they do accept it ...


2

What I would suggest is to create a brand new set of URLs for search result pages and then for anyone requesting the old URLs, produce an error page with an HTTP 410 status code. Also, make your search pages only accessible via the POST request method. Google won't crawl pages that are requested via POST if it is requested as a result from filling out a ...


2

Google supports Noindex: in robots.txt as an experimental feature. It sounds like this would be the perfect case for using it: User-Agent: * Disallow: /search/ Noindex: /search/


1

Google won't crawl the disallowed path specified in robots.txt but you can't control through robots.txt the references to your site search results from other sites. While Google won't crawl or index the content blocked by robots.txt, we might still find and index a disallowed URL from other places on the web. As a result, the URL address and, ...


1

To answer this, you have to know something of the mechanics behind how all of this works. Every time a link is discovered, it is entered into a link table within he index. The source URL, the page with the link, is at least already within the URL table within the index. If the target URL, the page being linked to, is not in the index, it is a dangling link. ...


0

It makes sense to split your question up in two parts, each addressed separately. Can backlinks to robots.txt blocked pages boost domain's ranking? No, these backlinks shouldn't be affecting your domain's authority and thus rankings if they are pointed to pages which are blocked using robots.txt. However, I can imagine search engines not completely ...


1

If you stop the wrong URL´s from gettings accessed on your page, you might lower traffic or render backinglinks ineffective, yes Example:If you block traffic to your e.g. contact page(/contacts.html), google and other search engines will get to know this and stop linking your contact page You should only block URL´s on your robots.txt which´s content you ...



Top 50 recent answers are included