Tag Info

New answers tagged

2

Disallow: /article/*/ Yes, this should be OK and work as required in the main search engines. Note that this must appear on one line, not split as in your example. The wildcard * is an extension to the original robots.txt standard and "designates 0 or more instances of any valid character" (according to Google).


3

According to Google's Webmaster Guidelines: Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines. More often than not, these pages can come across as "spammy" and of low value.


2

Googlebot is very well aware of WordPress sites and it's highly unlikely that it will punish a WordPress site for having content on several very common WordPress pages. If you start playing around with robots.txt and rel=unfollow it's easy to shoot yourself in the foot. You could install one of the SEO plugins that only places short versions of articles on ...


0

If you are using IIS, you can add IIS rewrite rules to your web.config to specify which type of robots.txt to return, depending on the subdomain the user (and thus the crawler) is browsing to. You can specify special HTTP_HOST pattern conditions to specify which robots.txt file should be used for which domain. An article which explains this perfectly: ...


3

As you already noticed that bots are crawling your pages again, it is only a matter of time when they will crawl more of your pages and show indexed snippets in their search results. While it will likely not result in any problems, your current robots.txt is invalid according to the original specification, because your record doesn’t contain a Disallow line ...


0

I see on your robots.txt User-agent: * To allow all, you should declare a user agent with a specific directive. Use : User-agent: * Allow: / This will make sure that Google knows that everything under the root folder is allowed to be in the index. Give it a few weeks, Google's crawler deals with dozens of thousands of sites and it does not update in a ...


1

Google Webmaster Tools index count will fluctuate constantly especially with dynamic sites or those using common platforms such as forum software. The best way to tell how many of these URL's Google are dropping from their index is to use the site operator in Google's web search, for example:- site: example.com/forum This will show you all indexed URL's ...


1

Unfortunately, the fastest way to remove pages from the Google index, you abandoned. There is nothing wrong with using the robots.txt file to remove pages from the Google index. Having switched to noindex, it will take some time for the spider to fetch all of the pages and update the index. The speed will depend upon the freshness of your site in the past. ...



Top 50 recent answers are included