Tag Info

New answers tagged

2

According to Google's Webmaster Guidelines: Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines. More often than not, these pages can come across as "spammy" and of low value.


2

Googlebot (the web crawler of Google) can discover your page by different ways: if it finds a link to your page from a web page (inside or outside your domain name) if you submit the URL to your Google Webmaster Tools account through a sitemap.xml In that case, the page can be indexed. In most of cases, Googlebot decides to index it (for example, it's ...


0

No, Google doesn't automatically index your pages, if no other site links to them. You should put a robots.txt file on the root directory of your server, which can prevent the search engine from indexing the pages of a specific folder. It could look like this: User-agent: * Disallow: /


1

If web crawlers get to this page one way or another, and if it is not marked as noindex and it is not blocked in your robots.txt, then yes, they may decide to index it. You may also use noarchive to make sure they don't keep a cached version of it.


1

Google Webmaster Tools index count will fluctuate constantly especially with dynamic sites or those using common platforms such as forum software. The best way to tell how many of these URL's Google are dropping from their index is to use the site operator in Google's web search, for example:- site: example.com/forum This will show you all indexed URL's ...


1

Unfortunately, the fastest way to remove pages from the Google index, you abandoned. There is nothing wrong with using the robots.txt file to remove pages from the Google index. Having switched to noindex, it will take some time for the spider to fetch all of the pages and update the index. The speed will depend upon the freshness of your site in the past. ...


0

You can log into google webmaster tools and re-index your site. Found that is speeds up the process.


2

I'd probably run a crawler to collect all the URLs of your 1.5 site (a sitemap generator, or link checker should do this for you). When the site is live, Joomla 2.5 has a redirect manager (under components) which lists all the URLs which have thrown a 404. You should use the redirect manager to set up 301s on all the pages you know have changed. If you ...



Top 50 recent answers are included