The process of recording where particular words occur. In the web context search engines crawl the web and make a list of which pages have particular words - this is the index.

learn more… | top users | synonyms

4
votes
2answers
753 views

Can a sitemap index file reference sitemap files under different domains?

My company owns several domain names. We also host websites for our users at their custom domains. Our domains: company.com product1.com product2.com Users's domains: user1.com user2.com ...
4
votes
2answers
279 views

Best approach to have Google index content primarily generated by Javascript

I'm developing an app where customers can drop in a Javascript tag onto their website, and it will generate content. What is the best approach to have Google index this content and associate it with ...
3
votes
4answers
235 views

How sitemap.xml influences indexation?

If a page is missing in sitemap.xml (but crawable through HTML hyperlinks), may this cause it not to be indexed? (I mean it would be indexed without sitemap, but not indexed with sitemap because it is ...
3
votes
2answers
3k views

Why the number of indexed pages decreased for a “site:” search

Recently I created a website with hundreds of thousands of unique pages. A week ago, Google had indexed about 250,000 pages. It has now decreased to 90k URLs (using a site:domain.com query). What ...
3
votes
3answers
67 views

How to prevent a search engines from indexing a section of a page?

I have many pages with lots of text in it. But I will always have two sections of text and I want to prevent one section from appearing in search results, the other section must be indexed. <p ...
3
votes
3answers
938 views

Why are URLs with my domain name at the end showing up in analytics?

I have a multi-language WordPress site that has been live since the last 3 months and today when I checked Analytics, I saw a lot of weird URLs: ...
3
votes
1answer
955 views

Search Engine Indexing of a Database-Driven Website

Suppose I have a MySQL whose rows are dog breeds and whose columns are properties of the breeds (e.g. coat color). The purpose of the site is to allow users to query the database with questions like ...
3
votes
3answers
271 views

Changed my blog's URL: what next?

I started working on a blog from last month. My blogs URL was quite big (for ex. http://my-old-long-url-blog.blogspot.com). I used it because I thought having 2-3 keywords in my URL will help me (I am ...
3
votes
6answers
149 views

Stopping Google index some web pages I have

I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?
3
votes
2answers
455 views

How can I get Google to not index archive pages in Blogger?

I'm using Blogger with a custom domain. When I use the Google search site:example.com, the top results are in the form Example: 2010-01-10. These are archive pages that Blogger generates. My relevant ...
3
votes
3answers
775 views

How to let Google know that content on the page has just changed?

I'd never thought of such a problem in the whole SEO spectrum before I considered launching my own website. Imagine that I have a website which is updated once per month and Google spider knows ...
3
votes
1answer
797 views

Indexing a multilingual website

I am trying to create a multilingual website using the Django framework, all default content is English. I have added a native translation (Dutch). https://mydomain.com/ redirects to the user request ...
3
votes
3answers
2k views

Google webmaster Index Status. Total Indexed=0

I previously changed my domain from www.visualstudiolearn.blogspot.com to www.visualstudiolearn.com. I had around 300 posts with the previous domain name and most of them where showing up on Google. ...
3
votes
3answers
216 views

Why is the number of pages on my website indexed by Googlebot decreasing?

I am monitoring the number of pages indexed by Googlebot for one of my websites . I noticed that since last week, the number of pages has dropped by about 60 or so. Now to be fair, I have not added ...
3
votes
5answers
1k views

How quickly does Bing index URLs submitted through Webmaster Tools?

In Bing's Webmaster Tools, there is a Submit URLs page that allows you to "Submit URLs that are currently not in the Bing index or that should be reindexed due to recent changes." Does anyone know ...
3
votes
3answers
45 views

Indexed webpages stopped rising, while Submited keep rising normally

Last months I have noticed a strange behaviour in Google's Webmaster Tools. Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while ...
3
votes
4answers
2k views

How can I get a list of all indexed pages for my domain?

We're in the process of migrating an outdated e-commerce cart to Magento. Part of this process will entail configuring proper 301 redirects. I'm aiming to get a list of all indexed pages to throw ...
3
votes
1answer
223 views

How to allow Google to index protected content?

I'm working on my site and as it require users to log in it will be hard to Google to index the sites, because to see 90% of the content you have to be online. So I've made a script which looks for ...
3
votes
1answer
102 views

How can I force a certain URL for Google to use in search results?

Each page on my site can be accessed by a shortened URL and normal "long" URLs. The shortened URL redirects to the long URL and the long URL includes an optional title that's not parsed (much like on ...
3
votes
7answers
67 views

Will a missing <head> tag prevent Google from indexing a WordPress site?

I was recently trying to configure a Google Webmasters Tools account for a client. I spent almost an hour wondering why I couldn't get the site to validate, when I found that the opening tag was ...
3
votes
1answer
45 views

Odd/faulty page links being served to Googlebot

Yahoo recently bought Tumblr, where I write fairly long form scientific pieces. I noticed recently that a new script had been added to the end of pages, and it seems to be interacting badly with the ...
3
votes
1answer
807 views

Google Webmaster Tools Index Status 0 for one year

I've read this topic with answers but they do not address why zero for one year? but other areas in Google Webmaster Tools report 400 URL indexed. Google Webmaster Tools Index Status Only 2 Indexed. ...
3
votes
2answers
129 views

Traffic fall after a server problem

I have a website from which I analyse the traffic with Google analytics. Day after day the traffic (mainly from Google SE) incresed until I get a problem with my server. For one day the server has ...
3
votes
1answer
129 views

Indexing dynamic loading page

Does Google index dynamic loading page? For example index.html <div id="dynamic_message"></div> index.html after few seconds <div id="dynamic_message">Hello Google, do you see ...
3
votes
2answers
892 views

Can I force Bing to reindex our site?

Our website is ranking quite well (first page) on Google for quite a lot of keywords. However, our website is not on the first 10 pages of Bing for most of these keywords. I notice our Google ...
3
votes
1answer
466 views

Webmaster Tools show the “Total Indexed” number to be zero

I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
3
votes
2answers
195 views

Using .htaccess for proper indexing/SEO of timthumb images

I don't know a whole lot about the intricacies of SEO so I was wondering if someone here could provide some insight to this problem. When you use the php script timthumb on a website, it means images ...
3
votes
2answers
250 views

Google De-Index many pages at once?

On one of our websites, Google has been indexing something it wasn't supposed to. We fixed the problem so it shouldn't happen anymore, but are interested in requesting that Google de-index these ...
2
votes
4answers
123 views

Why doesn't Google remove my noindex and 404 pages?

I've added 404 error codes and no-index meta tag to pages quite some time ago. However, when I check in Google Webmaster Tools, looking at the graph of "removed" pages from their index, I see very low ...
2
votes
1answer
2k views

how to make sure an individual website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
2
votes
2answers
70 views

New domain and submission to search engines

I have registered a new domain with a hosting company. They offer the feature that for each new domain there is an associated placeholder page. Actually it is a "Site not configured page" with some ...
2
votes
3answers
81 views

Multiple site pages not accesible via main site nav- SEO implications?

I run a site that has some of the site pages accessible through the site's main navigation. Then, there are 50+ "state" pages that are accessible to a user if they click on a specific state on a ...
2
votes
2answers
99 views

How to explain visitors that we don't control Google's results?

I have a message boards. Once in a while, some people contact us to remove anything related to them. We comply and delete all the posts and all traces. Some of those visitors still complain that they ...
2
votes
3answers
65 views

Will the already indexed URLs be removed if I use a robots.txt for my site?

Many URLs from my site have been added to Google's index, but a lot of them are outdated and they will never result in a 404 error and take the user to my site's homepage. I have submitted a new ...
2
votes
2answers
82 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
2
votes
4answers
422 views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
2
votes
2answers
739 views

Difference between canonical and base tag

I am a little bit confused regarding the usage of canonical tags and base tags. I have placed canonical tag (rel="canonical") on my website because of my website opening with no-www. I have reviewed ...
2
votes
1answer
955 views

Google Webmaster tools showing sitemap URL's indexed but not showing up in search results

I have submitted simple text sitemap to Google Webmaster tools and after few days it was crawled, after few more most links were indexed according to GWT but it is not showing those links in google. ...
2
votes
2answers
314 views

I've changed my URL schema. How do I tell Google to index the new schema and forget the old one?

I had a site where the urls were constructed like this /index.php/Topic /index.php/AnotherTopic These were indexed in google, and search results returned that pointed to these. However, I've ...
2
votes
2answers
868 views

What does Google do with indexed pages returning 403?

I guess that Google removes already indexed pages that are now returning the HTTP error 403. Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
2
votes
2answers
119 views

This address should never have been indexed, how can I properly remove it?

I have a couple of subdomains on my website that I use for various purposes and I don't want/need them to be indexed by search engines. From the very beginning, since I created them, I added a ...
2
votes
3answers
70 views

Is there a legal way to serve data-dump content without incurring search engine penalties?

I have a games site and I'm writing an API that will pull relevant Q&A from my copy of the SE data dump. I will abide by all licensing rules. That said, I'm hesitant to do this because Google ...
2
votes
2answers
179 views

Will this sitemap get me de indexed from Google?

My site's URL (web address) is: http://quantlabs.net/private/sitemap.xml Description (including timeline of any changes made): Will this sitemap get me de-indexed from Google? My new site map just ...
2
votes
1answer
42 views

Google is re-indexing pages after redirecting URLs from HTTP to HTTPS incorrectly

I upgraded my site so that all pages have gone from using HTTP to HTTPS. I didn't consider that Google treats HTTPS pages differently than HTTP. I recreated my sitemap to so that all links now ...
2
votes
3answers
92 views

Why are my posts omitted from Google search results when my site is listed in it?

My domain is not new at all, in fact I have waited for quite a long time now for Google to show some love. If I do a search for my domain name, it shows the website is listed, but if I try to search ...
2
votes
2answers
202 views

How to prevent Google from indexing a static web page?

I have a static webpage written in HTML and CSS. I do not want it to be indexed by Google. So I made a robots.txt file in the parent directory and disallowed the URL. I want to know: Is that the ...
2
votes
2answers
42 views

Would Google penalize me for removing indexed pages?

I'm about to remove a large part of a website in order to replace it by a better alternative which, unfortunately, cannot be mapped to older URIs with URL rewriting. The part of the website I want to ...
2
votes
1answer
80 views

Should paginated sites be included in sitemap?

My website is ad listing for boats, boats' engines, boat rentals, etc (www.mynautics.com). There are ads listed on multiple pages. I have finished (not jet seen on link above) pagination: added ...
2
votes
3answers
1k views

Finding Google's indexed pages

How do I find out how many pages Google has indexed on my website? I know I can just Google site:http://foo.com, but I would like to record the data and not have to do it by hand. Is there an API for ...
2
votes
3answers
277 views

After adding *.xls to robots.txt how long will the results continue to show up in google

Is there an estimated amount of time before the document will not appear in google results? We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...