Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This question already has an answer here:

I require google to index the entirety of my site: 10 or so basic sections with 80,000+ individual job offer pages. However, to date, only 2,900 pages have been indexed by google.

I'd like to know why... only 2,900 but not 80,000+. I figure, if there were something wrong with my job offer pages, then there wouldn't be those 2,900 values indexed but rather 0...

I have a 'page' sitemap which, link by link, leads to each and every ad (paginated) My XML sitemap is lacking though. So does this mean you absolutely need an XML sitemap when you're dealing with tens of thousands of unique pages?

http://haytrabajo.mx is the URL

http://haytrabajo.mx/mapa is the 'page' sitemap

share|improve this question

migrated from stackoverflow.com Dec 24 '11 at 23:39

This question came from our site for professional and enthusiast programmers.

marked as duplicate by danlefree Feb 7 '13 at 17:53

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

I just answered this question on a post: webmasters.stackexchange.com/questions/23778/… Jason Weber – Jason Weber Dec 25 '11 at 20:14

Here's my generic answer which usually helps: sign up for Google Webmasters and Google Analytics as they may be able to give you some more information.

I'd also generate an XML sitemap with XML-Sitemaps (you could also look at automating generating this file) and upload it to /sitemap.xml - you can tell Google the location of your sitemap in Webmaster tools to aid them in finding your content and it can also show you which of your sitemap pages it has actually crawled.

share|improve this answer

When I check your site map I did not see that you have 80 000 + pages there, you got 6 categories pages, and that's it where all of your 80000 pages?.

Suggest that you expand the map and even create a unique map for each category and send it to GWT.

In addition, I suggest you build a more professional Footer at the bottom to show Google the site structure and improve the indexing your interior pages. You'll find excellent Footer examples here.

I have a couple more suggestions for you to improve navigation and you want i could expand a little more

share|improve this answer
Sitemap files have a limit of 50,000 urls and 10MB. Since this site has 6 main categories, it would probably be best to have a root sitemap for the informational content and category sitemaps for the actual job offer pages in those categories. – Fiasco Labs Dec 25 '11 at 1:31
up vote 0 down vote accepted

The 'false' sitemap found in /mapa did not work. I had to create an XML sitemap using one of those free xml generators and I spread my 80,000 entries into 4 pieces, so that now I have:

sitemap-index.xml which points to: sitemap-1.xml sitemap-2.xml sitemap-3.xml sitemap-4.xml

I then had to submit all these in google webmaster tools and set a custom crawl rate. Now it's just a matter of waiting....and waiting for the whole site to be indexed.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.