I require google to index the entirety of my site: 10 or so basic sections with 80,000+ individual job offer pages. However, to date, only 2,900 pages have been indexed by google.

I'd like to know why... only 2,900 but not 80,000+. I figure, if there were something wrong with my job offer pages, then there wouldn't be those 2,900 values indexed but rather 0...

I have a 'page' sitemap which, link by link, leads to each and every ad (paginated) My XML sitemap is lacking though. So does this mean you absolutely need an XML sitemap when you're dealing with tens of thousands of unique pages?

http://haytrabajo.mx is the URL

http://haytrabajo.mx/mapa is the 'page' sitemap


Here's my generic answer which usually helps: sign up for Google Webmasters and Google Analytics as they may be able to give you some more information.

I'd also generate an XML sitemap with XML-Sitemaps (you could also look at automating generating this file) and upload it to /sitemap.xml - you can tell Google the location of your sitemap in Webmaster tools to aid them in finding your content and it can also show you which of your sitemap pages it has actually crawled.


When I check your site map I did not see that you have 80 000 + pages there, you got 6 categories pages, and that's it where all of your 80000 pages?.

Suggest that you expand the map and even create a unique map for each category and send it to GWT.

In addition, I suggest you build a more professional Footer at the bottom to show Google the site structure and improve the indexing your interior pages. You'll find excellent Footer examples here.

I have a couple more suggestions for you to improve navigation and you want i could expand a little more

  • 2
    Sitemap files have a limit of 50,000 urls and 10MB. Since this site has 6 main categories, it would probably be best to have a root sitemap for the informational content and category sitemaps for the actual job offer pages in those categories. Dec 25 '11 at 1:31

The 'false' sitemap found in /mapa did not work. I had to create an XML sitemap using one of those free xml generators and I spread my 80,000 entries into 4 pieces, so that now I have:

sitemap-index.xml which points to: sitemap-1.xml sitemap-2.xml sitemap-3.xml sitemap-4.xml

I then had to submit all these in google webmaster tools and set a custom crawl rate. Now it's just a matter of waiting....and waiting for the whole site to be indexed.

  • how to set the crawl rate? which generator did you use?
    – Toskan
    Jan 13 '17 at 11:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.