1

We have just completed a revamp of a website with a new design and I had noticed something when running the sitemap generator for the new site. The new site uses javascript pagination which fits the design perfectly and is user friendly but displays no static links. When running the sitemap generator the paginated links were not being followed so it would discover only 780 pages as opposed to 60,000+

I managed to fix the sitemap issue by creating a PHP page which lists all pages on the site and adding that URL as an additional starting URL in the sitemap generator settings. I have also added a pink to the HTML sitemap generated from the frontend so that search engines (if they also have the same issue as the sitemap generator) will now find the links that could not be accessed via pagination.

My question is: Would these interventions be sufficient to ensure all pages are crawlable. Is it a must or recommended to alter the pagination system as it currently fits in well with the new design?

Looking forward to reading your thoughts.

  • What "sitemap generator" are you using? Something developed in-house? Providing all your pages have a unique URL then providing an HTML sitemap to guide search engine bots can only help. Whilst Google can understand JavaScript pretty well and may well be able to find these JS generated URLs (as PlanetScaleNetworks suggests), not all search engines will. – MrWhite Oct 27 '16 at 13:43
  • Hello. Thank you for your response. I am using the xml-sitemaps.com standalone sitemap generator. Yes, I have added a link to the html sitemap in the footer of the site in the hopes of aiding Google and other search engines in discovering all pages in the even they can not follow the javascript pagination. Regards – Ahmed Widaa Oct 27 '16 at 14:43
1

The thing you have to be careful with here is that not all generators are created equal. Just because the sitemap generator you use didn't work well with the javascript pagination doesn't mean that the pages wouldn't be crawlable. For a long time now Google has been able to crawl javascript and AJAX based content and can identify how javascript and CSS affects page content and take that into account when indexing, including identifying links that are generated by javascript which direct to other pages on your site.

The sitemap file is not the only way that Google can identify the pages on your site, Google will also follow links located on your site, including links generated by javascript or trigger content changes by javascript and will be able to work with those changes and additional dynamic links to crawl further pages on your site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.