1

I want to generate an XML sitemap for my website. Since my website's CMS is custom PHP and the blog's CMS is Wordpress. Here is my website provided below:

My problem is that I am not able to generate an XML sitemap for my blog through https://www.xml-sitemaps.com/. I do not understand what's the reason behind that. Since my website blog's CMS is WordPress, I have installed Yoast plugin where I can generate the sitemap for the blog.

I do not want to generate sitemap through Yoast or manually especially blog pages. There are more than 250 blogs have published on my website. I would like to generate sitemap through xml-sitemaps.com. Will it be possible to generate any other online tools? Why I am not able to generate blog's sitemap? Any Idea??

One more thing when I run the audit on SEMRUSH, SEO Powersuit, SF. There are almost 400 pages and blogs on my website. But Semrush bot has only crawled 54 pages as well as SF and Powersuit.

I did not do nofollow, noindex on meta, robots or any other areas and there is no broken links present on the website.

Can anybody please tell me, what is the problem? why my website crawls only 54 pages instead of 400 pages and how can I fix those issues?

Please have a look on a screenshot attached.

enter image description here

Improve this question

1 Answer 1

Reset to default
1

Using a crawler to generate your sitemap is pointless. If the sitemap crawler can crawl your site, so can Googlebot. Google's John Mueller says:

Automate it on your backend (generate the files based on your local database). That way you can ping sitemap files immediately when something changes, and you have an exact last-modification date. Don't crawl your own site, Google already does that.

Your sitemap's job is to:

  1. Get all the pages on your site discovered by Google (although a properly crawlable website doesn't need this.)
  2. Tell Google about your preferred URLs (although a site that uses redirects or canonical tags doesn't need this.)
  3. Give you extra stats in Google Search Console.

Sitemaps don't help get your content indexed or ranked. For the best chance of getting indexed and the best chance of getting ranked, you should ensure your pages link to each other. See The Sitemap Paradox.

You should try crawling your own site and see how robots can use it. The best part of this sitemap exercise for you is that has uncovered problems with your site such that it can't all be crawled.

I crawled your site myself using:

$ wget -r 'https://www.unifiedinfotech.net/'

It only found 34 HTML pages (and a lot of assets):

$ find . -type f | sed 's/\?.*//g;s/.*\.//;' | sort | uniq -c | sort -nr
185 jpg
 79 svg
 64 png
 34 html
 14 eot
 12 js
  8 woff
  8 ttf
  4 woff2
  3 gif
  3 css
  2 otf
  1 xml
  1 txt
  1 php
  1 mp4

I'm not sure exactly where all your pages are supposed to be. If you had a sitemap that was generated from your database and filesystem, you could compare the results of the crawl to your sitemap and figure out which pages the crawler can't get to.

One thing that I notice is that your blog is not crawlable at all. The crawler only found two pages:

$ find blog/ -type f | grep html
blog/wp-json/index.html
blog/index.html

Looking at your blog, that is because:

  • Your home page loads its article list via AJAX. Googlebot will be able to crawl the initially loaded articles because it is the most advanced crawler and executes JavaScript. Most other crawlers (including wget) won't be able to crawl your blog at all.
  • Your blog uses infinite scroll to load additional articles. Even Googlebot won't crawl infinite scroll. Googlebot doesn't try to scroll any pages to try to discover that more content loads. Googlebot will only crawl the pages from the initial load.
  • Your blog doesn't have any menus or navigation.
  • Your blog doesn't have "tags" or "categories" that usually assist crawling by creating hierarchy pages.
  • Your articles don't have a list of "related articles" like the "related questions" list on the right side of this page.
Improve this answer
11
  • Thank you for clarification. In my blog, I have done interlinking every blog pages. Yes, my blog uses an infinite scroll to load additional articles because google hasn't supported rel=next/prev. Resource: Is there any solution to fix these issues? Actually, I have disabled "tags" or "categories" options through Yoast pluging due to avoid duplication. I have enabled "tags" or "categories" options in Yoast. Now, If any duplication arises from such blogs then how do I fix these issues? Any idea?
    – Olivia
    Commented Jan 24, 2020 at 7:09
  • You are the second person I've come across in the last couple weeks that has disabled tags and categories for that reason. Where did you get that advice? It is bogus. See What is duplicate content and how can I avoid being penalized for it on my site? Google doesn't penalize for internal duplicate content. It only chooses not to index pages without unique content. The duplicate content on them won't hurt the rest of your site.
    – Stephen Ostermiller ♦
    Commented Jan 24, 2020 at 9:45
  • There is some merit to the argument that creating tags and pages that only apply to one or two posts is bad. That creates very thin pages. Best practice is to make sure that almost all tags and categories you use apply to several posts.
    – Stephen Ostermiller ♦
    Commented Jan 24, 2020 at 9:46
  • I personally disable tags on my blog and just use categories, but because they make WordPress both crawlable and allow users to navigate, I wouldn't disable both.
    – Stephen Ostermiller ♦
    Commented Jan 24, 2020 at 9:48
  • The other WordPress feature that makes it crawlable is pagination. Your infinite scroll disabled that. I'm not a fan of pagination for either SEO or users, so I prefer tags and categories.
    – Stephen Ostermiller ♦
    Commented Jan 24, 2020 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.