3

I am trying to understand why Google bot is crawling only one page only on my website.

At the Google console I registered a small XML sitemap with 9 URLs. Only one URL I marked as a high priority (0.8) and that changes on daily basis. This URL is /Home/Index

Based on this answer I removed all other sitemaps, and on Home/Index I placed several anchors to the pages that I wanted to be crawled too, and these anchors are changing each visit to the new anchors. And pages behind these anchors are having other several anchors, and so on.

What I am seeing, Google bot is crawling only /Home/Index multiple times per day and seems to be ignoring all pages behind the anchors. What might be the reason and how to change this?

I can also see that other bots are crawling those anchors.

Improve this question
15
  • 1
    How do you know what Googlebot is crawling? Are you watching your access logs, or are you basing it based on what Google is indexing?
    – Stephen Ostermiller ♦
    Apr 30, 2021 at 8:42
  • How old is your website? Did you just launch it in the last couple weeks? How long has it been since you created the links?
    – Stephen Ostermiller ♦
    Apr 30, 2021 at 8:42
  • Are you using any "single page" technology on your site that renders your site with JavaScript? Are you using Angular or React?
    – Stephen Ostermiller ♦
    Apr 30, 2021 at 8:44
  • 2
    Google ignores the priority field in XML sitemaps. I usually recommend omitting that field along with the last modified field which is also ignored. Most sites just need the a list of loc in their XML sitemap and nothing else. It shouldn't hurt to have an XML sitemap even if it doesn't help Google index and rank your content. Having a sitemap just isn't sufficient for SEO. It is safe (and probably better) to have your sitemap AND links to your pages.
    – Stephen Ostermiller ♦
    Apr 30, 2021 at 8:48
  • 2
    XML allow for images, videos, and hreflang to be included while txt sitemaps do not. XML sitemaps also support sitemap index files for sites with more than 50,000 pages. You probably don't need any of those features, so TXT would be fine, but XML is also fine.
    – Stephen Ostermiller ♦
    Apr 30, 2021 at 10:56

1 Answer 1

Reset to default
0

The priority setting has no influence on what the Google Crawler will crawl at the end. If one webpage has a low number of pages Google will crawl them all. Or do you have any specific settings in the robots.txt? If you upload a Sitemap in the robots which consists of one page other pages will not be regarded.

Improve this answer
1
  • Google may crawl them, but deliberately ignores many, or even most, while providing no reason why.
    – Patanjali
    Jan 28, 2022 at 4:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.