0

Google is finding pages on my site that are not listed in my site map. How does Google Search find a list of pages on a site to index?

  • 1
    The thinking that Google uses sitemaps to index your site is antiquated. As mentioned by several answers, search bots crawl links on various pages. For more information about sitemaps, read webmasters.stackexchange.com/questions/4803/the-sitemap-paradox – Trebor Oct 14 at 5:01
  • 1
    Do you use Google Analytics, or any other google product on your site? Links or no, if you do, you have put that URL on their radar. – StephenCollins Oct 15 at 16:29
  • Yes, each page has analytics built in. That sound like the source. – glez Oct 15 at 20:47
2

The sitemap is there, in part, to help Google find pages it might not otherwise locate, such as orphaned pages (pages with no links pointing to them). However, the main method Google uses to find pages is crawling - that's following links on pages. The links may be on your own website, or they could be links from other websites. If the indexed page is not in your sitemap, then there must be one or more links pointing to it from somewhere. Note that a link doesn't necessarily have to be visible on your website - it may just be something in your website source code, such as a meta tag or structured data URL.

| improve this answer | |
  • Its strange because I use a site map generator that does the same type of crawl and it doesn’t find the pages. – glez Oct 13 at 21:29
  • 1
    I imagine the sitemap generator will probably just crawl the <a> links on each page and will not look at URLs in other tags. This also doesn't consider links from other websites, including social platforms. – Jason Riley Oct 13 at 21:34
1

I have an old website that I send only the home page to Google and test how Google indexing my other pages.

The result: Google index my first page then search in this page for other links for my website, after that this new pages are indexed too and every links in the new pages for my website are added for next indexing pages, in my Facebook I add other link for hiding page in my website and its also added. Also if you use Google adsense or Google analytics in your pages it will be indexed too.

Same time pages are find by Google but not indexed for same reason like low graphics or similar to other pages from your website or other websites.

| improve this answer | |
  • I have 1 page with links to the others it found but that home page is orphaned unless the user is logged in by design. I don’t believe there are external links to it either. I cant tell how google is finding these pages assuming Google cant get a file list of the site some other way – glez Oct 13 at 23:12
  • 1) If you use Google analytics maybe this home page is find by Google before it request a login user. 2) it could be someone who logged and shared the links of your pages – Abrahem haj hle Oct 13 at 23:22
  • I do use analytics. I wonder if Google can show me how they found it or if theres some clues in. the web server access log – glez Oct 13 at 23:24
  • You mean backlinks, Google console have a list of websites that share links to your pages. Same other websites can help you find who share your links too. – Abrahem haj hle Oct 13 at 23:30
  • I was thinking of any link that led them to the page – glez Oct 13 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.