0

Can someone explain how Google's crawlers discover URLs? I have a situation where a URL (https://example.com/en/sombreros) on my site redirects users to a different language version because 'sombreros' is not the correct English term for hats. This URL is not indexed as it's a redirect page. Interestingly, Google Search Console indicates this URL was discovered through another URL, which doesn't have any reference to the first URL nor is it mentioned anywhere on the domain. How is Google's crawler discovering such URLs? Are they combining category names across languages, or might there be another way they're finding these links?

2 Answers 2

0

Crawling is the primary way that Googlebot discovers URLs.

The other way is through links. Google has discovered this URL because a page links to it. Thus, it is discovered, but not indexed.

4
  • So crawling also includes making up the urls by combining different languages with different category name translations?
    – Tauri28
    Commented Mar 28 at 15:45
  • Forgive me but I don't understand what you mean Commented Mar 29 at 4:59
  • Well you said the page is discovered through links. I said there are no links linking to that URL. I said it it the original post. I said it in the comment.
    – Tauri28
    Commented Mar 29 at 17:04
  • I need to visually see how these links are mapped can you provide this? Commented Apr 1 at 18:28
0

If you have different language versions, then effectively you are linking to them.

Google primarily discovers URLs from links, either internal or external. From the horses mouth

Crawling
The first stage is finding out what pages exist on the web. There isn't a central registry of all web pages, so Google must constantly look for new and updated pages and add them to its list of known pages. This process is called "URL discovery". Some pages are known because Google has already visited them. Other pages are discovered when Google follows a link from a known page to a new page: for example, a hub page, such as a category page, links to a new blog post. Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl.

Once Google discovers a page's URL, it may visit (or "crawl") the page to find out what's on it. We use a huge set of computers to crawl billions of pages on the web. The program that does the fetching is called Googlebot (also known as a crawler, robot, bot, or spider). Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site. Google's crawlers are also programmed such that they try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500 errors mean "slow down").

However, Googlebot doesn't crawl all the pages it discovered. Some pages may be disallowed for crawling by the site owner, other pages may not be accessible without logging in to the site.

During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit. Rendering is important because websites often rely on JavaScript to bring content to the page, and without rendering Google might not see that content.

Crawling depends on whether Google's crawlers can access the site. Some common issues with Googlebot accessing sites include:\

  • Problems with the server handling the site
  • Network issues
  • robots.txt rules preventing Googlebot's access to the page
1
  • Yes, I have /en/hats and /es/sombreros but no /en/sombreros. There is no such link as /en/sombreros found in any language versions. The page itself exists only to direct the user to correct language url if he/she tries to edit the language manually leaving the category name unchanged.
    – Tauri28
    Commented Mar 28 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.