0

Is there a way to know when a new website comes into existence. What I mean is how crawlers such as search engines know if something exists in first place. So let's say a new website www.example.com started to host their website on a server with IP 150.150.150.150. How can crawler know it started to exist which is the domain owner started to host at a server.

A few things I have tried-

  1. Ping the server for PTR record, it will provide hostname but will not tell what domains/websites exist on the server.
  2. DNS server can't tell the domains just with an IP
  3. You can ping the domain and that will give you an IP but
  4. if you try to get domain with IP address only, that is not possible.

Now, let's say the website admin does not submit their sitemap or other domain related things/URLs anywhere and even then the search engines index it. How does that happen, is there a way to map an IP or hostname back to domain which I might be missing here.

2
  • Are you interested in getting your newly registered domain crawled are are you trying to keep it a secret? Jun 27, 2021 at 14:37
  • Not sure what exactly it means, but for the majority of it, most of the people, including me, would like crawling to happen after something goes public (if it's put up for public viewing.)
    – Arun Kumar
    Jun 29, 2021 at 10:22

2 Answers 2

1

Crawling: Google searches the web with automated programs called crawlers, looking for pages that are new or updated. Google stores those page addresses (or page URLs) in a big list to look at later. We find pages by many different methods, but the main method is following links from pages that we already know about.

Indexing: Google visits the pages that it has learned about by crawling, and tries to analyze what each page is about. Google analyzes the content, images, and video files in the page, trying to understand what the page is about. This information is stored in the Google index, a huge database that is stored on many computers.

Serving search results: When a user performs a Google search, Google tries to determine the highest quality results. The "best" results have many factors, including things such as the user's location, language, device (desktop or phone), and previous queries. For example, searching for "bicycle repair shops" would show different answers to a user in Paris than it would to a user in Hong Kong. Google doesn't accept payment to rank pages higher, and ranking is done algorithmically.

Below link will provide you details on How do search engines crawl the web & how it really work.

https://developers.google.com/search/docs/beginner/how-search-works

2
  • thanks @Nikhil, "following the already known links" justifies a big part of it so that's really helpful. Now, what happens to the websites which just came live like yesterday and there is absolutely no linkback to them from anywhere. Is there a way to know about those or not at all?
    – Arun Kumar
    Jun 16, 2021 at 11:27
  • @ArunKumar - below link may help you to understand the process and steps. support.google.com/webmasters/answer/34397?hl=en
    – Nikhil
    Jun 17, 2021 at 11:04
0

Now, let's say the website admin does not submit their sitemap or other domain related things/URLs anywhere and even then the search engines index it

The ONLY way Google discovers new URLs is via links (from websites, sitemaps, etc.,). Still, if Google can crawl and index, there is only one possibility. There exists a link to the website from a property not controlled by the website owner, period. Unless there is a link to a website or a sitemap that is explicitly made available to Google, the website will remain in dark forever.

Ref: https://developers.google.com/search/docs/beginner/how-search-works#crawling

The first step is finding out what pages exist on the web. There isn't a central registry of all web pages, so Google must constantly search for new pages and add them to its list of known pages. Some pages are known because Google has already visited them before. Other pages are discovered when Google follows a link from a known page to a new page.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.