How to know when a new website comes into existence?

Question

Is there a way to know when a new website comes into existence. What I mean is how crawlers such as search engines know if something exists in first place. So let's say a new website www.example.com started to host their website on a server with IP 150.150.150.150. How can crawler know it started to exist which is the domain owner started to host at a server.

A few things I have tried-

1. Ping the server for PTR record, it will provide hostname but will not tell what domains/websites exist on the server.
2. DNS server can't tell the domains just with an IP
3. You can ping the domain and that will give you an IP but
4. if you try to get domain with IP address only, that is not possible.

Now, let's say the website admin does not submit their sitemap or other domain related things/URLs anywhere and even then the search engines index it. How does that happen, is there a way to map an IP or hostname back to domain which I might be missing here.

Answer 1

Crawling: Google searches the web with automated programs called crawlers, looking for pages that are new or updated. Google stores those page addresses (or page URLs) in a big list to look at later. We find pages by many different methods, but the main method is following links from pages that we already know about.

Indexing: Google visits the pages that it has learned about by crawling, and tries to analyze what each page is about. Google analyzes the content, images, and video files in the page, trying to understand what the page is about. This information is stored in the Google index, a huge database that is stored on many computers.

Serving search results: When a user performs a Google search, Google tries to determine the highest quality results. The "best" results have many factors, including things such as the user's location, language, device (desktop or phone), and previous queries. For example, searching for "bicycle repair shops" would show different answers to a user in Paris than it would to a user in Hong Kong. Google doesn't accept payment to rank pages higher, and ranking is done algorithmically.

Below link will provide you details on How do search engines crawl the web & how it really work.

https://developers.google.com/search/docs/beginner/how-search-works

Answer 2

Now, let's say the website admin does not submit their sitemap or other domain related things/URLs anywhere and even then the search engines index it

The ONLY way Google discovers new URLs is via links (from websites, sitemaps, etc.,). Still, if Google can crawl and index, there is only one possibility. There exists a link to the website from a property not controlled by the website owner, period. Unless there is a link to a website or a sitemap that is explicitly made available to Google, the website will remain in dark forever.

Ref: https://developers.google.com/search/docs/beginner/how-search-works#crawling

The first step is finding out what pages exist on the web. There isn't a central registry of all web pages, so Google must constantly search for new pages and add them to its list of known pages. Some pages are known because Google has already visited them before. Other pages are discovered when Google follows a link from a known page to a new page.