How does Google crawl database driven web sites that don't have static files?

Question

I have an index.php file on my server. Google will crawl this and retrieve the information.

I also have a file named product.php. This dynamically gets content from a database to populate all the content. Does Google just crawl the product.php, or does it crawl every page populated from the database that uses the product.php for a template?

Answer 1

I.e. you have a site with two pages: a startpage and a product page. Lets look a the startpage: it can have adresses (urls) like:

• example.com
• example.com/index/
• example.com/index.html
• example.com/index.php
• example.com?page=1
• etc.

Whether Google will crawl index.html or index.php is depending from one factor: which file gets the bot coming into example.com. There could be default server setup of the file loading order (firstly html, then php etc.), it could be custom server setup, like htaccess redirect at Apache, like overwrite example.com with example.com/index.jsp or something other.

Similar happens with an URLs like example.com/product: whether Google crawls /index.php or /product.php is depending of which file gets Google (which file sends the server as an answer to the GET query of googlebot).

To answer you question as you ask it: the file product.php (and index.php and other files) are working all like an art of template with an unique address (an URL). Which file Google crawls (index.html, index.php or product.php) and which content is inside the crawled file is depending of two following steps:

1. Step 1: server setup decides, which address (URL) is to sens as an answer to the GET query. This setup includes any kind of serving: direct, redirect, overwrite.
2. Step 2: After the URL is clear (product.php), it is served and the file (template) begins to be filled up with the content (static content is already there, dynamic content must be prepared through database query or javascript dynamic serving)