1

A dynamically generated website doesn't have most of its content hard coded on the HTML file. Sometimes it waits for a user to login or wait for an input to generate its views. So it would look something like this.

 <section id = 'blog-content' class="section">
     <div class="columns is-centered">
         <?php echo $content ?>
     </div>
 </section>

In that example, the variable $content contains paragraphs and a title that someone wrote on a certain topic for their blog fetched from their database. How does a crawler know the content of that page if its content is dynamically generated? Since the content is not hard coded, the crawler wouldn't know the content exist because all it would see is the code i post above, right?

EDIT

To expand on what stephen said, the url would be something like this.

www.example.com/blog/$title

The $title variable has the same value as the one in the $content variable above.

3
  • Does the dynamic website have a mapping between URLs and content? Usually there would be some sort of variable in the URL with a content id. – Stephen Ostermiller May 24 '20 at 10:20
  • It is possible to build crawlable dynamic websites, but it is also possible to build dynamic websites that can't be crawled by search engines. Your examples of user login and requiring input are two that might not get crawled, depending on how they are implemented. Is the content not available unless the user is logged in? What type of user input? Does it use POST or GET requests, or something else? – Stephen Ostermiller May 24 '20 at 10:23
  • @StephenOstermiller i edited my question – Firdaus Indradhirmaya May 24 '20 at 12:07
4

Short answer: That PHP code is run on the server before sending the response to the crawler, so by the time the page reaches the crawler, all that info is already populated.

For sites written using server-side languages such as your example, here's the full lifecycle when a user visits a page:

  1. The user's browser sends an HTTP request to the server for a certain path (such as /an/example/page/).

  2. The server receives the request and determines the appropriate server-side code to run to generate the page. It executes this code, if any (or none if it's a static site).

  3. The server sends the final generated, by that point static HTML page back to the user's browser.

Note that all the code is finished running on the server before the server actually sends any information back to the user's browser (or web crawler).

Things are a little different when the page is generated in part by client-side code (JavaScript) instead, which is a topic for a different discussion.

Regarding waiting for a user to log in or take action, generally search engine crawlers are cookieless and take no user actions, so anything hidden behind a login won't get crawled. Stuff hidden from crawlers behind logins like this is called the deep web, which is a cool term if you ask me.

2
  • Let's see if i understand you correctly. What you are saying is that when a web crawler crawls through the internet, it would see the same information that we would see? So no matter if its a dynamic or a static page, no matter if the meta tags, title and all of its content is dynamically generated, it would have no problem detecting all of the things i mentioned as long as the URL is not something behind a login lets say. Is that correct? – Firdaus Indradhirmaya May 24 '20 at 12:12
  • Correct, with one caveat. If parts of the page are rendered client-side using JavaScript, most crawlers don't execute JavaScript like browsers do. Googlebot is an exception, it does, for the most part. – Maximillian Laumeister May 24 '20 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.