1

I have a php website(symfony) with dynamic content, some is published by the admin and some by the loggedin users. I have created a dynamic sitemap that gets updated whenever a content get added, edited or deleted.

Is this a good and optimized way to do it?

I added the URL of the sitemap to the robots.txt file and I should as well submit it to search engines(google).

I am wondering if I submit it and it's dynamic then google will have only that version I submitted at that time?

While the sitemap is dynamically updated, will it automatically read each time the new version or should I resubmit it after some time which I wont be able to specify because users posts also update the sitemap?

  • First of all, Google does not want to see search result pages. This can result in a penalty. Secondly, when you place a URL in a sitemap, what are you putting in robots.txt? This is unclear. As well, Google will check a sitemap periodically for updates. You do not have to resubmit it. Last, sitemaps are mostly useless unless your site is HUGE or has content behind paywalls or logins. It is good to update your sitemap of course. However, Google will mostly use it to audit its ability to crawl your site. Nothing more. – closetnoc Aug 6 '17 at 17:27
  • No search engine should work with URL that returns dynamic contents (you can update the amend the contents but not something that totally different). As long as you build web pages that conform with the old day website best practice, Google ranks content accordingly. to make things easy, you can embed a google search button on your website(which embed a site:yourwebsite.com of your web contents) – mootmoot Aug 7 '17 at 10:57
  • closetnoc, thanks for your reply, you said that my website can be penalized if google see search result pages, should i then prevent them from being indexed? by blocking access to them in robots file or by adding the meta "noindex" to the page. secondly when i update my sitmap(programmatically) i don't put anything in robots file. what i meant is that, for google to consider your sitemap you can also add the url to your sitemap in robots file. you said that sitemaps are mostly used to audit google ability to crawl a website, i think it's also used to generate sitelinks? – ZeSoft Aug 8 '17 at 7:42
  • Hello mootmoot, thank you for your reply, i have already integrated my own search code, so should i then prevent google from crawling my dynamic content such as search result pages? and would an article home page that lists preview of articles be considered the same, and so must be blocked from being crawled? – ZeSoft Aug 8 '17 at 7:48
2

Is this a good and optimized way to do it?

I would say no, because first time when Google see some URL, then they will index it, and later if that URL is removed then index page will not going to remove immidietly, Google assign there might be some problem and hence return a 404 error in search console, and Google will try to re-crawl same URL again even it is not linked anywhere. Yes Google re-crawl some pages again to check weather the content is updated or not.

And when the URL is changed, then new URL will going to index, here you should do 301 redirection for better visibility, it means remove the old one and index the new one.

I added the URL of the sitemap to the robots.txt file and I should as well submit it to search engines(google).

What you want to opimize? Google, Bing or Yandex? When you add your sitemap into robots.txt then all search engine follow your sitemap in regular days, even dead search engine like ask.com says, we don't accept new site submission, but if there is sitemap link then we will crawl it. But having a sitemap link into robots.txt means it is easy to scrape your all content by bad bot. But only popular site need to worry about it like Stackoverflow hide their sitemap from all users execept search engine bot.

I am wondering if I submit it and it's dynamic then google will have only that version I submitted at that time?

Yes Google don't understand dynamic thing in sitemap. They take action at a time.

While the sitemap is dynamically updated, will it automatically read each time the new version or should I resubmit it after some time which I wont be able to specify because users posts also update the sitemap?

Sitemap crawler set a time based on frequency of updated content, but sitemap is not only one crawler which index pages, Google follow a link as well to index content. Google crawl frequency is based on many things.

The old crawler(Since 1998) is still used heavily by Google.

  1. The site having a good amount of Pagerank crawl often, example Wikipedia.
  2. The site having a good amount of Backlinks crawl often, example Stackoverflow.

The new sitemap crawler.

  1. Sitemap crawler.
  2. Google submit Tool or Fetch/Render tool.
  3. Refresh crawler (I don't know what google call it) but they re-crawl indexed pages, even when remove all pointing links.
  4. Google now use many other sources as well to find popular link like Google DNS service or may be Social media trends. I don't have more info in this.
  • hello Golyo, thank you for your reply, if generating dynamic sitemap is a not a good option, is there any other alternative for dynamic website? and in general you are right if an indexed page is removed from website there should be a redirection to the new one but that means that i have to use soft delete to keep the old data regarding the old page to be able to spot old pages or how this is normally handled? and when can i get rid of that data? – ZeSoft Aug 8 '17 at 22:59
  • First don't use sitemap and let google discover your pages automatically, if they index it properly then you don't require sitemap at all. Another solution is generate last week feed, i.e. generate feed when things does not change (something like weekly newsletter) and then submit your feed as a sitemap into Google search console. You can create sitemap as well like that. – Goyllo Aug 9 '17 at 5:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.