Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We have a web portal with almost a million products and way more other urls. I wrote a script that checks database. If there is a new url needed or an old one update, this script will update/create the XML Sitemaps. But we have several servers behind the load balancer at our rented AWS space.

Further this script checks database for each url if there was an update so that it updates the appropriate xml file too.

My question is how to spread those XML Sitemaps over all webservers behind this AWS load balancer?

Our approaches/ideas:

  • we could just generate them on one server with a cron job and copy them to the other servers, but this could be difficult because of automatic raising numbers of servers and so on.
  • we put them on our S3 - but this one is not avaible thru our domain, so I guess google will have a problem with it
  • I let my script run on every webserver but change it in a way that it will generate each time all xml files if they do not exist. But then I would have conflicts with updated URLs in my database, where I saved timestamp of last changed value of every url

Is there another - better - solution that I do not know? Are there any special services by amazon for such cases?

share|improve this question
    
Do you know that there are limitations on the size of an sitemap after it is de-compressed? –  Liam Sorsby Apr 4 at 12:17
    
50.000 urls per XML Sitemap and up to 500 XML Sitemaps in the index XML Sitemap. –  Jurik Apr 4 at 12:34

1 Answer 1

We decided to save all xml files zipped to database and get them from there as soon as a bot asks for a specific XML Sitemap.

Here are the theoretical steps:

  1. Create XML files with 45.000 urls. (45k according to this post)
  2. Save every url to database with information in which XML Sitemap it is in.
  3. gzip those files (8mb => 400-600kb)
  4. save them to a xml file table in database (use medium blob, because files are zipped between 400-600kb large)
  5. store index xml sitemap in this table too
  6. when google requests a file, get its content from database and set headers like

    header('Content-Type: application/x-gzip');
    header('Content-Encoding: gzip');
    header('Content-Disposition: attachment; filename="' . $xmlFile['name'] . '"');
    echo $xmlFile['content']; //source: database ?>

Beside this I've scripts that keep those xml files up to date and create new files or add new urls to existing xml files (with space).

So if anything changes my xml files will be up to date and saved to database, so that every webserver behind the loadbalancer can provide up to date XML Sitemaps.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.