2

We have several domains managed by the same code base. They share content and assets between each other but are separate entities online.

I have started generating sitemaps for these sites and hosting them on S3, this works and is a sane method for us as we have no shared file system as such.

But, I obviously cannot submit the sitemap through the UI as it requires the sitemap be on the same domain, which it isnt.

Another option is to use robots.txt to specify your sitemap (as discussed here; Can I host a sitemap on another domain?) - but as all our sites share the same docroot, we can't do that either.

Maybe we could do some fancy URL rewriting to map the sitemaps? I'm not sure what the best option is for us. Any help is much appreciated.

EDIT
I'm trying to leave the sitemap.xml.gz files on s3 but serve the xml index using php on the sites. So you can visit /xml/site1.xml and get the sitemap index which points to s3. I'll report back with how it goes.

EDIT 2
I have added an xml doc on each domain with the index of sitemaps. This is at /xmlsitemap/sitemap.domain.xml. The index contains the externally hosted sitemap (on s3). I then added 7 x 'sitemap:' entries into robots.txt - so this should validate the external sitemaps. It doesn't, all sitemap indexes are returning errors that the sitemap is blocked by robots.txt.

The robots.txt is not blocking the sitemap, but it keeps reporting that it is.

Any suggestions?

EDIT 3
Some links ... [Removed]

EDIT 4 & RESOLUTION
So, to resolve this issue I am generating the xml (both sitemap with delta and index) and storing it in the db. Then i'm just serving the generated xml using nginx/php through registered paths. As we're behind Varnish we can cache the output for 24h or so. Its working well so far.

I also noticed that my s3 bucket did not have a robots.txt so the robots message may have been caused by that and not the local one.

So, resolved by using none of my original options.

Improve this question
5
  • amazon aws provide large number of solution, why you did not link your bucket to domain directly, because you can easily map your DNS to Amazon cloud. I did not understand why you can't host all the thing on amazon s3, ec2.
    – Goyllo
    Apr 12, 2016 at 7:08
  • Hi @Goyllo we have all sites and services on AWS infrastructure. We don't use the cloud or cdn services, they are a separate thing. How would you handle this without cloudfront? (is it even possible without cloudfront?)
    – Christian
    Apr 12, 2016 at 7:20
  • 1
    "but as all our sites share the same docroot, we can't do that either." - as you suggest, you could probably rewrite the URL. For example, on Apache: webmasters.stackexchange.com/questions/83922/…
    – MrWhite
    Apr 12, 2016 at 7:40
  • 1
    Christian - You can use Route53 with s3, Here is two( 12 ) good resource about that.
    – Goyllo
    Apr 12, 2016 at 7:46
  • @Goyllo we can't use a Route 53 subdomain because Google allows only the www.domain.com.au domain to be used.
    – Christian
    Apr 13, 2016 at 7:18

1 Answer 1

Reset to default
2

You can easily achieve this using the same doc_root without needing to worry about uploading the sitemap to the S3 bucket.

Using the following rewrite code

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(domain1\.com)$
RewriteRule ^sitemap\.xml$ /domain1-sitemap.xml [L]
RewriteCond %{HTTP_HOST} ^(domain2\.com)$
RewriteRule ^sitemap\.xml$ /domain2-sitemap.xml [L]
RewriteCond %{HTTP_HOST} ^(domain3\.com)$
RewriteRule ^sitemap\.xml$ /domain3-sitemap.xml [L]

You place this code in a single .htaccess file in the site root and generate your sitemap files using the naming convention domain-sitemap.xml and the rewrite rules will rewrite sitemap.xml to the appropriate domain to suit the host header.

Improve this answer
9
  • Thanks, thats a great option but I can't really use the docroot at all. We destroy and create instances daily, so docroot is not permanent. Also, as these will be changing we can't really put them in git either.
    – Christian
    Apr 13, 2016 at 0:22
  • What language is your site written in?
    – Chris Rutherfurd
    Apr 13, 2016 at 0:45
  • 1
    Would you consider placing a sitemap generation script in PHP into the root directory by any chance to dynamically create the sitemap each time it is hit. What I am thinking could be done as an expansion of my answer above would be to still have the rewrite rule but instead of targeting an XML file have it rewrite to a file such as sitemap.php?domain=domain1.com. You could then have the script generate the sitemap at the time that it is called, because it generates it when it is called no need to update the file constantly, and it would be able to be added through Git. Thoughts?
    – Chris Rutherfurd
    Apr 13, 2016 at 5:40
  • 1
    Unfortunately that may be your only option. I have checked everything I can find and while you could 301 redirect a request for a sitemap to a new URL such as a subdomain which could be connected to S3 there is no way to do it transparently as a rewrite so you may need to create the sitemap index as the PHP with the rewrite and have it link to the separate domain for the generated sitemaps.
    – Chris Rutherfurd
    Apr 13, 2016 at 7:48
  • 1
    Any chance you could post a link to the robots.txt and sitemap.xml for one of the domains. Hopefully you don't mind posting the link so that I can see them as Google would see them as without seeing the actual files and what headers Google might be getting back when trying to access the sitemaps I would just be shooting blind giving you possibilities which may or may not be the cause.
    – Chris Rutherfurd
    Apr 14, 2016 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.