2

I am building a single page app (SPA) with Angular. The data to be displayed inside the app is fetched from the backend using REST calls. It concerns a social media website which means it will have a lot of user generated pages.

This also means that the sitemaps can become very large making it unfeasible to package them in the angular app itself. Also the content of the sitemaps can become out of date rapidly because users will be constantly adding, updating or removing their user generated content.

Right now i'm storing my sitemaps in an amazon s3 bucket but people who answered this question say google ignores cross-domain sitemaps.

So how should i offer large sitemaps for an SPA? I have an EC2 instance in the backend that receives the REST calls. This instance also autogenerates the sitemaps periodically and puts them in a folder called static/sitemaps that is also exposed to the internet. Could i use my EC2 instance to offer the sitemaps or would that be the same thing as using a s3 bucket. I'd prefer not to offer the sitemaps from EC2 because storage on EC2 is more expensive then it is on s3.

The last thing i could do is create a subdomain for the sitemaps. For instance sitemaps.example.com next to the example.com domain. Perhaps i could redirect crawlers visiting sitemaps.example.com to the sitemaps index file that is stored in my s3 bucket. But i'm not sure whether that works either.

So the question is; For a single page app, What is the proper way to offer large sitemap.xml files to webcrawlers?

Thank you

Improve this question

2 Answers 2

Reset to default
3

One solution would be to use CloudFront to tie various services together under a single domain. CloudFront is Amazon's content delivery network (CDN.) You point your DNS to CloudFront and it fetches content from various origin servers based on behavior routing rules. It can also cache content on edge nodes if desired to speed up performance.

You could leave your sitemaps in the S3 bucket and your site hosted in its current place (like an EC2 instance or load balancer.) You would configure two "origins" in CloudFront, one for the bucket and one for the main site. You would configure behavior to serve https://example.com/ from the main site origin but https://example.com/sitemaps from the bucket.

See Use a CloudFront web distribution to serve content from multiple origins

Improve this answer
4
  • Great answer. However cloudfront also costs money and the sitemaps could potentially become very large or very numerous. I'm reading in this amazon article that amazon buckets can be reached via a subdomain through the use of aliases and CNAME. Would it work if i use a subdomain for the sitemaps instead?
    – Maurice
    Jan 4, 2022 at 19:58
  • 1
    I believe that even a subdomain is a separate origin for sitemap purposes. See Sitemaps - one per subdomain or one for the base domain?
    – Stephen Ostermiller ♦
    Jan 4, 2022 at 20:09
  • I think that you said that your Angular site is static. Another solution could be to serve it from the same S3 bucket as your sitemaps and just assign your domain to that.
    – Stephen Ostermiller ♦
    Jan 4, 2022 at 20:15
  • after having read a bit more about cloudfront i think i'm going to go for your solution @Stephen Ostermiller. Thanks
    – Maurice
    Jan 4, 2022 at 22:27
1

It is important to define the purpose of the XML sitemap. The XML sitemap itself is a list of URLs that are important to you and you want to be reached by all crawling bots so that they can be crawled and indexed as soon as possible. This makes crawling your entire project much easier. However, this sitemap should be kept up-to-date and within 50 thousand URLs, which is the limit defined by Google.

I would suggest a slightly different strategy. Because your content will be updated periodically and you cannot keep the sitemap up-to-date, I would add the URLs of those pages that are major. For example, if your temporary pages are divided into categories, I would add to sitemap.xml all available categories, this will help the bots to reach the categories where all those temporary pages will be. In order for them to be quickly crawled and indexed, I would invest in Core Web Vitals. Those two practices will make the whole process much smoother.

Improve this answer
1
  • thank you for your answer. The reason why i'm using xml sitemaps is because a large portion of the user generated content is only accessible after doing some javascript actions (for instance selecting something from a dropdown menu, then pressing a 'retrieve' button). So if i want to index categories i would first have to make webpages for each category. I might do that in the future, just for helping crawlers navigate to the user generated content more easily
    – Maurice
    Jan 4, 2022 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.