1

I apologize for the title, but I can't really explain it . Here's what I am asking

Can I have seo specific urls in my sitemap.xml that will have canonical links to the actual urls.

E.g: Sitemap xml has a url of /foo/bar?seo=true and the content has a canonical link to /foo/bar

The reason we are considering this option:

We have a javascript SPA, that needs to be pre-rendered for bots. In order to avoid the overhead of checking on every request, we thought we can use sitemap.xml to redirect bots to a specific seo url, which will go through a pre-redenderer (e.g: phantomjs) and return the actual content, which includes a canonical link to the actual url.

Improve this question
5
  • 1
    If you think Googlebot only comes from sitemap then it is wrong, they follow links(Internal+External), re-crawl index webpages, re-crawl webpages based on pagerank, and crawl webpages based on popularity.
    – Goyllo
    Jan 9, 2017 at 8:31
  • 1
    And canonical link does not redirect bots, it just indicated that, that webpages you're prefer to index.
    – Goyllo
    Jan 9, 2017 at 8:32
  • 1
    I understand that the bots will attempt to crawl a bunch of other urls and they will receive just the shell of the SPA. But we hoped that if all our pages are indexed in sitemap.xml and Google respects the canonical link, then the content from ?seo=true will be indexed under the canonical url, so it wouldnt matter if the bot cant crawl the regular urls
    – Dogoku
    Jan 9, 2017 at 9:07
  • Google will prefer canonical link tag all the time, except when webmaster use parameter URL in internal link structure. And in your case Google will index your '/foo/bar'
    – Goyllo
    Jan 9, 2017 at 9:18
  • Have you tried Google's "Fetch as Googlebot" tool in GSC? Google is pretty good at rendering JS these days; you might not have to do anything special?
    – MrWhite
    Jan 9, 2017 at 10:21

3 Answers 3

Reset to default
2

You could, but you won't get the guarantee that it won't break users experience, leading a user to reach your website with ?seo=true appended to his request.

To make an example, Google specify that

We attempt to respect this, but cannot guarantee this in all cases.

meaning you should then check that they're actual users rather than bots. To be fair, IMHO, I won't expect that to happen that often, thus could effectively reduce the overhead due to a reduced number of requests to require special handling.

Furthermore you're not sure that a bot reaches your website via a link (which may be external to your website) rather than using the sitemap, and in that case you'll not serve the pre-rendered page. Although you have a sitemap the crawler algorithm may decide to crawl the page immediately without looking at the sitemap (at least at that moment).

Improve this answer
1
  • 1
    I understand that the bots will attempt to crawl a bunch of other urls and they will receive just the shell of the SPA. But we hoped that if all our pages are indexed in sitemap.xml and Google respects the canonical link, then the content from seo urls will be indexed under the canonical url, so it wouldn't matter if the bot cant crawl the canonical urls. It seems we were too ambitious :P
    – Dogoku
    Jan 9, 2017 at 9:08
2

You're fine to use parameter(?) in your XML sitemap, like ?seo=trueor similar words.

You can add same parameter to above question and you will see it display the same page, 

http://webmasters.stackexchange.com/questions/102669/can-i-use-sitemap-xml-to-redirect-bots-to-seo-specific-urls?seo=true

When Googlebot see your parameter URL from sitemap, and start crawling, it sees canonical link tag, which point to the non parameter URL, so they will assign you're prefer to index that webpage(/foo/bar), not the current one(/foo/bar?seo=true).

Feel free to use parameter URL in sitemap, but don't use it on internal link strcuture, I have seen many of webpages that index with parameter and it is just because Google priority the link structure compare to canonical link tag.

Improve this answer
2
  • 1
    "prefer to index that webpage" - Google can still index both (it needs to crawl and process/index both in order to determine that one is the canonical of the other. The canonical tag itself is only advisory.) And if you do a site: search you can sometimes see both returned in the SERPs. However, the canonical tag is a suggestion that the canonical URL should be returned in the SERPs instead, when a search would otherwise match the non-canonical URL.
    – MrWhite
    Jan 9, 2017 at 10:20
  • site: dork, even return robots.txt, XML sitemap, and lot's of other parameter URL as well. Google official docs on parameter also says If your site publishes content that can be reached via multiple URLs, you can choose a representative URL to appear in Google Search results by specifying a canonical (preferred) version of the URL
    – Goyllo
    Jan 9, 2017 at 12:23
0

Have you tried using an htaccess rule to apply additional $_GET params when your website is requested by Google?

From this Stack Overflow answer: 

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^(.*)\.google\.(.*) [NC]
RewriteRule ^(.*)$ https://www.my-site.it/$1 [L,R]
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.