How to avoid robots from indexing pages of my app through alternate URLs?

Question

I have this web app that is accessible through those different URLs.

URL_1: www.mycustomdomain.com            // THIS IS MY CUSTOM DOMAIN (PUBLIC TO THE USERS)
URL_2: www.my-project.firebaseapp.com    // THIS IS A DEFAULT FIREBASE URL
URL_3: my-project.web.app                // THIS IS A DEFAULT FIREBASE URL

Obviously I want Google to index my pages using my custom domain, and not the Firebase default URLs.

I don't publicize the default URLs anywhere, but what if Google discovers them at some point? What is the proper way of letting Google know that those pages/URLs should NOT be indexed?

OPTION #1

Should I always add a <link rel="canonical" href={"https://www.mycustomdomain.com/some-page"}/> for each page?

This way, if Google happens to crawl the default URLs, it should see that canonical pointing to the custom domain, right? Is this the correct approach?

OPTION #2

Since I'm dynamically generating the robots.txt, I could detect that the request is coming from URL_2 or 3 and simply block the entire website from being crawled, by adding Disallow: /.

How would you solve this?

Answer 1

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you suggest.

The other way is to simply issue 301 Permanent Redirect so that all traffic uses the official URLs. This will also allow you to enforce the use of HTTPS as it also happens that Googlebot indexes both the HTTP and HTTPS version when they exist and there is no redirect between them.

Given that you control the site, you are even free to not issue the redirect for internal traffic. This is common if the internal server does not have an SSL certificate. You can redirect when the IP address is internal to your network, for example.

Robots is generally not a good option because bots can choose to ignore it. Googlebot will respect it but you may have to cover for other bots.

Answer 2

those pages/URLs should NOT be indexed

Well, they should be indexed, but not at the other hostnames.

You should certainly implement OPTION#1 and configure a rel="canonical" element. This should also protect you against non-canonical query strings (intentional or malicious).

AND, ideally, you would 301 redirect any requests for non-canonical hostnames to the canonical one.

Blocking requests with robots.txt does not necessarily prevent indexing if the URLs are linked to. And if the non-canonical URL does get linked to you won't benefit from any SEO if crawling is blocked. You certainly should not block with robots.txt if you implement external redirects, as the redirects will not be followed.