I'm trying to properly index a multi-lingual express app which is serving an angular universal application. The app redirects the user based on their preferred language, if the provided path is not localized:
If preferred language is German:
/path/to/content -> /de/path/to/content
The redirection happens thanks to this piece of express code:
function app() {
const server = express();
const languages = ['es', 'en', 'de'];
# Provide the localized builds of the angular app for each localized path
languages.forEach((locale) => {
const appServerModule = getAppServerModule(locale);
server.use(`/${locale}`, appServerModule.app(locale));
});
# redirect request to the localized version
server.get('*', (req, res, next) => {
var acceptedLanguage = req.acceptsLanguages(...languages);
if (acceptedLanguage) {
var locale = languages.find(i => i === acceptedLanguage);
req.url = '/' + locale + req.url;
} else {
# If the preferred language cannot be provided, redirect to english
req.url = '/en/' + req.url;
}
return server._router.handle(req, res, next);
});
return server;
}
If no language matches, the English version is provided.
The problem is that the google crawler is getting for some reason the Spanish version for the unlocalized paths.
I guess a reason could be that I'm not redirecting but changing the route, but this was the only successful way that I found to redirect to the different localized angular universal app, although it looks quite hacky.
From a user perspective it seems to be working fine, so I wonder if a possible solution could be to just block the unlocalized paths from being indexed with use of robots.txt:
# Allow all URLs (see http://www.robotstxt.org/robotstxt.html)
user-agent: *
# Disallow not localized paths
disallow: /*
# Allow localized paths
allow: /en/*
allow: /es/*
allow: /de/*
Does anybody know if that would work? Are there any better option?
--- Update
The robots.txt strategy doesn't seem to solve the problem