0

TLTR

I have a website (SPA) where SEO is very important. It was very poorly indexed and search console tells me it is due to 'noindex' in the robots meta, while I cannot see how Google comes to that conclusion.

BACKGROUND

I have this Single Page Application built with VueJS, which I launched at the start of this year. I noticed it was poorly indexed in Google. Google has missed out on indexing indexed a lot of my pages; including the most important ones with the most content. I do control the robots meta tag trough javascript, but cannot see how this would cause these issues.

CASE

I will post an example URL here where indexing by Google is blocked, while I want it to be indexed and give all information/research I have found about it:

URLS:

https://trainervinden.nl/blog/afvallen-met-een-personal-trainer

Alternative URL, different page type, same results: https://trainervinden.nl/trainer/marit-de-heer

Sitemap

The page is part of the sitemap: https://trainervinden.nl/sitemap.xml

Robots.txt

We don't have one AFAIK.

Chrome Inspector

I see the robots meta tag in header from Chrome inspector, and it looks good to me:

Meta tag in header from inspector

Page Source

Default robots meta tag from page source (it is a SPA though, and robots meta tag is also controlled by javascript): Robots meta in Page source from SPA

Web Developer Tools Chrome Plugin

Response headers, looking good to me. No robots stuff here:

Result response headers Web Developer Tools

Meta tags from Web Developer Tools, still looking as I expect:

Meta tags from Web Developer Tools

Google Search Console

Pushing the button Test live url (the same results as the existing info behind tab Google Index) gives results I do not understand:

Result for Test live url

I can provide any extra information needed to solve this issue; just ask!

So basically I have no idea where this "no-index" Google sees is coming from and am a web guy in despair right now. Any hint helping me out is much appreciated; more than you can imagine at this point :)

Improve this question
4
  • Why do you describe that as a single page site? It clearly has lots of pages.
    – Stephen Ostermiller ♦
    Mar 23, 2021 at 20:39
  • Hi Stephen, a SPA (Single Page Application) is just a website that uses javascript to render the website in the client (browser of the user). It is a technique rather than a limit on number of pages and my website is a SPA.
    – Rogier
    Mar 24, 2021 at 14:10
  • I would use the terminology "JavaScript rendered" rather than "single page application" in such a case.
    – Stephen Ostermiller ♦
    Mar 24, 2021 at 15:16
  • Sure, although it is a really widely used term for these modern kind of applications.
    – Rogier
    Mar 24, 2021 at 19:32

1 Answer 1

Reset to default
2

"index" is not an approved value for use in a robots meta tag. See: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag Instead you should use "all".

<meta name=robots content=all>

Alternatively, you can omit the meta robots tag altogether. Without a tag, the default is to allow crawling and indexing.

I think is what is happening is that Google is seeing "index" and matching it to the closest possible approved value, which is "noindex."

Improve this answer
4
  • Hi Stephen, thanks for your answer. I have changed the content to all as you suggested. I cannot believe I missed that. My issue is still not fixed though. The problem persists. I have found out a way of reproducing the issue though. If I check the pages through search.google.com/test/mobile-friendly I see page is not found (while it is found through normal browser). The fact that it is not found redirects the user to a "not found" page which sets the robots meta tag to noindex.
    – Rogier
    Mar 24, 2021 at 14:16
  • I'm not sure how the mobile friendly test gets to the 404 page. I get a page on the command line with curl --head https://trainervinden.nl/blog/afvallen-met-een-personal-trainer which results in a "200 OK" status. Testing in a variety of browsers shows problems with Internet Explorer not showing the page, but IE doesn't show a 404 error either: app.crossbrowsertesting.com/public/i2a11f6a3a18c91f/screenshots/…
    – Stephen Ostermiller ♦
    Mar 24, 2021 at 15:26
  • I am trying to figure out what is going wrong right now. Will let you know if I find out what is going on. It is actually not a 404 status even if the element is not found. It seems to be due to CORS issues with the backend.
    – Rogier
    Mar 24, 2021 at 19:34
  • The issues have been fixed and were cause by a robots.txt file on the api endpoints... The noindex was due to a redirect to a not found if an error occurred when the data was fetched I built into the frontend
    – Rogier
    Mar 24, 2021 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.