1

I'm building a webapp using Nextjs. As some parts of the app is server side rendering, I have two sitemaps. One generated by Next and the other one generated dynamically.

All pages are successfully added to the search console. However I have also some urls including the bracket slugs like

https://mywebsite.com/[mySlug]
https://mywebsite.com/[mySlug]/aaa
https://mywebsite.com/[myOtherSlug]/bbb

Those urls are not in the sitemaps. Do you have insight on why those urls are present into the search console results?

To be clearer, I have urls like

https://mywebsite.com/my_slug_1
https://mywebsite.com/my_slug_1/bbbb

And they are indexed correctly but the fact that https://mywebsite.com/[mySlug] is indexed is not correct as the bracket slug is just a way to tell Next that the [mySlug] can be any string possible

Thanks!

3
  • Do these URLs show as indexed? Do they work (200 status)? – Kannan Jun 21 at 12:55
  • @Kannan Those url are not indexed as the search console listed them as Error 500 (which is normal) as [mySlug] can be any string possible but the url containing the precise slug [mySlug] should not be visible and crawled as there are no link leading to it – BghinC Jun 21 at 14:45
  • 1
    The only way Google could have found those links is via internal/external links. Since these are not valid URLs, you may have to check for unintended links leading to those URLs. – Kannan Jun 21 at 15:23
1

Google doesn't rely just on XML sitemaps to know what pages you have on your site and which URLs it should index. See my diatribe on the subject: SEO Myth: Google will only index the pages listed in your XML sitemap.

Google will index URLs that are not in your sitemap if:

  • Those URLs have had links either from your own site or from another site. It doesn't matter if the links still exist, once Googlebot finds a page through a link it tends to remember it forever.
  • The URLs return 200 OK status with unique content.

If the URLs duplicate other content from your site for which the preferred URL is in your sitemap, Google will usually (but not always) index the URL from your sitemap rather than the URL it finds only through links.

The best way to deal with URLs that you don't want to have indexed it to do one of the following:

  • Redirect them to a different URL that you do want indexed to get a different URL with the same content indexed instead.
  • Put a <meta name="robots" content="noindex"> tag in place to instruct search engines not to index them.
  • Password protect them so that Googlebot can't see the contents when it crawls.
  • Disallow them in robots.txt so that Googlebot doesn't even try to crawl them. (Although Google will occasionally index pages it can't crawl, so this method is best for pages that can appear in search results but which you don't want crawlers hitting.)
  • Return "410 Gone" status to mark them as removed.

Even if the URLs are not indexed, you may see them in various Search Engine Console reports. Search Engine Console reports on all the URLs from your site that Googlebot crawls, not just the ones that are in your sitemap or the ones which Google chooses to index.

3
  • I might not be clear in the initial question. Those urls are not available anywhere in the app as it is just brackets slug. mywebsite.com/my_slug_1 exists as the slug is valid but mywebsite.com/[mySlug] doesn't as it is a way to tell Next [mySlug] can be any string possible. Moreover, those urls returns an error 500 and listed as error 500 in the search console – BghinC Jun 21 at 14:40
  • Googlebot scans JavaScript code for things that look like they could be links. It then treats them as if they are links, crawls them, and reports on them in search console. Your JS code may have something like var s = "/[mySlug]" in it. This doesn't hurt your site at all other than adding cruft to the reporting in GSC. – Stephen Ostermiller Jun 21 at 16:57
  • Ok great I think this is what is happening. Too bad it's "polluting" the GSC with those errors but if there is no way to remove then, that's it. Thank you for your answers ! – BghinC Jun 22 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.