"URL is unknown to Google" error on a "new" forum site

Question

Summary:

We've recently made certain sections of our (previously) private forum and community open so that users can access areas of the site without registering. The site, however, is encountering indexing issues...

The main page home page is being indexed and is discoverable via search. However, the other key pages, forum pages especially, aren't being indexed or crawled - and are potentially being blocked by our site privacy policy.

The site has been created using our learning management system.

(GSC live test screenshot of homepage)

Naturally, I requested Google to index certain pages via GSC, and I haven't heard anything back for more than two weeks.

Info on the URLs:

These are forum threads or "/discussion/" pages that return the "URL is unknown status to Google" status and are not being crawled or indexed:

https://example.com/mod/forum/discuss.php?d=25132

I just tested the live URL for these pages, and the live test page screenshot from GSC is showing our privacy policy instead of the actual page!

It seems like the privacy policy page has some sitewide robot.txt or CORS issues that are the root cause? (Again, I'm not super technical and this is really confusing me now!)

What do you guys think is the root cause of these indexing issues?

Other potential causes and contributors, but I'm betting on the problem being the site policy:

• No sitemap to help Google determine what the important pages are— although Google can crawl and index pages without there being a sitemap.

• Canonical URLs - by default, none of the pages have self-referencing canonical URL tags, probably related to the configuration of the site/platform.

• Many of the pages have the same </ H1 > although I can’t see why this would trigger crawl and index issues.

I've asked our team/devs who deal with hosting settings for the site to check if something could be preventing Googlebot from crawling and indexing the site properly (although I'm not sure if this requires a log file analysis or if it’s a simple act of checking firewall-related settings etc.)?

Answer 1

I believe the issue here is a lot simpler than you're expecting. It appears that your website is 303-redirecting users to your privacy policy page if they haven't accepted the privacy policy yet.

Googlebot can't accept the privacy policy and doesn't have any memory across page visits, it just follows the redirect every time. So all Googlebot sees is your privacy policy, for every discussion page on your site.

To get these pages indexed, you'll need to prevent your site from redirecting all of your pages to your privacy policy page for new users. Perhaps you can allow users to view your content without accepting the policy, and only require acceptance when they create an account or otherwise go to contribute?