I have a domain like example.com under which everything is blocked and links to an SPA.

Like example.com, and we've disabled everything like so:

User-agent: *
Disallow: /

Now this SPA uses parameters to fully control content on the site, such as for example:


Now there are many test_project values we use, and 100% of the time the content being displayed from these projects should never be crawled. So current rule works.

However we do get the request now that for example test_project_a specifically needs to have all its pages crawled, meaning we need some kind of exception to the Disallow rule whenever specifically the value of project is equal to test_project_a, but still apply to any other URL where this parameter's value is different.

I have seen on Google's documentation that there are some ways to manipulate parameters mostly to "block" some specific parameters but it doesn't appear to be based on the exact value, only the parameter name, for example it will check project= but not test_project_a, at least as far as I understand.

Does anyone know any way to achieve this?

  • Can you switch to using history.pushState() to remove the # from your URLs? Here is Google's guide to fixing SEO for an SPA that includes that suggestion: codelabs.developers.google.com/codelabs/… Jan 25, 2022 at 9:51
  • Hello @StephenOstermiller, is the # at the beginning of the URL really an issue? I feel like the question is more about the parameter "values" rather than the # sign. Is there correlation between the two that I don't see? Jan 26, 2022 at 2:57
  • Anything after the # is called the "fragment identifier." It typically instructs browsers to scroll to a specific place in a document. Any "parameters" used in it aren't really parameters. There is no formal structure for it. When you use a # right after the domain name, there is only one document to download from your site which can really throw crawlers for a loop because they have to "crawl" virtual pages that look like they don't exist. Jan 26, 2022 at 10:23


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.