2

I want to check which URLs are indexed in google. But I want to exclude all URLs containing a question mark in the URL.

For example I want a list of pages like:

  • https://example.com
  • https://example.com/hello
  • https://example.com/welcome.php
  • https://example.com/folder/

but not pages containing a question mark like:

  • https://example.com?
  • https://example.com/hello?par=1
  • https://example.com/welcome.php?par=2
  • https://example.com/folder/?par=3

What I tried so far:

I tried several Google site searches

  • site:https://example.com -site:?
  • site:https://example.com -?
  • site:https://example.com -inurl:"?"

But this doesn't work, or does not work as required. It it possible to get the results I want, and if so, how?

Why I am asking:

We want to have more control over what pages are indexed in google and which pages google gives attention to. We don't want pages containing a query string to be indexed at all. Those pages have either a canonical tag (some of these pages are still indexed) or a noindex tag. We set up changes in our website structure and would like to track the effects in the serps.

Related to this is our desire to limit the number of pages crawled. Currently google crawls about 250k pages of our site of which only 25k are eligible for being indexed.

  • Google site: searches don't list all the pages indexed anyway. You'll need to get the data from search console rather than from searching with site: on Google. – Stephen Ostermiller Jun 4 at 13:27
  • 2
    Why do you want to filter by no query string? Google views pages with query strings as completely different URLs, so it is important to see if Google is indexing some URLs that way so that you can redirect or set canonicals to resolve duplication. – Stephen Ostermiller Jun 4 at 13:28
  • @StephenOstermiller The google search console only allows 1.000 pages to be analyzed. It is also not possible to filter those pages, at least not in the way we want to. I updated my question to give a little background on why I am asking this. – Nebu Jun 5 at 9:37
  • "pages have either a canonical tag (some of these pages are still indexed)" - Just to clarify... a canonical tag doesn't prevent pages from being indexed (after all, the pages need to be indexed in order to determine that one is canonical). However, the canonical tag is intended to prevent the non-canonical pages being returned in "normal" search results. Note that a site: search is not a "normal" search - a site: search can return non-canonical URLs and even URLs that are redirected, which would otherwise be omitted from "normal" searches. – MrWhite Jun 5 at 11:13
1

Google says that the counts reported by site: queries are estimates. In my experience, they are not accurate at all. Even if you found a Google site: query that returned the results you want, it isn't going to give you useful information.

The better way is to track your coverage in Google Search Console.

  1. Create an XML sitemap containing only the 25K URLs without a question mark that you want to be indexed.
  2. Add that sitemap to Google Search Console.
  3. Wait until Google reports how many URLs are discovered in it (it could say zero for quite a while)
  4. Click on the sitemap and then select "See Index Coverage"
  5. That will show you are report of exactly how many are "errors, "valid with warnings", "valid", and "excluded". You will be able to track it over time, as well as dive into any errors or excluded URLs and figure out which URLs have issues and get more information about any issues.
    enter image description here
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.