1

I've started working on our website and I've found millions of "Search" URL's which I don't think should be getting crawled & indexed (e.g. .../search/?q=brown&prefn1=brand&prefv1=C.P. COMPANY|AERIN|NIKE|Vintage Playing Cards|BIALETTI|EMMA PAKE|QUILTS OF DENMARK|JOHN ATKINSON|STANCE|ISABEL MARANT ÉTOILE|AMIRI|CLOON KEEN|SAMSONITE|MCQ|DANSE LENTE|GAYNOR|EZCARAY|ARGOSY|BIANCA|CRAFTHOUSE|ETON).

I tried to disallow them on the Robots.txt file, but our Sessions dropped about 10% and our Average Position on Search Console dropped 4-5 positions over 1 week. Looks like over 50 Million URL's have been blocked, and all of them look like all of them are like the example above and aren't getting any traffic to the site.

I've allowed them again, and we're starting to recover. We've been fixing problems with getting the site crawled properly (Sitemaps weren't added correctly, products blocked from spiders on Categories pages, canonical pages being blocked from Crawlers in robots.txt) and I'm thinking Google were doing us a favour and using these pages to crawl the product pages as it was the best/only way of accessing them.

Should I be blocking these "Search" URL's, or is there a better way about going about it??? I can't see any value from these pages except Google using them to crawl the site.

|improve this question
2

You should Disallow those search URLs even if it results in a drop in traffic. It is against Google's webmaster guidelines to have your site search results pages indexed. Having indexed site search results pages could cause Google to penalize your entire site.

Matt Cutts explains Google's reasoning on this: Search results in search results. It is bad user experience for users to click from the Google search results just to see another set of search results. There are a potentially infinite number of URLs that can be created by your site search, many of which duplicate each other and the other content on your site.

If you are finding that Google is using these pages to crawl your site fully, you need to find other ways to interlink the pages on your site. As a general rule, every page on your site should link to ten other related pages somehow.

|improve this answer
  • Thanks @Stephen, that was my thinking too! I've been reading up about it; I think noindex might be a better way to go! That way they can keep crawling product pages, but "Search Results" won't show in on Google. – Frankie Kilkelly Jul 25 '19 at 16:13
  • That might buy you some time, but eventually the result will be the same. Google has said Googlebot treats noindex,follow and noindex,nofollow the same. That is Google doesn't pass link juice through non-indexed pages. Google: Long Term Noindex Will Lead To Nofollow Also You will need to implement better interlinking between your pages at some point in the next few weeks if you want to keep your rankings. – Stephen Ostermiller♦ Jul 25 '19 at 16:23
  • Cool, thanks Stephen! We've sorted out our Sitemaps, and links from Category pages to products (that were blocked for some reason) have been allowed, so I think it all should work out okay and interlink smoothly. Hopefully we shouldn't need the noindex pages fairly soon – Frankie Kilkelly Jul 25 '19 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.