I have loads of booking engine pages being indexed. I don't want to block the actual booking page from Google as it's helpful for Organic traffic, but I have lots of useless pages from the booking engine being indexed too:

https://book.condorferries.co.uk/en/author/crdaru/ https://book.condorferries.co.uk/ag-fr/category/uncategorized/https://book.condorferries.co.uk/en/cw_content_type/vehicle/ https://book.condorferries.co.uk/en/cw_content_type/cabin/

To name a few, but there are a lot - about 700

Do I add these URLs individually to robots.txt or leave as they are?

Thank you

  • Why would you create useless pages on your site? If the pages aren't useful, get rid of them. – Stephen Ostermiller Jan 26 '19 at 9:57

If you are worried that search crawlers are spending too much time trying to index those pages, you can add them to your robots.txt, either individually or use a prefix to disallow all pages that start with a specific string.

This will send a signal to the crawlers not bother ever fetching those pages, so the crawlers will spend more time looking at your useful pages instead.

Another option is to use meta noindex on them instead of robots.txt if you want to specifically tell the crawler to de-index the pages as it crawls. But it sounds like you just want the crawler to ignore the pages altogether, not bother crawling them in the first place, and spend its time crawling elsewhere - which is robots.txt disallow is best for.

Edit: I just visited your links and it seems like the pages are completely useless, and should never be visited by anyone including people or bots. Is that a fair characterization? In that case, don't mess around with robots.txt or noindex, just configure your web server not to serve the pages.

  • Yes they are completely useless. I'm dealing with lots of useless pages from a site redesign where the agency didn't know much about websites.... I don't know anything about configuring web servers - will do some research! – Becky Jan 28 '19 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.