Booking Engine Pages Being Indexed

Question

I have loads of booking engine pages being indexed. I don't want to block the actual booking page from Google as it's helpful for Organic traffic, but I have lots of useless pages from the booking engine being indexed too:

https://book.condorferries.co.uk/en/author/crdaru/ https://book.condorferries.co.uk/ag-fr/category/uncategorized/https://book.condorferries.co.uk/en/cw_content_type/vehicle/ https://book.condorferries.co.uk/en/cw_content_type/cabin/

To name a few, but there are a lot - about 700

Do I add these URLs individually to robots.txt or leave as they are?

Thank you

Answer 1

If you are worried that search crawlers are spending too much time trying to index those pages, you can add them to your robots.txt, either individually or use a prefix to disallow all pages that start with a specific string.

This will send a signal to the crawlers not bother ever fetching those pages, so the crawlers will spend more time looking at your useful pages instead.

Another option is to use meta noindex on them instead of robots.txt if you want to specifically tell the crawler to de-index the pages as it crawls. But it sounds like you just want the crawler to ignore the pages altogether, not bother crawling them in the first place, and spend its time crawling elsewhere - which is robots.txt disallow is best for.

Edit: I just visited your links and it seems like the pages are completely useless, and should never be visited by anyone including people or bots. Is that a fair characterization? In that case, don't mess around with robots.txt or noindex, just configure your web server not to serve the pages.