I'd have to know more about the specifics of your trackers and CRM to advise you to whether or not you should prevent bots from seeing them. You might want to prevent bots from seeing them if:
- It will mess up your tracking stats for bot hits
- The CRM has has different keywords for different users, or doesn't appear to all users.
- The CRM has popups that cover your main content.
You should NOT use the
User-Agent header to alter the page in any way for bots, including removing trackers and CRM. Google calls altering the page for Googlebot "cloaking" and will penalize sites that do so. Instead, there are approved way of preventing bots from hitting these.
The approved way of preventing bots from crawling your tracker and CRM is by including them in your
robots.txt file. If your HTML includes
Then you can disallow them in your
If your trackers and CRM are hosted by third parties, you may find that the third parties already disallow them via
robots.txt. If there third party doesn't block it, it is possible to link to a local URL that redirects to the third party and block your redirect URL in
Trigger on something only users do
The other way of preventing bots from seeing something is to trigger it based on things that bots don't do. This is a especially relevant for the CRM. For example, you can prevent bots from seeing a "Do you need help? Chat now" message.
- Trigger something only on the 2nd (or later) page view based on cookies. Bots don't retain cookies between page views.
?bot=true is going to cause SEO problems. If you redirect bots to that, search engines will crawl and index that URL parameter. Then users will get directed to the bot pages.