Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
We can identify bots with their User-Agent (1, 2), but that's not so straightforward. The benefit is to skip large and probably useless resources for bots, and just serve core content
What's the best practice for this? Is it worth skipping superfluous resources for bots? and if so what's also the best practice for detecting bots?
Is this also fine to use a specific ?bot=true query string for example, to check it manually (and make sure those pages are not indexable) or would it hurt SEO
I'd have to know more about the specifics of your trackers and CRM to advise you to whether or not you should prevent bots from seeing them. You might want to prevent bots from seeing them if:
You should NOT use the User-Agent header to alter the page in any way for bots, including removing trackers and CRM. Google calls altering the page for Googlebot "cloaking" and will penalize sites that do so. Instead, there are approved way of preventing bots from hitting these.
robots.txtThe approved way of preventing bots from crawling your tracker and CRM is by including them in your robots.txt file. If your HTML includes
<script src=/crm.js></script>
<img src=/track.gif>
Then you can disallow them in your robots.txt:
User-Agent: *
Disallow: /crm.js
Disallow: /track.gif
If your trackers and CRM are hosted by third parties, you may find that the third parties already disallow them via robots.txt. If there third party doesn't block it, it is possible to link to a local URL that redirects to the third party and block your redirect URL in robots.txt.
The other way of preventing bots from seeing something is to trigger it based on things that bots don't do. This is a especially relevant for the CRM. For example, you can prevent bots from seeing a "Do you need help? Chat now" message.
Using ?bot=true is going to cause SEO problems. If you redirect bots to that, search engines will crawl and index that URL parameter. Then users will get directed to the bot pages.
Pardon my pun. I do come in peace (and respect your goal).
However, to answer your question, no it's not worth it. Mature Search Engine bots, such as Googlebot, are well trained at this point. If a resource doesn't help it understand a site or render content (.js) these resources will likely be ignored anyways.
Let's say you were to make this happen with a query string like ?bot=true. You'd have to first detect bots, then append URLs with your query string, and finally either dequeue resources via PHP or yoink them from the DOM via JavaScript (gross).
I'm sure there are other methods, but that's besides the point.
The remedy would be worse than the ailment - you'd trade skipping a handful of scripts (that Googlebot probably will ignore anyways) for doubling your crawlable/indexable URLs:
https://example.com/
https://example.com/?bot=true
To avoid that you'd then have to also enqueue the following annotations on the new query string pages you create:
<meta name="robots" content="noindex noarchive" /><link rel="canonical href="https://example.com/">Best practice for detecting a user-agent is via feature detection. Sniffing should be avoided. Read more.
Best practice for blocking bots is via robots.txt. Under normal circumstances we're only interested in blocking privacy sensitive content.
