3

We can identify bots with their User-Agent (1, 2), but that's not so straightforward. The benefit is to skip large and probably useless resources for bots, and just serve core content

What's the best practice for this? Is it worth skipping superfluous resources for bots? and if so what's also the best practice for detecting bots?

Is this also fine to use a specific ?bot=true query string for example, to check it manually (and make sure those pages are not indexable) or would it hurt SEO

1
+50

I'd have to know more about the specifics of your trackers and CRM to advise you to whether or not you should prevent bots from seeing them. You might want to prevent bots from seeing them if:

  • It will mess up your tracking stats for bot hits
  • The CRM has has different keywords for different users, or doesn't appear to all users.
  • The CRM has popups that cover your main content.

You should NOT use the User-Agent header to alter the page in any way for bots, including removing trackers and CRM. Google calls altering the page for Googlebot "cloaking" and will penalize sites that do so. Instead, there are approved way of preventing bots from hitting these.

Use robots.txt

The approved way of preventing bots from crawling your tracker and CRM is by including them in your robots.txt file. If your HTML includes

<script src=/crm.js></script>
<img src=/track.gif>

Then you can disallow them in your robots.txt:

User-Agent: *
Disallow: /crm.js
Disallow: /track.gif

If your trackers and CRM are hosted by third parties, you may find that the third parties already disallow them via robots.txt. If there third party doesn't block it, it is possible to link to a local URL that redirects to the third party and block your redirect URL in robots.txt.

Trigger on something only users do

The other way of preventing bots from seeing something is to trigger it based on things that bots don't do. This is a especially relevant for the CRM. For example, you can prevent bots from seeing a "Do you need help? Chat now" message.

  • Trigger something only on the 2nd (or later) page view based on cookies. Bots don't retain cookies between page views.
  • Trigger something using JavaScript after an on-page delay. Most bots don't execute JavaScript. A few, including Googlebot do. Even when Googlebot runs JavaScript, it only does so for about 5 seconds per page. If you were to trigger something after a user has been on the page for 10 seconds, bots are not going to see it.
  • Trigger something using JavaScript on user interaction. Even when bots run JavaScript, they don't simulate interactions with the page. Anything triggered when the user clicks, scrolls, moves the mouse, or otherwise interacts with the page is going to be invisible to bots.

Using ?bot=true is going to cause SEO problems. If you redirect bots to that, search engines will crawl and index that URL parameter. Then users will get directed to the bot pages.

1
  • trackers: google analytics, facebook pixel - CRM: Intercom, but thanks that answers perfectly, we managed to load intercom on click and for trackers they're most likely ignored as you explained, thanks
    – caub
    Dec 31 '21 at 19:23
1

Instructing bots to skip superfluous scripts in and of itself seems superfluous

Pardon my pun. I do come in peace (and respect your goal).

However, to answer your question, no it's not worth it. Mature Search Engine bots, such as Googlebot, are well trained at this point. If a resource doesn't help it understand a site or render content (.js) these resources will likely be ignored anyways.

Let's say you were to make this happen with a query string like ?bot=true. You'd have to first detect bots, then append URLs with your query string, and finally either dequeue resources via PHP or yoink them from the DOM via JavaScript (gross).

I'm sure there are other methods, but that's besides the point.

The remedy would be worse than the ailment - you'd trade skipping a handful of scripts (that Googlebot probably will ignore anyways) for doubling your crawlable/indexable URLs:

https://example.com/
https://example.com/?bot=true

To avoid that you'd then have to also enqueue the following annotations on the new query string pages you create:

  • <meta name="robots" content="noindex noarchive" />
  • <link rel="canonical href="https://example.com/">

Outstanding Questions

  • Best practice for detecting a user-agent is via feature detection. Sniffing should be avoided. Read more.

  • Best practice for blocking bots is via robots.txt. Under normal circumstances we're only interested in blocking privacy sensitive content.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.