11

I have a large website which I want to be visited by my own bot early on the morning, every day (for caching purpose). I want my bot visits 10,000 pages per day.

My natural traffic is about 5000 pages per day. So I am worried that Google thinks I am faking traffic in order to manipulate the numbers, reputation or anything SEO-related.

The bot and the website will share the same IP so Google will understand easily that both are connected.

  • Do you think my bot will get me into trouble from Google?
  • If so, how could I prevent it?
Improve this question
5
  • 9
    Google will never know about your traffic unless it is from Google itself.
    – Muhammad Dyas Yaskur
    Commented Oct 11, 2021 at 10:08
  • 5
    And what about Google Analytics ? Google can see any trafic on my website because of GA tracker.
    – ordiminnie
    Commented Oct 11, 2021 at 10:12
  • 15
    What does "for caching purposes" mean? Are you trying to warm a cache? Which one? Why would it require twice as many hits to warm the cache as actual visits you get? It sounds like you should take a step back and see if there is a better way to achieve your goal.
    – Stephen Ostermiller ♦
    Commented Oct 11, 2021 at 10:20
  • 3
    @StephenOstermiller Yes I am warming my own cache (server cache, done by .net core framework). I have a large directory (+200 000 pages) but some pages are slow. We know the reason and are working on it, but, in the meantime, we are looking for a quicker work-around.
    – ordiminnie
    Commented Oct 11, 2021 at 10:37
  • 8
    @ordiminnie: Then don't use GA or have your bot block GA (like any reasonable normal visitor with UBO would be doing already).
    – R.. GitHub STOP HELPING ICE
    Commented Oct 11, 2021 at 18:31

4 Answers 4

Reset to default
41

Google has no way of knowing when you hit your own site with your own bot. Google can't see anything that happens on your server.

Even if you use Google Analytics, your bot won't show up in it. Google Analytics relies on client side JavaScript to record hits. Most bots don't execute JavaScript. If your bot does execute JS, you should be able to configure it not to hit www.google-analytics.com so that it doesn't execute the GA snippet. Alternately you could configure your web site not to include the GA tracking snippet when your bot hits.

Even if hits do show up in Google Analytics, Google says that it doesn't use GA data for ranking purposes.

Even if Google saw your bot hits in a place that it could use them for ranking purposes, it is unlikely that they would effect your rankings. Sites get crawled by bots all the time. In fact, more than half of all web traffic is bots, not actual users. There would be no reason for Google to adjust rankings based on whether or not a site is getting hit by a bot. I would just make sure that your bot uses a user-agent header that clearly identifies it as a bot.

The only way that this crawling could effect your SEO is indirectly. If this crawling were to slow down your site significantly to the point that users were getting frustrated, Google could notice that and adjust your rankings because of perceived performance problems or poor usability.

Improve this answer
0
15

Google doesn't care about your traffic. Even if you use Google Analytic, Google never uses that data for search engine ranking algorythm.

Gary Illyes from Google says in a tweet

We don't use anything from Google analytics in the "algo".

Improve this answer
3
  • It may still use it for discovery and crawling of previously unknown URLs. After all, if the webmaster installed GA and users are browsing it, clearly there is something there. They are saying "nothing with GA has any effect on rank", which is telling the SEO community not to waste time trying to game it.
    – Harper - Reinstate Monica
    Commented Oct 13, 2021 at 0:06
  • 1
    Google does always come out with advice against its vulnerabilities 😅
    – Mike Ciffone
    Commented Oct 13, 2021 at 5:17
  • 1
    Even if Google lies for some advice, I will always trust all advice from Google itself, unless there is any test which is proof that Google is lying.
    – Muhammad Dyas Yaskur
    Commented Oct 13, 2021 at 6:02
3

Google won't know about traffic to your website unless the 'bot (or human user's web browser) does TWO things:

  • Google Analytics asked you to install some Javascript on your web pages. The browser or bot must actually execute that Javascript code.
  • The Javascript code will ask the browser to send an HTTP request to a Google website. The browser or bot must actually make that request.
  • Google's website will expect a cookie and will try to set one if it doesn't get one. Your 'bot would need to store that cookie and send it on future queries, or likely be disregarded on the assumption that it is a 'bot.

Having written many 'bots, I can tell you it's rather easy to just not implement those features :) If it's somebody else's 'bot code you're reusing, then check it to make sure it doesn't do that.

Improve this answer
1
  • Other cases can happen very trivially. Imagine you have an image stored on some Google controlled property and the bot is constructed to mirror pages that is to download dependencies like images, then Google can know when "someone" visited the site even without javascript or cookies, depending on how the referer header field is used. Maybe not here, but those infamous 1x1 hidden images were used for tracking.
    – Patrick Mevzek
    Commented Oct 13, 2021 at 14:39
1

I will add a contrarian opinion regarding whether or not Google uses GA for SEO purposes. I have had at least two experiences where negative bots hit my website that impacted rankings. Not enough to slow my traffic down, just create a high bounce rate. However, within just a couple of hours I saw a significant drop in all page positions in GSC (10 position drop).

While Google says they don't use GA data, I'm not sure that any one person knows the whole algorithm. And remember, they're the one's who said for years there was no microphone in the Nest thermometer. Just my opinion.

Improve this answer
1
  • 1
    "I'm not sure that any one person knows the whole algorithm." I am pretty sure that even if someone knows the whole algorithm (which is already a pretty big if) then it is certainly not shared publicly so any users of Google have to just imagine what it is and be vulnerable to any change.
    – Patrick Mevzek
    Commented Oct 13, 2021 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.