Skip to main content
Share Your Experience: Take the 2024 Developer Survey
41 votes

Prevent Google from thinking my own bot is faking traffic on my website

Google has no way of knowing when you hit your own site with your own bot. Google can't see anything that happens on your server. Even if you use Google Analytics, your bot won't show up in it. ...
Stephen Ostermiller's user avatar
15 votes

Prevent Google from thinking my own bot is faking traffic on my website

Google doesn't care about your traffic. Even if you use Google Analytic, Google never uses that data for search engine ranking algorythm. Gary Illyes from Google says in a tweet We don't use anything ...
Muhammad Dyas Yaskur's user avatar
14 votes
Accepted

Is it possible to keep the existence of a domain secret?

Is it possible to keep the existence of a domain secret? Secret to whom? And incidentally, why does it need to be secret? Secret to your OS, applications you use (including plugins in your browser, ...
Patrick Mevzek's user avatar
13 votes

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

In short: you can't. Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html ...
cusco's user avatar
  • 131
12 votes

Is it possible to keep the existence of a domain secret?

No not possible. ICANN, the org maintaining DNS provides something that's called Centralized Zone Data Service (CZDS). Which provides lists of all registered domains. Here is a bunch of info on that: ...
sgtfrankieboy's user avatar
11 votes
Accepted

Googlebot submitting thousands of requests to our map locater and using up the API quota

To stop googlebot from searching via googlemaps put a file named robots.txt in the root of your domain. e.g. https://www.wikipedia.org/robots.txt Sample robots.txt: User-agent: Googlebot Disallow: /...
satibel's user avatar
  • 226
10 votes

Should I publish an empty ads.txt file for a site that does not run any ads?

Yes. According to version 1.0.2 of the ads.txt specification: 3.2.1 FILES WITHOUT AUTHORIZED ADVERTISING SYSTEM RECORDS Some publishers may choose to not authorize any advertising system by ...
Royce Williams's user avatar
10 votes
Accepted

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

Incorrectly used meta tags Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two. <meta name="...
Stephen Ostermiller's user avatar
9 votes
Accepted

What determines the frequency the Wayback Machine crawls one's website?

The Wayback Machine archive is a combination of data from a large number of different crawls: Alexa crawls, which appear after a 6 month delay Our own crawls, which are seeded from the Alexa top ...
Greg Lindahl's user avatar
9 votes
Accepted

Can Googlebot discover URLs that are not in an anchor?

Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/ ...
Milo Tischler's user avatar
9 votes

Do search engines still crawl a noindex page

Yes, Google still crawl webpages that have noindex tag. But if you have same content on two different webpages and one URL contain noindex tag, while second does not, then you should not worry about ...
Goyllo's user avatar
  • 5,507
8 votes

Facebook crawler with no user agent spamming our site in possible DoS attack

Sources say that Facebook/Externalhit does not respect crawl-delay in robots.txt because Facebook doesn't use a crawler, it uses a scraper. Whenever one of your pages is shared on Facebook, it ...
Michael d's user avatar
  • 4,803
8 votes
Accepted

Ok to use meta noindex on 404 pages?

There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed. If this was a PHP page then you could obviously just send a 404 Not Found header as part of the ...
MrWhite's user avatar
  • 42.8k
8 votes
Accepted

How do I "noindex" a text (.txt) file?

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. ...
Stephen Ostermiller's user avatar
8 votes
Accepted

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek ...
Maximillian Laumeister's user avatar
8 votes

Is it possible to keep the existence of a domain secret?

They do have a list of all domains. Agencies responsible for many TLDs (top level domain, e.g. .com), publish a zone file, a list of all registered domains in that TLD, along with their DNS servers. ...
Harper - Reinstate Monica's user avatar
8 votes

Does googlebot crawl the variable name of a php

Except in the case of misconfiguration, the PHP code is executed on the server and what gets transmitted to the client is only the result of the execution which may be a mix of static HTML + dynamic ...
Patrick Mevzek's user avatar
7 votes

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt: User-agent: MJ12bot Disallow: /
Sjoerd Linders's user avatar
7 votes
Accepted

Do search engines still crawl a noindex page

As Goyllo has already stated, search engine bots will crawl pages that have a noindex meta tag. If you think about it, they need to crawl the page in order to see the noindex meta tag in the first ...
MrWhite's user avatar
  • 42.8k
7 votes
Accepted

Should I publish an empty ads.txt file for a site that does not run any ads?

Ads.txt has zero to do with SEO or even UX. It is specific to programmatic advertising. It might be of interest if your websites display ads that are purchased on real time bidding (RTB) exchanges. ...
Henry Visotski's user avatar
7 votes
Accepted

Ampersand (&) in actual URL and sitemap

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with &amp; ... ...
MrWhite's user avatar
  • 42.8k
7 votes

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

Not only can you not ensure it, it's highly likely that the link will be retrieved by programs not under the user's control. Many spam and phishing filters will pre-fetch any web pages linked to in ...
Mark's user avatar
  • 236
7 votes
Accepted

Do SEO analysis bots have any advantage in terms of website traffic?

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic ...
Matthew Edgar's user avatar
7 votes
Accepted

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/ This does look like a mistake in the Google docs IMO (as you suggest) for several reasons: That example ...
MrWhite's user avatar
  • 42.8k
6 votes

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I tried the robots.txt method and it didn't work. So I contacted the website on their email [email protected]: Hello, Can you please remove my personal website dimitarnestorov.com from your ...
Dimitar Nestorov's user avatar
6 votes

How to properly (dis)allow the archive.org bot? Did things change, if so when?

There are really 2 issues here: Will the robots.txt on your site Disallow (block) Wayback from crawling your site. Will Wayback crawl your site. For point #1: As others have said, the correct entry ...
Kevin Fegan's user avatar
6 votes

Are Googlebot and other search engine crawlers able to detect if element is hidden after page load by JavaScript?

Googlebot now renders pages and views the page as a user sees it when it loads, including applying CSS and running JavaScript. Google will detect text that is hidden using either CSS or JavaScript. ...
Stephen Ostermiller's user avatar
6 votes
Accepted

How to block only Yandex bot using robots.txt

There are many "Yandex bots". If you want to block all the Yandex bots then: User-agent: Yandex Disallow: / However, if you only want to block the main indexing "YandexBot" then: User-agent: ...
MrWhite's user avatar
  • 42.8k
6 votes

Secure way to share unfinished web-page with a customer?

If you know your customer IP Address, the safer way to share unfinished web-page is to deny access to everyone but you and it in a .htaccess file at the root of your working directory. order deny,...
gael's user avatar
  • 1,050
6 votes
Accepted

Why are bots repeatedly crawling pages of my site that I never created?

Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site. An old site These ...
Stephen Ostermiller's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible