41
votes
Prevent Google from thinking my own bot is faking traffic on my website
Google has no way of knowing when you hit your own site with your own bot. Google can't see anything that happens on your server.
Even if you use Google Analytics, your bot won't show up in it. ...
- 94.4k
35
votes
Accepted
Is there a difference between an empty robots.txt and no robots.txt at all?
Do crawlers behave differently in these two cases?
A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling.
You might however receive lots of ...
dan♦
- 15k
24
votes
Accepted
Only allow Google and Bing bots to crawl a site
The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing".
And yes, it means that they are not allowed ...
- 21.5k
21
votes
Will banning bots make a site harder to find on search engines?
Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your ...
- 313
19
votes
Accepted
Are web crawlers able to find a secondary robots.txt in a sub-directory?
No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site:
Where to put it
The short answer: in the top-level directory of ...
- 8,832
16
votes
Is there a difference between an empty robots.txt and no robots.txt at all?
No. There's no difference.
You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler ...
- 398
15
votes
Prevent Google from thinking my own bot is faking traffic on my website
Google doesn't care about your traffic. Even if you use Google Analytic, Google never uses that data for search engine ranking algorythm.
Gary Illyes from Google says in a tweet
We don't use anything ...
- 1,101
14
votes
Does the Google spider render JavaScript?
Google now officially processes JavaScript.
In order to solve this problem, we decided to try to understand pages by executing JavaScript. It’s hard to do that at the scale of the current web, but we ...
- 85.8k
14
votes
Accepted
Is it possible to keep the existence of a domain secret?
Is it possible to keep the existence of a domain secret?
Secret to whom? And incidentally, why does it need to be secret?
Secret to your OS, applications you use (including plugins in your browser, ...
- 7,202
13
votes
How does Google find a domain with no links to it?
There are at least 3 ways:
Links to your site.
Using Google Webmaster Tools (now called Search Console)
Registrar dumps, triggers, and other options.
Google will find many new sites quickly from ...
- 32.5k
13
votes
Accepted
Will banning bots make a site harder to find on search engines?
Banning bots will not let any search engine get the content of the site.
Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get ...
- 690
13
votes
Accepted
SEO - Responsive Website and Duplicated Menus
You have nothing to worry about. You can use display: none; to switch menus. Search engines are much better at understanding JS and CSS.
As long as you are not intentionally trying to manipulate ...
- 3,648
13
votes
How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?
In short: you can't.
Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html ...
- 131
12
votes
Accepted
Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?
You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.
RewriteRule ^robots\.txt$ /robots.php [L]
That way, you power it with a dynamic script, ...
- 94.4k
12
votes
Accepted
Should we drop AJAX crawling scheme?
Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site ...
- 8,832
12
votes
Is it possible to keep the existence of a domain secret?
No not possible. ICANN, the org maintaining DNS provides something that's called Centralized Zone Data Service (CZDS). Which provides lists of all registered domains.
Here is a bunch of info on that: ...
- 229
11
votes
Accepted
What bots are really worth letting onto a site?
Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search ...
- 32.5k
11
votes
Accepted
How to properly (dis)allow the archive.org bot? Did things change, if so when?
Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014).
Their FAQ How can I have my ...
- 21.5k
11
votes
Accepted
Googlebot submitting thousands of requests to our map locater and using up the API quota
To stop googlebot from searching via googlemaps
put a file named robots.txt in the root of your domain.
e.g. https://www.wikipedia.org/robots.txt
Sample robots.txt:
User-agent: Googlebot
Disallow: /...
- 226
10
votes
Accepted
Screen resolution of Googlebot mobile?
The current GoogleBot Smartphone agent, as tested with the 'Fetch as Google' Tool is essentially a fake iPhone using a headless Webkit Engine, running on a Linux x86_64 desktop machine.
The default ...
- 215
10
votes
Accepted
Does Google penalize sites for showing a different menu for mobile compared to desktop?
Google expects differences between mobile and desktop sites. Even major differences, including differences in link structure, are not a problem.
Google crawls the web with different Googlebot user ...
- 94.4k
10
votes
Accepted
How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?
Incorrectly used meta tags
Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two.
<meta name="...
- 94.4k
9
votes
Accepted
How does "Noindex:" in robots.txt work?
Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work.
Here is what Google's John Mueller says about Noindex: ...
- 94.4k
9
votes
Accepted
Can Googlebot discover URLs that are not in an anchor?
Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/
...
- 503
9
votes
Do search engines still crawl a noindex page
Yes, Google still crawl webpages that have noindex tag.
But if you have same content on two different webpages and one URL contain noindex tag, while second does not, then you should not worry about ...
- 5,487
8
votes
How to block baidu spiders
I have just successfully blocked the Chinese searchbot Baiduspider from accessing any content on my site. I made the decision to do so because of the following reasons.
Reasons for deciding to block
...
- 283
8
votes
Accepted
What determines the frequency the Wayback Machine crawls one's website?
The Wayback Machine archive is a combination of data from a large number of different crawls:
Alexa crawls, which appear after a 6 month delay
Our own crawls, which are seeded from the Alexa top ...
- 248
8
votes
Facebook crawler with no user agent spamming our site in possible DoS attack
Sources say that Facebook/Externalhit does not respect crawl-delay in robots.txt because Facebook doesn't use a crawler, it uses a scraper.
Whenever one of your pages is shared on Facebook, it ...
- 4,720
8
votes
Accepted
Ok to use meta noindex on 404 pages?
There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed.
If this was a PHP page then you could obviously just send a 404 Not Found header as part of the ...
- 41.4k
8
votes
Accepted
How do I "noindex" a text (.txt) file?
The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. ...
- 94.4k
Only top scored, non community-wiki answers of a minimum length are eligible