41
Google has no way of knowing when you hit your own site with your own bot. Google can't see anything that happens on your server.
Even if you use Google Analytics, your bot won't show up in it. Google Analytics relies on client side JavaScript to record hits. Most bots don't execute JavaScript. If your bot does execute JS, you should be able to ...
answered Oct 11 '21 at 10:52
34
Do crawlers behave differently in these two cases?
A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling.
You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here.
So, is it safe to just delete an empty robots....
24
The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing".
And yes, it means that they are not allowed to crawl anything.
You might want to omit the * in /bedven/bedrijf/*.
In the original robots.txt specification, * has no special meaning, it’s just a ...
21
Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt.
Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.
19
No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site:
Where to put it
The short answer: in the top-level directory of your web server.
or on Google's help pages (emphasis mine):
A robots.txt file is a file at the root of your site that indicates those parts of your site you don’...
15
No. There's no difference.
You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.
15
Google doesn't care about your traffic. Even if you use Google Analytic, Google never uses that data for search engine ranking algorythm.
Gary Illyes from Google says in a tweet
We don't use anything from Google analytics in the "algo".
14
Google now officially processes JavaScript.
In order to solve this problem, we decided to try to understand pages by executing JavaScript. It’s hard to do that at the scale of the current web, but we decided that it’s worth it. We have been gradually improving how we do this for some time. In the past few months, our indexing system has been rendering a ...
14
Is it possible to keep the existence of a domain secret?
Secret to whom? And incidentally, why does it need to be secret?
Secret to your OS, applications you use (including plugins in your browser, etc.), ISP, DNS resolver you use (any public one?), registrar and registry involved, certainly not. Which makes already a long list.
(Add to that, potentially, ...
13
There are at least 3 ways:
Links to your site.
Using Google Webmaster Tools (now called Search Console)
Registrar dumps, triggers, and other options.
Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...
13
Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure
If your site is password protected properly, there shouldn't be any need to block particular user agents :) x
13
Banning bots will not let any search engine get the content of the site.
Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic.
Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...
13
You have nothing to worry about. You can use display: none; to switch menus. Search engines are much better at understanding JS and CSS.
As long as you are not intentionally trying to manipulate things to get a better ranking. Using display: none; to hide big blocks of text will get you penalized. So if you are only using to hide your desktop menu on ...
13
In short: you can't.
Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html meta tags.
__
However (a bit outside of the scope):
You should have the <token> be a long format random as stated on a comment to your question.
You ...
12
You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.
RewriteRule ^robots\.txt$ /robots.php [L]
That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...
answered Oct 22 '15 at 9:23
12
Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site being indexable by search engines other than Google, you may want to still support this scheme.
Certainly, if you already have set up support for ...
12
No not possible. ICANN, the org maintaining DNS provides something that's called Centralized Zone Data Service (CZDS). Which provides lists of all registered domains.
Here is a bunch of info on that: https://czds.icann.org/help
11
Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...
11
Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014).
Their FAQ How can I have my site's pages excluded from the Wayback Machine? refers to Removing Documents From the Wayback Machine, which documents that their bot is called ia_archiver.
So ...
11
To stop googlebot from searching via googlemaps
put a file named robots.txt in the root of your domain.
e.g. https://www.wikipedia.org/robots.txt
Sample robots.txt:
User-agent: Googlebot
Disallow: /search-store/
Where /search-store/ is the page that sends the request to google maps.
If it happens to be something else than Googlebot, you can try disabling ...
10
The current GoogleBot Smartphone agent, as tested with the 'Fetch as Google' Tool is essentially a fake iPhone using a headless Webkit Engine, running on a Linux x86_64 desktop machine.
The default non-responsive viewport width is that of an iPhone at 980px.
With a viewport
<meta name="viewport" content="width=device-width, initial-scale=1"> applied, ...
10
Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
10
I wouldn't count on all spiders being able to follow a redirect to get to a robots.txt file. See: Does Google respect a redirect header for robots.txt to a different file name?
Assuming you are hosted on an Apache server, you could use mod_rewrite from your .htaccess file to to serve the correct file for the correct domain:
RewriteEngine On
RewriteCond %{...
answered Apr 25 '14 at 18:15
10
Google expects differences between mobile and desktop sites. Even major differences, including differences in link structure, are not a problem.
Google crawls the web with different Googlebot user agents for mobile. As long as your server shows that version of Googlebot the same thing that your actual mobile users see, you don't have any penalty risk. ...
answered Jan 13 '15 at 14:51
10
Incorrectly used meta tags
Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two.
<meta name="robots" content="noindex,nofollow">
Robots.txt and robots meta tags are mutually exclusive
Disallowing pages in robots.txt prevents robots ...
answered Dec 8 '20 at 16:45
9
You can add a dollar sign to the end of the string which means it will only match exactly that entry:
# Files
User-agent: *
Disallow: /mage$
This will only block the mage file if it come straight after the root domain:
www.example.com/mage
If there are any other preceding directories, you must add these o the entry. So to block the file located below:
...
9
Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work.
Here is what Google's John Mueller says about Noindex: in robots.txt:
We used to support the no-index directive in robots.txt
as an experimental feature.
But it's something that I wouldn't rely on.
And I don'...
answered Sep 10 '15 at 19:05
9
Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/
Google crawled stuff like
<script>var test="www.example.net/page.html";</script>
or simply
www.example.net/page.html
So i think they only look for a ...
9
Yes, Google still crawl webpages that have noindex tag.
But if you have same content on two different webpages and one URL contain noindex tag, while second does not, then you should not worry about it, because out of all duplicate content only one webpage is indexed by Google. Rest of webpages are crawlable but not indexed in Google search result, so that ...
8
I have just successfully blocked the Chinese searchbot Baiduspider from accessing any content on my site. I made the decision to do so because of the following reasons.
Reasons for deciding to block
Approximately every 20th request to my server was from a baidu bot. This is unpolite behavior. Baidubot accounts for 5% of my sites bandwidth usage.
I make a ...
Only top voted, non community-wiki answers of a minimum length are eligible