Skip to main content

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
42 votes
9 answers
35k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
WebbyTheWebbor's user avatar
34 votes
3 answers
10k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
Foo Bar's user avatar
  • 453
29 votes
1 answer
6k views

Bingbot request for trafficbasedsspsitemap.xml which does not exist

The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are Path: /trafficbasedsspsitemap.xml Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; +...
Spongeboy's user avatar
  • 393
26 votes
5 answers
6k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
user avatar
26 votes
4 answers
1k views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
Jeff Atwood's user avatar
25 votes
3 answers
14k views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
Howdy_McGee's user avatar
21 votes
3 answers
19k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
Nicu Surdu's user avatar
21 votes
6 answers
5k views

Good tool to crawl my site and help me find dead link and unlinked files [closed]

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
MrChrister's user avatar
19 votes
3 answers
5k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
samthebrand's user avatar
18 votes
3 answers
12k views

Are web crawlers able to find a secondary robots.txt in a sub-directory?

I have a sub-directory that I would like to hide from web crawlers. One way to do this is to use a robots.txt in the root directory of the server and add a rule to exclude that sub-directory. However,...
user avatar
17 votes
3 answers
4k views

Should I publish an empty ads.txt file for a site that does not run any ads?

I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
Louis Somers's user avatar
16 votes
7 answers
33k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
Zhianc's user avatar
  • 375
16 votes
4 answers
5k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
Cesar's user avatar
  • 405
15 votes
4 answers
3k views

Prevent malicious bots from posting spam

I remember a site closed due to misuse and I wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
user avatar
15 votes
3 answers
3k views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
DisgruntledGoat's user avatar
14 votes
8 answers
5k views

Is it possible to keep the existence of a domain secret?

If I bought a domain like a9j47fn83jd8j49.tld, and only a friend and I ever visit it, who would know about it? It sounds like for most TLDs, there is some reporting that has to happen after you buy a ...
dcc310's user avatar
  • 243
14 votes
6 answers
24k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
the's user avatar
  • 1,580
14 votes
2 answers
2k views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 (+http://www.example.com/bot....
jlh's user avatar
  • 241
13 votes
4 answers
559 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...
philfreo's user avatar
  • 271
12 votes
6 answers
16k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
eisberg's user avatar
  • 123
12 votes
2 answers
20k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. User-...
Konsole's user avatar
  • 223
12 votes
5 answers
6k views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
someone2332's user avatar
12 votes
4 answers
10k views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
tonelot's user avatar
  • 191
12 votes
2 answers
5k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
Frank E's user avatar
  • 151
12 votes
4 answers
12k views

Adsense click bot is click bombing my site

I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then de-...
Graham's user avatar
  • 411
12 votes
1 answer
19k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
yokoloko's user avatar
  • 405
11 votes
4 answers
6k views

Prevent Google from thinking my own bot is faking traffic on my website

I have a large website which I want to be visited by my own bot early on the morning, every day (for caching purpose). I want my bot visits 10,000 pages per day. My natural traffic is about 5000 pages ...
ordiminnie's user avatar
11 votes
2 answers
1k views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
blunders's user avatar
  • 2,080
11 votes
2 answers
4k views

What determines the frequency the Wayback Machine crawls one's website?

I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
Manu H's user avatar
  • 213
11 votes
1 answer
2k views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
davidcondrey's user avatar
11 votes
2 answers
10k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.example.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
Gaia's user avatar
  • 345
10 votes
3 answers
7k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
Baumr's user avatar
  • 1,444
10 votes
1 answer
3k views

Googlebot submitting thousands of requests to our map locater and using up the API quota

We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map. Recently we had begun to notice that the site was ...
Burgi's user avatar
  • 287
10 votes
8 answers
9k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
10 votes
1 answer
2k views

How does "Noindex:" in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
Stephen Ostermiller's user avatar
10 votes
1 answer
748 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
grautur's user avatar
  • 201
9 votes
11 answers
24k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
Codebeat's user avatar
  • 211
9 votes
2 answers
598 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
Mario Trucco's user avatar
9 votes
2 answers
4k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
SameOldNick's user avatar
9 votes
2 answers
1k views

Which meta "robots" tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
Tim Pietzcker's user avatar
9 votes
3 answers
1k views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
Kris's user avatar
  • 4,487
9 votes
1 answer
7k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
Yin Yang's user avatar
  • 245
9 votes
2 answers
327 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success --...
sramsay's user avatar
  • 91
8 votes
2 answers
5k views

Facebook crawler with no user agent spamming our site in possible DoS attack

Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and ...
L Martin's user avatar
  • 4,013
8 votes
1 answer
736 views

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says the /fish rule matches any path that starts with /fish (...) it doesn'...
Déjà vu's user avatar
  • 183
8 votes
1 answer
11k views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record (...
Cyberherbalist's user avatar
8 votes
1 answer
4k views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
Vaibhav Jalan's user avatar
8 votes
1 answer
1k views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
John Washam's user avatar
7 votes
7 answers
4k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
Tensigh's user avatar
  • 175
7 votes
5 answers
32k views

What does "Disallow: /search" mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
Sathiya Kumar V M's user avatar

1
2 3 4 5
19