Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

28
votes
1answer
5k views

Bingbot request for trafficbasedsspsitemap.xml which does not exist

The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are Path: /trafficbasedsspsitemap.xml Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; +...
27
votes
8answers
20k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
25
votes
4answers
1k views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
24
votes
2answers
6k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
22
votes
5answers
5k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
21
votes
6answers
4k views

Good tool to crawl my site and help me find dead link and unlinked files [closed]

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
18
votes
3answers
15k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
17
votes
3answers
5k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
15
votes
3answers
2k views

Prevent malicious bots from posting spam

I remember a site closed due to misuse and I wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
14
votes
7answers
27k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
14
votes
2answers
7k views

Can robots.txt be in a server's sub-directory?

I have a sub-directory that I would like to hide from the search engine web crawlers. One way to do this is to use a robots.txt in the root directory of the server (the standard way). However, anyone ...
13
votes
2answers
8k views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
12
votes
3answers
3k views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
12
votes
1answer
19k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
12
votes
4answers
512 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...
11
votes
6answers
13k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
11
votes
2answers
988 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
11
votes
8answers
9k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
11
votes
4answers
7k views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
11
votes
2answers
4k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
11
votes
2answers
9k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.example.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
11
votes
4answers
9k views

Adsense click bot is click bombing my site

I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then de-...
10
votes
2answers
15k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. User-...
10
votes
3answers
3k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
10
votes
1answer
3k views

Googlebot submitting thousands of requests to our map locater and using up the API quota

We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map. Recently we had begun to notice that the site was ...
10
votes
1answer
968 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
10
votes
1answer
710 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
9
votes
2answers
516 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9
votes
2answers
4k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
9
votes
3answers
830 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
9
votes
1answer
983 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
9
votes
2answers
298 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success --...
8
votes
5answers
3k views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
8
votes
2answers
1k views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
8
votes
2answers
1k views

Which meta “robots” tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
8
votes
2answers
886 views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 (+http://www.example.com/bot....
8
votes
2answers
105 views

Does Google's webcrawler follow HTML imports?

Does googlebot - Google's webcrawler - follow browser-side HTML imports such as the following? <link rel="import" href="header.html"> I'm thinking of hosting web content on my web server that ...
7
votes
7answers
4k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
7
votes
5answers
12k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
7
votes
2answers
772 views

Do search engines still crawl a noindex page

Do search engines crawl a page that has a 'noindex' meta attribute? The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex'...
7
votes
2answers
980 views

@font-face, google fonts and SEO

Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
7
votes
1answer
722 views

Does Google penalize sites for showing a different menu for mobile compared to desktop?

I'm really concerned about a design I have been handed by a designer. It changes the menu for mobile and desktop. On mobile you can only access the main categories, that will lead to all the content ...
7
votes
7answers
4k views

Is there an index of the IP addresses used by indexing bots?

I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list. Is ...
7
votes
2answers
807 views

Facebook crawler with no user agent spamming our site in possible DoS attack

Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and ...
7
votes
3answers
1k views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
6
votes
2answers
4k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
6
votes
1answer
246 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
6
votes
2answers
13k views

How to identify if IP address is really google's IP

From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with 74....
6
votes
1answer
711 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
6
votes
1answer
346 views

Will Google and other search engines follow IP address only sites (i.e., no URL)

Will Google and other search engines follow links to and index sites that don't have a URL, but instead just use their IP address's (obviously they would need a unique IP) ?