Questions tagged [web-crawlers]
A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)
809
questions
37
votes
8answers
27k views
Preventing robots from crawling specific part of a page
As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
31
votes
2answers
8k views
Is there a difference between an empty robots.txt and no robots.txt at all?
On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all.
Do crawlers behave differently in ...
29
votes
1answer
5k views
Bingbot request for trafficbasedsspsitemap.xml which does not exist
The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are
Path: /trafficbasedsspsitemap.xml
Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; +...
26
votes
4answers
1k views
Is Yahoo Search the same as Bing Search now?
We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
25
votes
5answers
5k views
Does the Google spider render JavaScript?
Does the Google spider knows how to render JavaScript, or only HTML?
22
votes
6answers
4k views
Good tool to crawl my site and help me find dead link and unlinked files [closed]
I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
22
votes
3answers
11k views
SEO - Responsive Website and Duplicated Menus
Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
19
votes
3answers
17k views
What user agent should I set?
There is Ask bot, which sets this header:
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Considering this, I have the following questions:
If I'm writing a web crawler named Goofy, what user agent ...
19
votes
3answers
5k views
Is it possible to slow the Baiduspider crawl frequency?
Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy."
I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
18
votes
3answers
11k views
Are web crawlers able to find a secondary robots.txt in a sub-directory?
I have a sub-directory that I would like to hide from web crawlers.
One way to do this is to use a robots.txt in the root directory of the server and add a rule to exclude that sub-directory. However,...
16
votes
7answers
31k views
How to block baidu spiders
Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
15
votes
4answers
3k views
Prevent malicious bots from posting spam
I remember a site closed due to misuse and I wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
15
votes
3answers
3k views
How does Google find a domain with no links to it?
I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else.
I just discovered tonight that the page is already indexed in ...
13
votes
4answers
4k views
Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?
I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
13
votes
4answers
532 views
Status of Crawlable Ajax?
I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang).
http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html
...
12
votes
6answers
15k views
What is the best way to exclude bots from view counts?
My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
12
votes
6answers
17k views
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
...
12
votes
4answers
9k views
Google doesn't crawl CDN files
I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website.
Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
12
votes
2answers
5k views
Should I block bots from my site and why?
My logs are full of bot visitors, often from Eastern Europe and China.
The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on.
Should I block these bots from my site and ...
12
votes
1answer
19k views
Why does baidu crawl my site like crazy
When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks.
It's not that i care much about it but I'm really curious about why he's doing ...
11
votes
2answers
1k views
What bots are really worth letting onto a site?
Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site?
My first ...
11
votes
1answer
1k views
Combine user-agents in robots.txt
Can user-agents be listed together, followed by their common rules like so in a robots.txt?
User-agent: Googlebot
User-agent: ia_archiver #Alexa
User-agent: BingPreview #Microsoft
User-agent: bingbot ...
11
votes
2answers
10k views
Google Webmaster Tools tells me that robots is blocking access to the sitemap
This is my robots.txt:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Sitemap: http://www.example.org/sitemap.xml.gz
But Google Webmaster Tools tells me that robots is blocking access ...
11
votes
4answers
11k views
Adsense click bot is click bombing my site
I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then de-...
10
votes
2answers
18k views
Only allow Google and Bing bots to crawl a site
I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site.
User-...
10
votes
2answers
1k views
Plus sign in front of URLs in user agents
I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format:
examplebot/1.2 (+http://www.example.com/bot....
10
votes
1answer
3k views
Googlebot submitting thousands of requests to our map locater and using up the API quota
We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map.
Recently we had begun to notice that the site was ...
10
votes
8answers
9k views
How long did it take for your new website to be indexed by search engines [duplicate]
Possible Duplicate:
Why isn't my website in Google search results?
I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
10
votes
1answer
1k views
How does “Noindex:” in robots.txt work?
I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.
Disallow: /page-one.html
...
asked Sep 10 '15 at 13:17
10
votes
2answers
2k views
Should I publish an empty ads.txt file for a site that does not run any ads?
I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
10
votes
1answer
734 views
How did Google manage to crawl my 403 pages?
I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
9
votes
2answers
568 views
Should we drop AJAX crawling scheme?
So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9
votes
2answers
4k views
Ask.com sitemap crawler down (for good)?
I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
9
votes
3answers
889 views
Tactics for dealing with misbehaving robots
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.
Obviously we've had ...
9
votes
2answers
322 views
Disqus thread migration. Gotchas?
I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads.
I've had partial success --...
8
votes
3answers
4k views
Screen resolution of Googlebot mobile?
Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it?
It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
8
votes
5answers
4k views
Deny access to Archive.is
I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent).
Do you know if it is possible?
8
votes
2answers
2k views
Is it possible for web crawlers to see static pages without following a link to them?
If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
8
votes
2answers
3k views
Facebook crawler with no user agent spamming our site in possible DoS attack
Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and ...
8
votes
2answers
1k views
Which meta “robots” tag gets preference?
My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
8
votes
1answer
1k views
Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files
We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
7
votes
7answers
4k views
Will banning bots make a site harder to find on search engines?
I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing.
I want to parse the logs and get accurate statistics as to how many ...
7
votes
2answers
1k views
Do search engines still crawl a noindex page
Do search engines crawl a page that has a 'noindex' meta attribute?
The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex'...
7
votes
2answers
1k views
@font-face, google fonts and SEO
Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
7
votes
2answers
17k views
How to identify if IP address is really google's IP
From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with 74....
7
votes
1answer
2k views
Does htpasswd authentication prevent Google from crawling my site?
I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing.
I currently ...
7
votes
1answer
928 views
Does Google penalize sites for showing a different menu for mobile compared to desktop?
I'm really concerned about a design I have been handed by a designer. It changes the menu for mobile and desktop. On mobile you can only access the main categories, that will lead to all the content ...
7
votes
7answers
5k views
Is there an index of the IP addresses used by indexing bots?
I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list.
Is ...
7
votes
3answers
2k views
Should I disallow crawling of HTTP after moving to HTTPS?
I am migrating my well indexed website from HTTP to HTTPS only.
Steps planned:
All HTML pages canonical to be HTTPS now
All HTTP pages will 301 redirect to HTTPS with the same URL
All sitemaps will ...
7
votes
1answer
9k views
What happens if I try to set a cookie on a bot?
I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record (...
