A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

25
votes
1answer
5k views

Bingbot request for trafficbasedsspsitemap.xml which does not exist

The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are Path: /trafficbasedsspsitemap.xml Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; ...
23
votes
4answers
847 views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
20
votes
6answers
2k views

Good tool to crawl my site and help me find dead link and unlinked files [closed]

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
13
votes
4answers
3k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
12
votes
4answers
452 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...
11
votes
2answers
815 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
9
votes
1answer
602 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
9
votes
4answers
2k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
8
votes
2answers
2k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
8
votes
3answers
590 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
8
votes
2answers
2k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
8
votes
1answer
4k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
8
votes
3answers
1k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
8
votes
4answers
5k views

Adsense click bot is click bombing my site

I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then ...
8
votes
1answer
214 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success ...
8
votes
0answers
478 views

Is there a search engine that indexes source code of a web-page? [closed]

I need to search the web for sites that are in our industry that use the same Adwords management company, to ensure that the said company is not violating our contract, as they have been accused of ...
7
votes
2answers
668 views

@font-face, google fonts and SEO

Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
7
votes
7answers
2k views

Is there an index of the IP addresses used by indexing bots?

I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list. Is ...
7
votes
8answers
4k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
7
votes
1answer
98 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
6
votes
2answers
320 views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
6
votes
1answer
314 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
6
votes
2answers
284 views

Which meta “robots” tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
6
votes
1answer
224 views

Will Google and other search engines follow IP address only sites (i.e., no URL)

Will Google and other search engines follow links to and index sites that don't have a URL, but instead just use their IP address's (obviously they would need a unique IP) ?
6
votes
1answer
192 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
6
votes
1answer
6k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
6
votes
1answer
243 views

List of all URI elements for a website

Does anyone know a good way of getting a list of all the URI elements for a website? I plan on moving a website to a new CMS and would like to setup some 301 redirects for articles, images, css, and ...
6
votes
2answers
219 views

Does the adsense bot ever get bored?

Each time someone clicks on a page like http://www.example.com/crazy.cgi?input=silly+banana where the "silly banana" bit represents some arbitrary user input which changes each time, I get yet ...
5
votes
5answers
1k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
5
votes
3answers
2k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
5
votes
2answers
183 views

Tell Google the list of URLs to crawl

Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find ...
5
votes
3answers
509 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
5
votes
1answer
153 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
5
votes
5answers
219 views

What measures can be taken to make sure Google is aware of the existence of a newly created page?

Consider a website with a large number of pages. New pages are published regularly. When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
5
votes
5answers
780 views

Which tools can help limit maximum page view per IP to limit scrapers and bots?

I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
5
votes
3answers
755 views

Disadvantages of a fake phpMyAdmin honeypot that causes ip blacklisting and robots.txt disallow/exclusion of the honeypot?

I'm trying to figure out whether I should set up a honeypot system with a fake phpMyAdmin (site gets hits all the time with people spidering for insecurities with that app). My thought was to create ...
5
votes
1answer
83 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
5
votes
2answers
599 views

How is crawler seeing unlinked directories / files?

I'm running a crawler on my website to test for broken links and such. It starts by using a URL like www.domain.com One curious thing is that it is showing directories with no internal links. For ...
5
votes
3answers
739 views

Should I use nofollow on internal links that don't need to be crawled?

We have a product navigation engine that allow our customers to filter products based on attributes (size, color, etc.). We are implementing an upgrade that allows them to use "multi-select" ...
5
votes
2answers
352 views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
5
votes
2answers
148 views

Extremely large spike in traffic on the 1st - 4th of every month from mobile browsers

I've noticed that on the 1st - 4th of the recent months (since January), several sites I maintain are getting thousands of requests from mobile browsers, whereas throughout the rest of the month, the ...
5
votes
1answer
460 views

What is the notified-CompliancePage page?

I sometimes have bots (I assume) hitting pages such as: /notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw== ...
5
votes
1answer
618 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
5
votes
1answer
233 views

How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache

The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
5
votes
1answer
220 views

What is User-Agent “AF_ID=<xxxxxx>”

We periodically (more frequently recently) have very agressive crawling activity coming from EC2 instances that give us a user agent that looks like "AF_ID=". I've looked around for common ...
5
votes
3answers
114 views

Does meta name=robots content=nofollow prevent indexing iframes?

Let's say I have a publicly accessible page that contains an iframe back to a different page on my own domain. By default, google (and other crawlers) will index that page and the iframe's content. It ...
5
votes
4answers
285 views

Total Indexed 0, 99 not selected in webmaster tools

I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
4
votes
2answers
107 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
4
votes
2answers
266 views

Why doesn't this web crawler obey requests to stop crawling my pages?

I have a hidden page that verifies internal payments from PayPal. The page has the following in its HTTP headers: X-Robots-Tag: noindex, nofollow But once in a while I see this in the page logs: ...
4
votes
3answers
203 views

Do i need to have a separate page for each article on my website?

I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google? Is is ...