A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
3answers
109 views

Free standalone crawlers

Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors? Something that shows me all 404's, errors etc
4
votes
1answer
227 views

Using rel=canonical and noindex in a 1-n partners environment

We sell complete sites including the domains to several partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original but the ...
0
votes
1answer
568 views

Apache real time robot detection

Background Our website is being a. scraped - complete pages downloaded b. spidered - complete pages and sub links followed c. mined - specific files downloaded at high speed by hundreds of ...
1
vote
1answer
265 views

Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm ...
0
votes
1answer
79 views

How do I make the home page of the website to come up in the rankings than the internal pages? [duplicate]

Possible Duplicate: What are the best ways to increase your site's position in Google? Suppose I have a website, e.g. www.example.com that comes at number 6 on the Google search ...
0
votes
0answers
107 views

Disallow age gate/verification page in robots.txt?

Should I disallow the age gate page in robots.txt? I need to make sure the crawlers can get through the gate.
3
votes
2answers
187 views

Can Google tell the difference between a link in a blog post and a link in a blog comment?

When Google Blog Search crawls a page, can it tell whether a link is in the article or whether it's in a comment? If it can, does it grade the links, that is, a link from a blog comment is only going ...
1
vote
2answers
1k views

Does searchbots index content in iframe [duplicate]

Possible Duplicate: Do we still need to avoid using frame/iframe for good SEO? I have a html page that has a map of a country with lots of dots indicating location of companies through out ...
3
votes
1answer
867 views

Why is BingBot causing so many 404 errors by removing letters from url's?

Since the ForceRecrawl problem went away Bing has come back with some new trends. I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
1
vote
1answer
214 views

Search engines crawling non-existent subdomains

A few years ago, a previous developer added my company's website to a link farm he ran. My company has a wildcard subdomain, so any subdomains will return the user to the homepage. As such, the dev ...
5
votes
1answer
219 views

What is User-Agent “AF_ID=<xxxxxx>”

We periodically (more frequently recently) have very agressive crawling activity coming from EC2 instances that give us a user agent that looks like "AF_ID=". I've looked around for common ...
7
votes
2answers
667 views

@font-face, google fonts and SEO

Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
1
vote
1answer
564 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
1
vote
4answers
116 views

Is it bad for SEO to have a site with no click navigation?

I have a site where users can only get to most of the sites content through search. If I make a sitemap that has the URLs for all of the potential searches is it ok that these pages cannot be click ...
1
vote
1answer
540 views

Question about robots.txt Google shows urls are blocked

I am working on this site dealsin.us and the Google webmasters tool shows that there are about 9800+ urls blocked by robots.txt. You can view the robots.txt here I have blocked some directories which ...
4
votes
2answers
634 views

How often are sitemap.xml checked for updates by crawlers?

How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
4
votes
2answers
529 views

Understanding Ajax crawling of search site

I have a couple of questions about Ajax crawling of a website, which is kind of search engine itself. The base article explains the mechanism of making AJAX applications crawlable. All this stuff ...
2
votes
1answer
183 views

Will google “forget” unlinked pages?

If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...
2
votes
1answer
267 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
0
votes
1answer
1k views

BingPreview user agent

I'm getting several invalid requests from a "BingPreview" user agent. The complete user agent header is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b The ...
3
votes
1answer
369 views

How to interpret number of URL errors in Google webmaster tools

Recently Google has made some changes to Webmaster tools which are explained below: http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html One thing I could not find out ...
5
votes
1answer
460 views

What is the notified-CompliancePage page?

I sometimes have bots (I assume) hitting pages such as: /notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw== ...
6
votes
1answer
6k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
0
votes
1answer
251 views

Link error checking tool for linux [duplicate]

I use Xenu Kubj Sleuth to find out 404s and server errors on websites, but I'm not enjoying it, is there a better free tool out there that can preferably run in bash in linux that does the same thing ...
5
votes
3answers
509 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
2
votes
2answers
1k views

404 errors showing in Google Webmaster Tools

My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found). But when I look at the source pages that are accused of ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
0
votes
0answers
56 views

Are there any good reasons to HTML5 for SEO at the moment? [duplicate]

Possible Duplicate: What are the SEO rules for HTML 5? Do crawlers ignore HTML5 tags (like <article>, <section>, <video>, <nav>, and so on...) when indexing the ...
2
votes
2answers
283 views

Tool to generate list of redirections (sitemap for .htaccess)

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site? I'm building new version of some site, but I don't want to get 404 errors for the old pages. ...
2
votes
1answer
2k views

how to make sure an individual website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
2
votes
2answers
151 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
1
vote
1answer
334 views

Web crawler tool that generate a list of all the anchors used on a site

I'm looking for a web crawler tool that'll go through all the anchor text on a whole web site and generate a list for me to check. I want to make sure all the anchors are really descriptive outside ...
0
votes
2answers
115 views

404s on password protected content

I'm new to WordPress and SEO, generally, but we've been running into problems with our site that don't seem to make sense to me. The problem is that our editor likes to schedule posts and/or mark ...
4
votes
1answer
570 views

Is there any reason to allow Yahoo! Slurp to crawl my site?

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
5
votes
5answers
219 views

What measures can be taken to make sure Google is aware of the existence of a newly created page?

Consider a website with a large number of pages. New pages are published regularly. When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
-1
votes
1answer
168 views

Why my web site was visited by ARPA?

20 minutes ago a user-agent with 66.116.153.122 IP address has visited my web site. It's domain is rev.opentransfer.com.122.153.116.66.in-addr.arpa. User-agent: Mozilla/5.0 (Windows; U; Windows NT ...
5
votes
2answers
350 views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
1
vote
1answer
94 views

In User Agent strings, what does “AskTbTRL” denote?

Here are a bunch of examples: http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
2
votes
1answer
82 views

Does a page blocked for search engines get indexed after link share (+1)

I do my website development always on a subdomain, and that subdomain is blocked for searchengines. I do not want my lorem ipsum content and development domain name indexed by search engines. Let's ...
5
votes
2answers
595 views

How is crawler seeing unlinked directories / files?

I'm running a crawler on my website to test for broken links and such. It starts by using a URL like www.domain.com One curious thing is that it is showing directories with no internal links. For ...
6
votes
2answers
319 views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
1
vote
2answers
374 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
1
vote
1answer
629 views

Why are nonexistant files listed in the TRANSFER log?

This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose. The most puzzling thing to me is that the GET address doesn't correspond ...
3
votes
2answers
281 views

How can I archive (for historical record) my website?

If I launch a website that displays 1000's of pages of content (text) delivered dynamically from data in a database and and I want some sort of record to prove that this data had been published (on ...
3
votes
1answer
424 views

How To Slow Down A Generic Bot?

There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a well-known search-engine, I'd rather slow it ...
3
votes
1answer
911 views

How To Track Down and Stop Rogue Bots?

Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed 164 GB this month. By comparison, Googlebot ...
5
votes
1answer
153 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
2
votes
2answers
11k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
4
votes
1answer
341 views

https & ajax crawling

We made on our webpage https://www.1point618.com a transition to ssl and now we using nearly entirely ajax to load the content. Therefore all urls of existing pages have changed. We used the 301 ...
8
votes
3answers
590 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...