A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

5
votes
1answer
419 views

What is the notified-CompliancePage page?

I sometimes have bots (I assume) hitting pages such as: /notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw== ...
6
votes
1answer
5k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
0
votes
1answer
248 views

Link error checking tool for linux [duplicate]

I use Xenu Kubj Sleuth to find out 404s and server errors on websites, but I'm not enjoying it, is there a better free tool out there that can preferably run in bash in linux that does the same thing ...
5
votes
3answers
480 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
2
votes
2answers
1k views

404 errors showing in Google Webmaster Tools

My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found). But when I look at the source pages that are accused of ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
0
votes
0answers
56 views

Are there any good reasons to HTML5 for SEO at the moment? [duplicate]

Possible Duplicate: What are the SEO rules for HTML 5? Do crawlers ignore HTML5 tags (like <article>, <section>, <video>, <nav>, and so on...) when indexing the ...
2
votes
2answers
274 views

Tool to generate list of redirections (sitemap for .htaccess)

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site? I'm building new version of some site, but I don't want to get 404 errors for the old pages. ...
2
votes
1answer
2k views

how to make sure an individual website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
2
votes
2answers
142 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
1
vote
1answer
326 views

Web crawler tool that generate a list of all the anchors used on a site

I'm looking for a web crawler tool that'll go through all the anchor text on a whole web site and generate a list for me to check. I want to make sure all the anchors are really descriptive outside ...
0
votes
2answers
114 views

404s on password protected content

I'm new to WordPress and SEO, generally, but we've been running into problems with our site that don't seem to make sense to me. The problem is that our editor likes to schedule posts and/or mark ...
4
votes
1answer
465 views

Is there any reason to allow Yahoo! Slurp to crawl my site?

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
5
votes
5answers
206 views

What measures can be taken to make sure Google is aware of the existence of a newly created page?

Consider a website with a large number of pages. New pages are published regularly. When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
-1
votes
1answer
162 views

Why my web site was visited by ARPA?

20 minutes ago a user-agent with 66.116.153.122 IP address has visited my web site. It's domain is rev.opentransfer.com.122.153.116.66.in-addr.arpa. User-agent: Mozilla/5.0 (Windows; U; Windows NT ...
5
votes
2answers
304 views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
1
vote
1answer
93 views

In User Agent strings, what does “AskTbTRL” denote?

Here are a bunch of examples: http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
2
votes
1answer
82 views

Does a page blocked for search engines get indexed after link share (+1)

I do my website development always on a subdomain, and that subdomain is blocked for searchengines. I do not want my lorem ipsum content and development domain name indexed by search engines. Let's ...
5
votes
2answers
532 views

How is crawler seeing unlinked directories / files?

I'm running a crawler on my website to test for broken links and such. It starts by using a URL like www.domain.com One curious thing is that it is showing directories with no internal links. For ...
6
votes
2answers
294 views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
1
vote
2answers
353 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
1
vote
1answer
599 views

Why are nonexistant files listed in the TRANSFER log?

This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose. The most puzzling thing to me is that the GET address doesn't correspond ...
3
votes
2answers
238 views

How can I archive (for historical record) my website?

If I launch a website that displays 1000's of pages of content (text) delivered dynamically from data in a database and and I want some sort of record to prove that this data had been published (on ...
3
votes
1answer
392 views

How To Slow Down A Generic Bot?

There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a well-known search-engine, I'd rather slow it ...
3
votes
1answer
836 views

How To Track Down and Stop Rogue Bots?

Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed 164 GB this month. By comparison, Googlebot ...
5
votes
1answer
149 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
2
votes
2answers
9k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
4
votes
1answer
338 views

https & ajax crawling

We made on our webpage https://www.1point618.com a transition to ssl and now we using nearly entirely ajax to load the content. Therefore all urls of existing pages have changed. We used the 301 ...
8
votes
3answers
558 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
0
votes
1answer
43 views

Google spidering: make content available but unsuable

I'm trying to figure out how to make it so that my member-based social network site can be spidered but is not possible for guests to post or interact with content in any way. What is the best way to ...
1
vote
1answer
513 views

Robots.txt practices with .htaccess redirections (inherits)

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
84 views

Do these IP addresses belong to robots?

When i check the traffic on my website, the top IP's have have statistics like this. > IP Page Hits > 91.212.x.x 7123 7123 > 91.x.x.x 249 249 These ...
1
vote
2answers
68 views

SEO: How to stop certain web-pages from “having importance” [duplicate]

Possible Duplicate: SEO - Reducing significance of irrelevant kewords. I wrote "having importance" since I do not really know the right way to say this. I have certain pages in my site ...
9
votes
4answers
2k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
2
votes
3answers
69 views

Anyone use Spiders apart from search engines and spammers?

Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would they ever use them? Thanks
5
votes
2answers
148 views

Extremely large spike in traffic on the 1st - 4th of every month from mobile browsers

I've noticed that on the 1st - 4th of the recent months (since January), several sites I maintain are getting thousands of requests from mobile browsers, whereas throughout the rest of the month, the ...
2
votes
2answers
252 views

Why do people crawl sites without downloading pictures?

Let me show you what I mean: IP Pages Hits Bandwidth 85.xx.xx.xxx 236 236 735.00 KB 195.xx.xxx.xx 164 164 533.74 KB 95.xxx.xxx.xxx 90 90 293.47 ...
5
votes
3answers
710 views

Should I use nofollow on internal links that don't need to be crawled?

We have a product navigation engine that allow our customers to filter products based on attributes (size, color, etc.). We are implementing an upgrade that allows them to use "multi-select" ...
3
votes
1answer
844 views

Is it possible to block an entire site to Google then list exceptions?

I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
2
votes
2answers
48 views

What is this robots.txt disallowing?

Robots.txt – It is currently disallowing “/” – GoogleBot image. Just curious what that is disallowing and if it’s important for search engines to index? You don’t want to restrict any search robots ...
1
vote
1answer
75 views

Does a Registrar forwarding your Domain to your hosting site url, violate Google AdSense TOS?

NameCheap offers to forward traffic to a url of your choosing, which is really handy if you are using a free web hosting service, and/or ever want to move to try another free host someday. However, I ...
5
votes
1answer
595 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
1
vote
1answer
853 views

MSNBot/BingBot not reporting it is a bot?

I have apache logs from my server, and I filter out visits from bots/crawlers/scrapers using a python script that checks for user agent strings containing text like 'bot', 'googlebot', etc. Lately, ...
5
votes
3answers
712 views

Disadvantages of a fake phpMyAdmin honeypot that causes ip blacklisting and robots.txt disallow/exclusion of the honeypot?

I'm trying to figure out whether I should set up a honeypot system with a fake phpMyAdmin (site gets hits all the time with people spidering for insecurities with that app). My thought was to create ...
2
votes
1answer
148 views

How can I let robots access my site?

I have a webpage on my university's server, which means I cannot access the root directory and change files there. Is there some other way to give access to robots to my page? Thanks.
4
votes
1answer
417 views

Does htpasswd authentication prevent Google from crawling my site?

I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing. I currently ...
0
votes
2answers
73 views

Pre-launch Disallow in robots.txt?

My team is launching a new site soon. Is it worth blocking spiders from indexing the landing page right now? What effects will this have on my long term SEO results. Is this question even ...
3
votes
2answers
652 views

How to correctly index multilanguage home page while having a default portal language

Hi there this is my first post, so first of all, sorry for being boring or for any mistakes. I have read many posts on how to correctly index a multi-language site but none of them really answer my ...
1
vote
3answers
141 views

Pseudo-external links and SEO

I don't know, how to name such a links. Main idea is following: I write a post at a community, which I do not control, and place a link to my web-site: www.mywebsite.com When post is saved, my ...
3
votes
5answers
453 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...