A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

5
votes
5answers
235 views

What measures can be taken to make sure Google is aware of the existence of a newly created page?

Consider a website with a large number of pages. New pages are published regularly. When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
0
votes
1answer
211 views

Why my web site was visited by ARPA?

20 minutes ago a user-agent with 66.116.153.122 IP address has visited my web site. It's domain is rev.opentransfer.com.122.153.116.66.in-addr.arpa. User-agent: Mozilla/5.0 (Windows; U; Windows NT ...
10
votes
3answers
1k views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
1
vote
1answer
99 views

In User Agent strings, what does “AskTbTRL” denote?

Here are a bunch of examples: http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
2
votes
1answer
84 views

Does a page blocked for search engines get indexed after link share (+1)

I do my website development always on a subdomain, and that subdomain is blocked for searchengines. I do not want my lorem ipsum content and development domain name indexed by search engines. Let's ...
5
votes
2answers
906 views

How is crawler seeing unlinked directories / files?

I'm running a crawler on my website to test for broken links and such. It starts by using a URL like www.domain.com One curious thing is that it is showing directories with no internal links. For ...
7
votes
2answers
464 views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
1
vote
2answers
499 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
2
votes
1answer
722 views

Why are nonexistant files listed in the TRANSFER log?

This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose. The most puzzling thing to me is that the GET address doesn't correspond ...
3
votes
2answers
528 views

How can I archive (for historical record) my website?

If I launch a website that displays 1000's of pages of content (text) delivered dynamically from data in a database and and I want some sort of record to prove that this data had been published (on ...
3
votes
1answer
563 views

How To Slow Down A Generic Bot?

There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a well-known search-engine, I'd rather slow it ...
3
votes
2answers
2k views

How To Track Down and Stop Rogue Bots?

Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed 164 GB this month. By comparison, Googlebot ...
5
votes
1answer
215 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
3
votes
2answers
15k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
5
votes
1answer
369 views

Does Google crawl and index AJAX on HTTPS sites?

We made transitioned out website to HTTPS. We also now use nearly AJAX to load almost all the content. Therefore all URLs of existing pages have changed. We used 301 redirects as recommended and we ...
8
votes
3answers
721 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
0
votes
1answer
47 views

Google spidering: make content available but unsuable

I'm trying to figure out how to make it so that my member-based social network site can be spidered but is not possible for guests to post or interact with content in any way. What is the best way to ...
1
vote
1answer
597 views

Robots.txt practices with .htaccess redirections (inherits)

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
91 views

Do these IP addresses belong to robots?

When i check the traffic on my website, the top IP's have have statistics like this. > IP Page Hits > 91.212.x.x 7123 7123 > 91.x.x.x 249 249 These ...
1
vote
2answers
76 views

SEO: How to stop certain web-pages from “having importance” [duplicate]

Possible Duplicate: SEO - Reducing significance of irrelevant kewords. I wrote "having importance" since I do not really know the right way to say this. I have certain pages in my site ...
16
votes
6answers
8k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
2
votes
3answers
72 views

Anyone use Spiders apart from search engines and spammers?

Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would they ever use them? Thanks
5
votes
2answers
150 views

Extremely large spike in traffic on the 1st - 4th of every month from mobile browsers

I've noticed that on the 1st - 4th of the recent months (since January), several sites I maintain are getting thousands of requests from mobile browsers, whereas throughout the rest of the month, the ...
2
votes
2answers
288 views

Why do people crawl sites without downloading pictures?

Let me show you what I mean: IP Pages Hits Bandwidth 85.xx.xx.xxx 236 236 735.00 KB 195.xx.xxx.xx 164 164 533.74 KB 95.xxx.xxx.xxx 90 90 293.47 ...
5
votes
3answers
806 views

Should I use nofollow on internal links that don't need to be crawled?

We have a product navigation engine that allow our customers to filter products based on attributes (size, color, etc.). We are implementing an upgrade that allows them to use "multi-select" ...
4
votes
1answer
1k views

Is it possible to block an entire site to Google then list exceptions?

I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
2
votes
2answers
62 views

What is this robots.txt disallowing?

Robots.txt – It is currently disallowing “/” – GoogleBot image. Just curious what that is disallowing and if it’s important for search engines to index? You don’t want to restrict any search robots ...
1
vote
1answer
91 views

Does a Registrar forwarding your Domain to your hosting site url, violate Google AdSense TOS?

NameCheap offers to forward traffic to a url of your choosing, which is really handy if you are using a free web hosting service, and/or ever want to move to try another free host someday. However, I ...
5
votes
1answer
696 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
1
vote
1answer
1k views

MSNBot/BingBot not reporting it is a bot?

I have apache logs from my server, and I filter out visits from bots/crawlers/scrapers using a python script that checks for user agent strings containing text like 'bot', 'googlebot', etc. Lately, ...
5
votes
3answers
1k views

Disadvantages of a fake phpMyAdmin honeypot that causes ip blacklisting and robots.txt disallow/exclusion of the honeypot?

I'm trying to figure out whether I should set up a honeypot system with a fake phpMyAdmin (site gets hits all the time with people spidering for insecurities with that app). My thought was to create ...
2
votes
1answer
157 views

How can I let robots access my site?

I have a webpage on my university's server, which means I cannot access the root directory and change files there. Is there some other way to give access to robots to my page? Thanks.
5
votes
1answer
851 views

Does htpasswd authentication prevent Google from crawling my site?

I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing. I currently ...
0
votes
2answers
98 views

Pre-launch Disallow in robots.txt?

My team is launching a new site soon. Is it worth blocking spiders from indexing the landing page right now? What effects will this have on my long term SEO results. Is this question even ...
3
votes
2answers
1k views

How to correctly index multilanguage home page while having a default portal language

Hi there this is my first post, so first of all, sorry for being boring or for any mistakes. I have read many posts on how to correctly index a multi-language site but none of them really answer my ...
2
votes
3answers
162 views

Pseudo-external links and SEO

I don't know, how to name such a links. Main idea is following: I write a post at a community, which I do not control, and place a link to my web-site: www.mywebsite.com When post is saved, my ...
3
votes
5answers
505 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
2
votes
2answers
176 views

Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?

My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
8
votes
6answers
8k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
3
votes
2answers
81 views

How to set crawl rate in Google Webmaster Tools for seven days or a month?

My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...
2
votes
1answer
525 views

Do all search engines obey nofollow rel attribute on hyperlinks

I have been informed that search engine support is inconsistent for the rel attribute with the 'nofollow' value. Is this the case. From my research Google, Yahoo and Bing seem to obey this attribute ...
2
votes
2answers
800 views

All the IPs of search engine bots?

if I want to filter out all the search engine bots for example for a script that counts users that are online is there website or a list which is allways up to date and includes the ip addresses of ...
1
vote
2answers
798 views

SEO: disallowing Google from indexing forms?

From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not? I mean my forms hardly ever contains keyword/keyphrases. So I'm wondering what's the point of letting ...
1
vote
2answers
55 views

Crawling web to build a directory of a niche industry for SEO

I heard that some web sites crawl web to build a directory of companies in a niche domain. So, when people search for the name of the company they find that directory page. I wonder if this is ...
2
votes
1answer
147 views

What is user agent Issbot

I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
3
votes
3answers
361 views

Is my robots.txt working as it should?

I want crawlers to have access to http://www.example.com but not http://www.example.com/ My robots.txt is as follows: User-agent: * Allow: /$ Disallow: / My site is in Google search results, but I ...
0
votes
2answers
132 views

Can internal linking create excessive web server load?

SE sites allow tags chaining like this: /questions/tagged/seo+google+search-engines+research+keywords This site has more than 700 tags. If up to 5 tags can be chained, there are ...
1
vote
2answers
115 views

SEO - Crawl Errors

I've been optimizing my website for search engines, and for some reason I have a crawl error on a page that has not been found at 'www.peach-designs.com/a' which is not a page, is not a link, or is ...
1
vote
1answer
609 views

How does bing-bot( is that the right spider-name? ) and googlebot interpret 301 redirect?

I've been looking for documentation on how the Microsoft and Google bots interpret 301 redirects. It seems that google-bot stores documents on a url based index system. But I haven't been able to ...
1
vote
2answers
206 views

Adult website crawlability issue

I got a mission and asking for some help with this scenario for a new website. The site is an adult escorts advertising service. It is developed fine, works and looks good, but the issue is with ...