A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

2
votes
1answer
719 views

Why are nonexistant files listed in the TRANSFER log?

This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose. The most puzzling thing to me is that the GET address doesn't correspond ...
3
votes
2answers
522 views

How can I archive (for historical record) my website?

If I launch a website that displays 1000's of pages of content (text) delivered dynamically from data in a database and and I want some sort of record to prove that this data had been published (on ...
3
votes
1answer
558 views

How To Slow Down A Generic Bot?

There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a well-known search-engine, I'd rather slow it ...
3
votes
2answers
2k views

How To Track Down and Stop Rogue Bots?

Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed 164 GB this month. By comparison, Googlebot ...
5
votes
1answer
215 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
3
votes
2answers
15k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
5
votes
1answer
369 views

Does Google crawl and index AJAX on HTTPS sites?

We made transitioned out website to HTTPS. We also now use nearly AJAX to load almost all the content. Therefore all URLs of existing pages have changed. We used 301 redirects as recommended and we ...
8
votes
3answers
720 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
0
votes
1answer
47 views

Google spidering: make content available but unsuable

I'm trying to figure out how to make it so that my member-based social network site can be spidered but is not possible for guests to post or interact with content in any way. What is the best way to ...
1
vote
1answer
596 views

Robots.txt practices with .htaccess redirections (inherits)

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
91 views

Do these IP addresses belong to robots?

When i check the traffic on my website, the top IP's have have statistics like this. > IP Page Hits > 91.212.x.x 7123 7123 > 91.x.x.x 249 249 These ...
1
vote
2answers
76 views

SEO: How to stop certain web-pages from “having importance” [duplicate]

Possible Duplicate: SEO - Reducing significance of irrelevant kewords. I wrote "having importance" since I do not really know the right way to say this. I have certain pages in my site ...
16
votes
6answers
8k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
2
votes
3answers
72 views

Anyone use Spiders apart from search engines and spammers?

Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would they ever use them? Thanks
5
votes
2answers
150 views

Extremely large spike in traffic on the 1st - 4th of every month from mobile browsers

I've noticed that on the 1st - 4th of the recent months (since January), several sites I maintain are getting thousands of requests from mobile browsers, whereas throughout the rest of the month, the ...
2
votes
2answers
288 views

Why do people crawl sites without downloading pictures?

Let me show you what I mean: IP Pages Hits Bandwidth 85.xx.xx.xxx 236 236 735.00 KB 195.xx.xxx.xx 164 164 533.74 KB 95.xxx.xxx.xxx 90 90 293.47 ...
5
votes
3answers
804 views

Should I use nofollow on internal links that don't need to be crawled?

We have a product navigation engine that allow our customers to filter products based on attributes (size, color, etc.). We are implementing an upgrade that allows them to use "multi-select" ...
4
votes
1answer
1k views

Is it possible to block an entire site to Google then list exceptions?

I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
2
votes
2answers
62 views

What is this robots.txt disallowing?

Robots.txt – It is currently disallowing “/” – GoogleBot image. Just curious what that is disallowing and if it’s important for search engines to index? You don’t want to restrict any search robots ...
1
vote
1answer
91 views

Does a Registrar forwarding your Domain to your hosting site url, violate Google AdSense TOS?

NameCheap offers to forward traffic to a url of your choosing, which is really handy if you are using a free web hosting service, and/or ever want to move to try another free host someday. However, I ...
5
votes
1answer
695 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
1
vote
1answer
1k views

MSNBot/BingBot not reporting it is a bot?

I have apache logs from my server, and I filter out visits from bots/crawlers/scrapers using a python script that checks for user agent strings containing text like 'bot', 'googlebot', etc. Lately, ...
5
votes
3answers
1k views

Disadvantages of a fake phpMyAdmin honeypot that causes ip blacklisting and robots.txt disallow/exclusion of the honeypot?

I'm trying to figure out whether I should set up a honeypot system with a fake phpMyAdmin (site gets hits all the time with people spidering for insecurities with that app). My thought was to create ...
2
votes
1answer
156 views

How can I let robots access my site?

I have a webpage on my university's server, which means I cannot access the root directory and change files there. Is there some other way to give access to robots to my page? Thanks.
5
votes
1answer
835 views

Does htpasswd authentication prevent Google from crawling my site?

I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing. I currently ...
0
votes
2answers
98 views

Pre-launch Disallow in robots.txt?

My team is launching a new site soon. Is it worth blocking spiders from indexing the landing page right now? What effects will this have on my long term SEO results. Is this question even ...
3
votes
2answers
1k views

How to correctly index multilanguage home page while having a default portal language

Hi there this is my first post, so first of all, sorry for being boring or for any mistakes. I have read many posts on how to correctly index a multi-language site but none of them really answer my ...
1
vote
3answers
160 views

Pseudo-external links and SEO

I don't know, how to name such a links. Main idea is following: I write a post at a community, which I do not control, and place a link to my web-site: www.mywebsite.com When post is saved, my ...
3
votes
5answers
505 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
2
votes
2answers
176 views

Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?

My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
8
votes
6answers
8k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
3
votes
2answers
81 views

How to set crawl rate in Google Webmaster Tools for seven days or a month?

My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...
2
votes
1answer
521 views

Do all search engines obey nofollow rel attribute on hyperlinks

I have been informed that search engine support is inconsistent for the rel attribute with the 'nofollow' value. Is this the case. From my research Google, Yahoo and Bing seem to obey this attribute ...
2
votes
2answers
781 views

All the IPs of search engine bots?

if I want to filter out all the search engine bots for example for a script that counts users that are online is there website or a list which is allways up to date and includes the ip addresses of ...
1
vote
2answers
792 views

SEO: disallowing Google from indexing forms?

From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not? I mean my forms hardly ever contains keyword/keyphrases. So I'm wondering what's the point of letting ...
1
vote
2answers
55 views

Crawling web to build a directory of a niche industry for SEO

I heard that some web sites crawl web to build a directory of companies in a niche domain. So, when people search for the name of the company they find that directory page. I wonder if this is ...
1
vote
1answer
145 views

What is user agent Issbot

I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
3
votes
3answers
361 views

Is my robots.txt working as it should?

I want crawlers to have access to http://www.example.com but not http://www.example.com/ My robots.txt is as follows: User-agent: * Allow: /$ Disallow: / My site is in Google search results, but I ...
0
votes
2answers
131 views

Can internal linking create excessive web server load?

SE sites allow tags chaining like this: /questions/tagged/seo+google+search-engines+research+keywords This site has more than 700 tags. If up to 5 tags can be chained, there are ...
1
vote
2answers
115 views

SEO - Crawl Errors

I've been optimizing my website for search engines, and for some reason I have a crawl error on a page that has not been found at 'www.peach-designs.com/a' which is not a page, is not a link, or is ...
1
vote
1answer
609 views

How does bing-bot( is that the right spider-name? ) and googlebot interpret 301 redirect?

I've been looking for documentation on how the Microsoft and Google bots interpret 301 redirects. It seems that google-bot stores documents on a url based index system. But I haven't been able to ...
1
vote
2answers
206 views

Adult website crawlability issue

I got a mission and asking for some help with this scenario for a new website. The site is an adult escorts advertising service. It is developed fine, works and looks good, but the issue is with ...
4
votes
5answers
1k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
2answers
327 views

If an visitors IP address contains “google” or a similar keyword, does this mean they were a crawler?

I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, ...
0
votes
2answers
282 views

Menu hidden links and bots!

I want to make a menu with a list with hidden links and when the user points the menu with he mouse the links will appear, my problem is that the bots may think this as a deceiving tactic and lower my ...
8
votes
0answers
520 views

Is there a search engine that indexes source code of a web-page? [closed]

I need to search the web for sites that are in our industry that use the same Adwords management company, to ensure that the said company is not violating our contract, as they have been accused of ...
2
votes
2answers
3k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
2
votes
1answer
251 views

Should I strategically use NoFollow on internal links to make our indexing crawl more efficient?

We have a well designed internal linking system that makes certain no page is more than four pages deep, and the indexing process has gone well for us. However, in addition to the hierarchical page ...
2
votes
3answers
186 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
10
votes
1answer
679 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...