Tagged Questions

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

4
votes
1answer
294 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
2
votes
1answer
723 views

What does “all” mean in meta robots tag?

Several websites have the following meta tag aimed at search engines like Google, Bing, etc.: <meta name="robots" content="all" /> What does it do?
5
votes
5answers
915 views

Which tools can help limit maximum page view per IP to limit scrapers and bots?

I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
3
votes
2answers
266 views

Can the .htaccess file slow down a website to a crawl? If so, are there better ways to solve these problems with different rewrite rules and such?

here is my htaccess file...... RewriteCond %{REQUEST_URI} ^/patients/billing/FAQ_billing\.html$ [OR] RewriteCond %{REQUEST_URI} ^/patients/billing/getintouch\.html$ RewriteRule ...
1
vote
0answers
56 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
2
votes
1answer
197 views

Ajax crawling : snapshot page dissimilar with user page

For ajax-crawling purpose I make snapshots pages. My problem is user view pages and snapshot pages are dissimilar (tags h1, h2...). Example: user view page: http://www.meomap.net/#!/place/1 ...
2
votes
1answer
269 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
6
votes
2answers
320 views

Which meta “robots” tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
0
votes
1answer
20 views

Prevent Advert Images Appearing in Search Spider Crawls

While testing out how web pages on my site appear to search engine spiders using http://www.browseo.net/ I noticed that advert images appear in the view. However, when I test a site like ...
3
votes
2answers
86 views

Am I being hacked or harvested? [closed]

I just did some log analysis of Q1 for one of my client's sites. There were about 30,000 sessions created for this site for each of the five separate IP addresses starting with 79.171.81.xxx. 150,000 ...
0
votes
2answers
113 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
1
vote
5answers
2k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
4
votes
3answers
208 views

Do i need to have a separate page for each article on my website?

I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google? Is is ...
1
vote
2answers
828 views

Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?

Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites. Do ...
3
votes
0answers
1k views

What is a good open source web crawler? [closed]

I'm looking for a good open source web crawler and i found these: DataparkSearch, GNU Wget, GRUB, Heritrix, ht://Dig, HTTrack, ICDL, mnoGoSearch, Nutch, Open Search Server, PHP-Crawler, tkWWW Robot, ...
1
vote
1answer
100 views

SEO preparation for upcoming new site

I booked a domain and I created an image saying that the site is soon to be launched. What SEO preparations should I make to avoid harming my SEO ranking? Because my page is blank, the image itself ...
2
votes
1answer
104 views

google removed links from the index [duplicate]

I asked google to crawl a site through webmaster tool and include it to the index. How actually it was done: Firstly, i picked some urls manually. Approx. after half of a day those pages were ...
1
vote
1answer
84 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
2answers
536 views

Fake links cause crawl error in Google Webmaster Tools

Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message: Google detected a significant increase in the number of URLs that return a 404 (Page ...
0
votes
1answer
119 views

Spiders crawling broken links

Tl:Dr Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin. The Problem It seems at least once a week we're getting a big ...
25
votes
1answer
5k views

Bingbot request for trafficbasedsspsitemap.xml which does not exist

The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are Path: /trafficbasedsspsitemap.xml Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; ...
1
vote
2answers
97 views

Can Google show some pages deleted six months back and blocked using Robot.txt?

If I have deleted some pages on my website six months back and blocked these pages using Robots.txt, can that non-existent page show up on Google?
2
votes
1answer
317 views

Is there a spider / link checker that can start deep inside a login-protected site

We use vendor hosted Blackboard for our distance education courses, but host course multimedia on our own servers. The multimedia server has been moved and the domain has changed. Blackboard DBAs have ...
1
vote
1answer
99 views

Crawling retail websites to use as templates

I'm setting up a commerce website. I really like the Overstock website, and I was wondering if could simply crawl the website for the webpages, remove Overstock's logos, and use their website as a ...
3
votes
2answers
692 views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
2
votes
1answer
182 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
8
votes
2answers
2k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
0
votes
2answers
423 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
3answers
2k views

Google Webmaster Tools - Crawl errors with 500 status code

I will try to explain the situation: In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs. After a while I changed something on my website, upgrading my pages ...
5
votes
4answers
296 views

Total Indexed 0, 99 not selected in webmaster tools

I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
0
votes
0answers
346 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
1
vote
2answers
192 views

Why is Verisign spidering my website?

In my logs, I keep noticing that 69.58.178.57, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
2
votes
1answer
84 views

Are the following assumptions about indexing and ranking correct?

Are the following statements true or false: If a page is crawled by Google, it automatically gets into the index (unless explicitly blocked by robots.txt) An indexed page always gets a ranking (high ...
2
votes
2answers
213 views

Can Google stop crawling/indexing your site because you are working a lot on it?

I think I read somewhere that Google could stop crawling/indexing a site/blog if there was a lot of work performed on it (like many pages are modified, deleted or loaded daily). It would wait until ...
3
votes
1answer
520 views

Preventing botvisits to website

Every time some user shares my website's address inside his/her tweets, the following bots come to my website: UnwindFetchor/1.0 (+http://www.gnip.com/) ShowyouBot (http://showyou.com/crawler) ...
2
votes
2answers
125 views

Prevent crawler that doesn't honoring robots.txt

I have some problem, when I try to write robots.txt for my site ... I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
2
votes
1answer
173 views

Googlebot keeps returning to crawl my page then the rankings fluctuate

I know that in order to get ranked the page has to be crawled by Google bots, I have had a page example of which I will get a better understanding of how the system works with Google. I have made a ...
1
vote
1answer
148 views

What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?

I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
5
votes
1answer
256 views

How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache

The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
0
votes
1answer
140 views

403 Error Crawling Pages

I am receiving more than 6000 errors in Google Webmaster Tools. It is showing "access denied"; can anyone please help me out with resolving this?
1
vote
1answer
103 views

Recommend any tool to place .docx files on website for viewing so that search engines crawl it?

Recommend any tool to place .docx files on website for viewing so that search engines crawl it without any problem. I use scribd viewer, set the document as private on scribd and embed the file on my ...
1
vote
0answers
22 views

Does google crawl css scripts to check of hidden content [duplicate]

Possible Duplicate: hidden texts and page rank Does googlebot crawl CSS scripts to determine if content is hidden or not? If so, is it advisable to hide or not to hide content to improve on ...
1
vote
1answer
143 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
2
votes
2answers
59 views

If C-Panel Indexing Manager sets a folder to “No Indexing” can it be crawled by a webcrawler?

People are able to view directories / folders on my site right now. So, they could go to mysite.com/images and see the full index. To prevent this, C-Panel offers an option to set a directory / ...
0
votes
0answers
18 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
0
votes
1answer
259 views

mod_rewrite and SEO friendliness

My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes. I have a unique, large PHP script, and I use the Apache ...
3
votes
1answer
201 views

webmaster tools 500 crawl error for asp faceted navigation that does not exist

I am getting around 2,500 type 500 URL errors in Google webmaster tools. These pages are faceted navigation results that can not be reached by a site visitor. These pages do not exist. We are using ...
2
votes
2answers
82 views

Google crawls my website too slowly [duplicate]

Possible Duplicate: What are the best ways to increase a site's position in Google? I think Google crawls it too slowly (every three days). I updated it daily and it's kind of ...
8
votes
1answer
226 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success ...
4
votes
2answers
596 views

Number of page requests by any Bot in 5 secs

I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?