Crawling is a process associated with the actions of Bots. Search engine bots or scrapers will generally 'Crawl' a website to obtain data.
1
vote
3answers
83 views
How does Googlebot crawl a webpage? [closed]
Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
1
vote
1answer
37 views
Stop Google crawling links completely
I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told ...
1
vote
1answer
33 views
Bingbot Crawling URLS Like Folders, Causing Thousands Of 404 Errors
I first started seeing this a little over two months ago on a site I run but now I am seeing it on several others I manage. It appears the bot is trying to parse the URL into folders and this is ...
1
vote
1answer
28 views
Ahrefs is checking my site for backlinks all the time
I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good).
Is it good or bad, something like ...
0
votes
0answers
18 views
Using Ajax #! for Google but site is not being crawled any more
We used to have all links in a normal way (thus with parameters?), we changed to Ajax loading cause we installed a music player. Here's an example:
Old links: ...
1
vote
1answer
45 views
Webmaster tools reports fake 404 errors
I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...
2
votes
1answer
66 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
0answers
31 views
Does Google bot always treat dot as word separator?
One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
1answer
70 views
Ajax crawling : snapshot page dissimilar with user page
For ajax-crawling purpose I make snapshots pages. My problem is user view pages and snapshot pages are dissimilar (tags h1, h2...).
Example:
user view page: http://www.meomap.net/#!/place/1
...
1
vote
1answer
72 views
Googlebot craw rate is too slow for a huge site
I have a new website with 140k pages, according to my sitemaps.
After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
0
votes
1answer
56 views
Google adsense crawler unable to access your page [closed]
I have a problem which I have been trying to solve by Googling it but it seems I can’t get no answer. I have been using plain PHP and HTML on my site and everything was ok concerning my Google ...
4
votes
3answers
147 views
Do i need to have a separate page for each article on my website?
I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google?
Is is ...
0
votes
1answer
191 views
Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?
Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites.
Do ...
3
votes
0answers
475 views
What is a good open source web crawler? [closed]
I'm looking for a good open source web crawler and i found these:
DataparkSearch, GNU Wget, GRUB, Heritrix, ht://Dig, HTTrack, ICDL, mnoGoSearch, Nutch, Open Search Server, PHP-Crawler, tkWWW Robot, ...
2
votes
1answer
67 views
google removed links from the index [duplicate]
I asked google to crawl a site through webmaster tool and include it to the index.
How actually it was done:
Firstly, i picked some urls manually. Approx. after half of a day those pages were ...
0
votes
2answers
140 views
Fake Links Cause Crawl Error In Google Webmaster Tools
Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message:
Google detected a significant increase in the number of URLs that
return a 404 (Page ...
0
votes
0answers
14 views
Chinese IP's/Scanners [closed]
My website is getting many hits from china for no apparent reason, I know they must be bots, but what is their purpose? brute force, scan for vulnerabilities or collect statistics etc etc?
Should I ...
2
votes
2answers
114 views
How to remove Ever crawled web Pages by Google
I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
0
votes
2answers
166 views
GWT: reporting crawling errors for non existing links
Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed.
They all mention ...
1
vote
2answers
327 views
Google Webmaster Tools - Crawl errors with 500 status code
I will try to explain the situation:
In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs.
After a while I changed something on my website, upgrading my pages ...
5
votes
4answers
198 views
Total Indexed 0, 99 not selected in webmaster tools
I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
2
votes
1answer
80 views
Are the following assumptions about indexing and ranking correct?
Are the following statements true or false:
If a page is crawled by Google, it automatically gets into the index (unless explicitly blocked by robots.txt)
An indexed page always gets a ranking (high ...
2
votes
2answers
162 views
Can Google stop crawling/indexing your site because you are working a lot on it?
I think I read somewhere that Google could stop crawling/indexing a site/blog if there was a lot of work performed on it (like many pages are modified, deleted or loaded daily). It would wait until ...
0
votes
0answers
74 views
why my website doesn't ranked by alexa? [closed]
I created a website with WordPress and post 10+ article in period of two month, but alexa doesn't rank my website. i tried to change my theme, URL and other related things and submit my website URL ...
1
vote
1answer
94 views
Google ranking, page crawl
I know that in order to get ranked the page has to be crawled by Google bots, I have had a page example of which I will get a better understanding of how the system works with Google.
I have made a ...
3
votes
1answer
115 views
How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache
The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
0
votes
1answer
66 views
403 Error Crawling Pages
I am receiving more than 6000 errors in Google Webmaster Tools. It is showing "access denied"; can anyone please help me out with resolving this?
1
vote
1answer
61 views
Recommend any tool to place .docx files on website for viewing so that search engines crawl it?
Recommend any tool to place .docx files on website for viewing so that search engines crawl it without any problem.
I use scribd viewer, set the document as private on scribd and embed the file on my ...
1
vote
0answers
22 views
Does google crawl css scripts to check of hidden content [duplicate]
Possible Duplicate:
hidden texts and page rank
Does googlebot crawl CSS scripts to determine if content is hidden or not? If so, is it advisable to hide or not to hide content to improve on ...
1
vote
1answer
120 views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
0
votes
0answers
17 views
Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]
Possible Duplicate:
How do spambots work?
I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
3
votes
1answer
126 views
webmaster tools 500 crawl error for asp faceted navigation that does not exist
I am getting around 2,500 type 500 URL errors in Google webmaster tools. These pages are faceted navigation results that can not be reached by a site visitor. These pages do not exist. We are using ...
2
votes
2answers
69 views
Google crawls my website too slowly [duplicate]
Possible Duplicate:
What are the best ways to increase a site's position in Google?
I think Google crawls it too slowly (every three days). I updated it daily and it's kind of ...
2
votes
2answers
451 views
How to crawl a webPage with dynamic content added by javascript
I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using ...
2
votes
3answers
336 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
0
votes
0answers
11 views
Indexing big amount of keywords by google [duplicate]
Possible Duplicate:
What are the best ways to increase a site's position in Google?
I have a site with big amount of keywords to be index ;-)
It is a classified / listing site and I am ...
2
votes
1answer
49 views
adding tagged / dynamic pages in sitemap
ive got a blog thats been running for about a year ive made about 200 posts, and there should be about 220 pages to index (additional pages for about / contact ect).
When i go to crawl the site i get ...
2
votes
2answers
72 views
Is there a weight for html link?
I would like to know if there is any weight associated with a html link (its backlink) when google does crawling/indexing. Will 1,2 & 3 be ever considered as a backlink by Google?
1. <a ...
0
votes
0answers
148 views
Mobile Sitemap in Google Webmaster Tools [closed]
I have a problem with my mobile sitemap because google index the urls as web instead of mobile. Why?
You can check the sitemap and one url.
I need your help.
Thanks
7
votes
2answers
1k views
Ask.com Sitemap Crawler Down (for good)?
I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
0
votes
3answers
74 views
Free standalone crawlers
Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors?
Something that shows me all 404's, errors etc
1
vote
1answer
229 views
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site.
What's the symptom? Google + Bing crawl, but don't index any of the pages.
Let me be clear: I'm ...
3
votes
2answers
176 views
Can Google tell the difference between a link in a blog post and a link in a blog comment?
When Google Blog Search crawls a page, can it tell whether a link is in the article or whether it's in a comment?
If it can, does it grade the links, that is, a link from a blog comment is only going ...
3
votes
1answer
503 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
1
vote
1answer
490 views
How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?
I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
1
vote
4answers
109 views
Is it bad for SEO to have a site with no click navigation?
I have a site where users can only get to most of the sites content through search. If I make a sitemap that has the URLs for all of the potential searches is it ok that these pages cannot be click ...
4
votes
2answers
461 views
Understanding Ajax crawling of search site
I have a couple of questions about Ajax crawling of a website, which is kind of search engine itself.
The base article explains the mechanism of making AJAX applications crawlable. All this stuff ...
2
votes
1answer
154 views
Will google “forget” unlinked pages?
If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...
3
votes
1answer
286 views
How to interpret number of URL errors in Google webmaster tools
Recently Google has made some changes to Webmaster tools which are explained below:
http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html
One thing I could not find out ...
5
votes
3answers
423 views
Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]
Possible Duplicate:
Is it possible for web crawlers to see static pages without following a link to them?
I have some URLs (some pdfs and static HTML files) in my website that I want only ...
