Crawling is a process associated with the actions of Bots. Search engine bots or scrapers will generally 'Crawl' a website to obtain data.
1
vote
3answers
83 views
How does Googlebot crawl a webpage? [closed]
Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
0
votes
2answers
165 views
GWT: reporting crawling errors for non existing links
Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed.
They all mention ...
1
vote
1answer
37 views
Stop Google crawling links completely
I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told ...
1
vote
1answer
45 views
Webmaster tools reports fake 404 errors
I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...
1
vote
1answer
69 views
Ajax crawling : snapshot page dissimilar with user page
For ajax-crawling purpose I make snapshots pages. My problem is user view pages and snapshot pages are dissimilar (tags h1, h2...).
Example:
user view page: http://www.meomap.net/#!/place/1
...
1
vote
1answer
32 views
Bingbot Crawling URLS Like Folders, Causing Thousands Of 404 Errors
I first started seeing this a little over two months ago on a site I run but now I am seeing it on several others I manage. It appears the bot is trying to parse the URL into folders and this is ...
7
votes
2answers
1k views
Ask.com Sitemap Crawler Down (for good)?
I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
0
votes
1answer
190 views
Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?
Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites.
Do ...
1
vote
1answer
72 views
Googlebot craw rate is too slow for a huge site
I have a new website with 140k pages, according to my sitemaps.
After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
1
vote
1answer
28 views
Ahrefs is checking my site for backlinks all the time
I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good).
Is it good or bad, something like ...
3
votes
1answer
115 views
How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache
The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
0
votes
0answers
18 views
Using Ajax #! for Google but site is not being crawled any more
We used to have all links in a normal way (thus with parameters?), we changed to Ajax loading cause we installed a music player. Here's an example:
Old links: ...
5
votes
4answers
198 views
Total Indexed 0, 99 not selected in webmaster tools
I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
2
votes
1answer
66 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
0
votes
2answers
140 views
Fake Links Cause Crawl Error In Google Webmaster Tools
Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message:
Google detected a significant increase in the number of URLs that
return a 404 (Page ...
2
votes
2answers
112 views
How to remove Ever crawled web Pages by Google
I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
1
vote
0answers
31 views
Does Google bot always treat dot as word separator?
One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
0
votes
1answer
56 views
Google adsense crawler unable to access your page [closed]
I have a problem which I have been trying to solve by Googling it but it seems I can’t get no answer. I have been using plain PHP and HTML on my site and everything was ok concerning my Google ...
3
votes
4answers
329 views
Google Bots Crawling my Site daily is this good or a bad thing?
Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
1answer
503 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
2
votes
0answers
927 views
Is feeding Googlebot and Bingbot crawlers with “special” data common thing in RIAs?
So I faced next problem: my server and my RIA client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull no js crawler will see not ...
0
votes
1answer
66 views
403 Error Crawling Pages
I am receiving more than 6000 errors in Google Webmaster Tools. It is showing "access denied"; can anyone please help me out with resolving this?
4
votes
3answers
147 views
Do i need to have a separate page for each article on my website?
I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google?
Is is ...
3
votes
2answers
176 views
Can Google tell the difference between a link in a blog post and a link in a blog comment?
When Google Blog Search crawls a page, can it tell whether a link is in the article or whether it's in a comment?
If it can, does it grade the links, that is, a link from a blog comment is only going ...
1
vote
1answer
61 views
Recommend any tool to place .docx files on website for viewing so that search engines crawl it?
Recommend any tool to place .docx files on website for viewing so that search engines crawl it without any problem.
I use scribd viewer, set the document as private on scribd and embed the file on my ...
3
votes
0answers
470 views
What is a good open source web crawler? [closed]
I'm looking for a good open source web crawler and i found these:
DataparkSearch, GNU Wget, GRUB, Heritrix, ht://Dig, HTTrack, ICDL, mnoGoSearch, Nutch, Open Search Server, PHP-Crawler, tkWWW Robot, ...
2
votes
1answer
66 views
google removed links from the index [duplicate]
I asked google to crawl a site through webmaster tool and include it to the index.
How actually it was done:
Firstly, i picked some urls manually. Approx. after half of a day those pages were ...
1
vote
2answers
327 views
Google Webmaster Tools - Crawl errors with 500 status code
I will try to explain the situation:
In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs.
After a while I changed something on my website, upgrading my pages ...
3
votes
1answer
126 views
webmaster tools 500 crawl error for asp faceted navigation that does not exist
I am getting around 2,500 type 500 URL errors in Google webmaster tools. These pages are faceted navigation results that can not be reached by a site visitor. These pages do not exist. We are using ...
0
votes
0answers
74 views
why my website doesn't ranked by alexa? [closed]
I created a website with WordPress and post 10+ article in period of two month, but alexa doesn't rank my website. i tried to change my theme, URL and other related things and submit my website URL ...
0
votes
0answers
11 views
Indexing big amount of keywords by google [duplicate]
Possible Duplicate:
What are the best ways to increase a site's position in Google?
I have a site with big amount of keywords to be index ;-)
It is a classified / listing site and I am ...
3
votes
2answers
508 views
How to correctly index multilanguage home page while having a default portal language
Hi there this is my first post, so first of all, sorry for being boring or for any mistakes.
I have read many posts on how to correctly index a multi-language site but none of them really answer my ...
1
vote
2answers
529 views
SEO: disallowing Google from indexing forms?
From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not?
I mean my forms hardly ever contains keyword/keyphrases.
So I'm wondering what's the point of letting ...
1
vote
3answers
172 views
Why isnt google crawling on my blog? [duplicate]
I have a blog.I made it on August,2010,so it has been a while but I am unable to make Google crawl on my site.I have requested Google to crawl but I still there is no response.Any reasons to note why ...
0
votes
0answers
14 views
Chinese IP's/Scanners [closed]
My website is getting many hits from china for no apparent reason, I know they must be bots, but what is their purpose? brute force, scan for vulnerabilities or collect statistics etc etc?
Should I ...
1
vote
1answer
94 views
Google ranking, page crawl
I know that in order to get ranked the page has to be crawled by Google bots, I have had a page example of which I will get a better understanding of how the system works with Google.
I have made a ...
2
votes
1answer
80 views
Are the following assumptions about indexing and ranking correct?
Are the following statements true or false:
If a page is crawled by Google, it automatically gets into the index (unless explicitly blocked by robots.txt)
An indexed page always gets a ranking (high ...
2
votes
2answers
162 views
Can Google stop crawling/indexing your site because you are working a lot on it?
I think I read somewhere that Google could stop crawling/indexing a site/blog if there was a lot of work performed on it (like many pages are modified, deleted or loaded daily). It would wait until ...
0
votes
2answers
113 views
404s on password protected content
I'm new to WordPress and SEO, generally, but we've been running into problems with our site that don't seem to make sense to me.
The problem is that our editor likes to schedule posts and/or mark ...
1
vote
1answer
120 views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
1
vote
0answers
22 views
Does google crawl css scripts to check of hidden content [duplicate]
Possible Duplicate:
hidden texts and page rank
Does googlebot crawl CSS scripts to determine if content is hidden or not? If so, is it advisable to hide or not to hide content to improve on ...
0
votes
0answers
17 views
Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]
Possible Duplicate:
How do spambots work?
I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
2
votes
2answers
69 views
Google crawls my website too slowly [duplicate]
Possible Duplicate:
What are the best ways to increase a site's position in Google?
I think Google crawls it too slowly (every three days). I updated it daily and it's kind of ...
5
votes
5answers
186 views
What measures can be taken to make sure Google is aware of the existence of a newly created page?
Consider a website with a large number of pages. New pages are published regularly.
When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
2
votes
2answers
451 views
How to crawl a webPage with dynamic content added by javascript
I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using ...
2
votes
1answer
49 views
adding tagged / dynamic pages in sitemap
ive got a blog thats been running for about a year ive made about 200 posts, and there should be about 220 pages to index (additional pages for about / contact ect).
When i go to crawl the site i get ...
0
votes
3answers
74 views
Free standalone crawlers
Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors?
Something that shows me all 404's, errors etc
2
votes
3answers
336 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
6
votes
8answers
2k views
How long did it take for your new website to be indexed by search engines [duplicate]
Possible Duplicate:
Why isn't my website in Google search results?
I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
2
votes
2answers
72 views
Is there a weight for html link?
I would like to know if there is any weight associated with a html link (its backlink) when google does crawling/indexing. Will 1,2 & 3 be ever considered as a backlink by Google?
1. <a ...
