0
votes
0answers
9 views

Rejection of useless links during web crawling [on hold]

I wrote a web crawler in Python, which opens one website and collects all links from this website. But some links are not driving to other pages or subpages (with other informations), but are driving ...
1
vote
3answers
204 views

Right way to allow all access to website in robots.txt?

user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?
1
vote
1answer
98 views

Crawling retail websites to use as templates

I'm setting up a commerce website. I really like the Overstock website, and I was wondering if could simply crawl the website for the webpages, remove Overstock's logos, and use their website as a ...
0
votes
0answers
56 views

Are there any good reasons to HTML5 for SEO at the moment? [duplicate]

Possible Duplicate: What are the SEO rules for HTML 5? Do crawlers ignore HTML5 tags (like <article>, <section>, <video>, <nav>, and so on...) when indexing the ...
9
votes
4answers
2k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...