The web-crawlers tag has no wiki summary.
8
votes
1answer
154 views
Disqus thread migration. Gotchas?
I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads.
I've had partial success ...
1
vote
1answer
404 views
Robots.txt practices with .htaccess redirections (inherits)
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place.
We have a hosting account that enacts primary and add-on domains. All of our domains ...
2
votes
0answers
915 views
Is feeding Googlebot and Bingbot crawlers with “special” data common thing in RIAs?
So I faced next problem: my server and my RIA client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull no js crawler will see not ...
0
votes
0answers
42 views
Is there an automated way to take site inventory?
Is there a way to take site inventory using a crawler program that checks either the sources of images for specific servers that serve ads, or, that the crawler looks at a page for specific (html5?) ...
0
votes
0answers
48 views
Need to know how to configure robots.txt for an age gate/age verification website
is it the normal allow all in the file? I need to make sure the crawlers can get through the gate. Thanks!
