Hot answers tagged

6

The Wayback Machine archive is a combination of data from a large number of different crawls: Alexa crawls, which appear after a 6 month delay Our own crawls, which are seeded from the Alexa top million list and others ArchiveTeam crawls, done by volunteers ArchiveIt crawls, done by our 400+ partners, mostly libraries, many of which allow their data to be ...


5

Google, conceptually, uses an HTML DOM parser. What this does is break any web page HTML down into its basic structure and each HTML tag is given an ID. This ID represents the order of the HTML tags from beginning to end, any dependency between HTML elements such as a li tag is dependent upon a ul tag, any parent-child relationship between HTML elements ...


3

You could always check what your site looks like to googlebot: https://support.google.com/webmasters/answer/6066468?rd=2 and to be sure look at the access logs of your server. There doesn't seem to be any information on the engine used w/i googlebot but if it's Chrome 36+ it will work. Alternatively, you could use webcompontent.js Polyfills at http://...


2

There are really 2 issues here: Will the robots.txt on your site Disallow (block) Wayback from crawling your site. Will Wayback crawl your site. For point #1: As others have said, the correct entry for robots.txt is: User-agent: ia_archiver Disallow: Keep in mind that it might take a while (perhaps a good long while), for Wayback to notice any changes ...


1

The default directive for spiders is allow and when Google visits your robots.txt and sees the disallow command, it immediately knows that it isn't allowed to crawl the website anymore. The subsequent allows don't matter. Do note that when there are conflicting codes within the robots file or between the robots.txt and the Meta Robots tag of a page, spiders ...


1

If you have a new page, you probably have a link to it. Google visits your page often and will catch the link. If it's a matter of new content, that may take longer because I don't think Google examines content as often as it examines links. This is why they recommend a sitemap. When you have a significant change to a page, you can update your sitemap and, ...


1

To answer your first point having reviewed your example site you do not have any indexable content or keywords. To index the content Google needs content to index. As your initial page doesn't have anything except for links Google will not continue down the path of indexing those links as it makes the website appear as not legitimate and a possible blackhat ...


1

Use your chosen language to detect the user agent. Most legitimate search engine crawlers have their own user agent to identify it as a crawler. Then you can simply ignore those user agents.


1

From your comments on another answer, MJ12Bot is visiting your site less than once an hour (421 times in 25 days.) The best thing to do is to not worry about it. Crawl-Delay is useless for you because no crawler will obey a craw-delay so large.



Only top voted, non community-wiki answers of a minimum length are eligible