Tag Info

New answers tagged

2

To prevent your whole site from being crawled, then No. 2: User-agent: * Disallow: / This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path ...


1

Googles bot will stick around as long as it needs to unless it times out or is stupidly slow. Regardless what the content is 5 seconds is far to slow and your site will suffer a user experience algorithm against it, which in term will effect your rankings, after-all, no one likes a slow website or page. Rough idea on page speed: 0-1seconds awesome ...


2

Since you use tags that start with web, I assume you are building a system that requires HTML to be downloaded in order for a paid impression to work. What you need to do is learn robot behaviour by looking at the server access log files. On a server with apache installed, the file is typically named access_log. If your server is used frequently, you will ...


-2

Get a list of known referral spam URL's and filter them through Analytics. https://megalytic.com/blog/how-to-filter-out-fake-referrals-and-other-google-analytics-spam


1

There's some information about this on Wikipedia, Snapshots usually become available more than 6 months after they are archived or in some cases even later, 24 months or longer. The frequency of snapshots is variable, so not all tracked web site updates are recorded. There are sometimes intervals of several weeks or years between snapshots. ...


-1

I can now confirm that crawler will add the full path as the value of the param. Thus www.example.com/about/info will be crawled as www.example.com?_escaped_fragment_=about/info Note: In order for crawlers to crawl in this manner, <meta name="fragment" content="!"> has to be added to the head of your main ajax page e.g. index.html


4

The pages will be found and most likely be crawled. You're not far from the truth if you assume bots don't use Javascript. A crawler works similar to this: Go to a webpage and get it's contents Get all the information from the page, like keywords and all other SEO checks Get all internal and external links from the source Per link, go to step 1. It ...


0

I would recommend to add sitemap directive and Host (for Google and Yandex) in file robots.txt. Host allows websites with multiple mirrors to specify their preferred domain. Sitemap: http://www.example/sitemap.xml Host: example.com


2

I will answer your question exactly as asked in the hope that it will persuade you to reconsider if the root cause of your problem is not better resolved in some other way. Is it possible to completely block a page from being crawled under any circumstance? Not in a typically configured web server. The robots.txt file is just a polite request by your ...


1

Yes, Google can do that and it is best to assume that anything that is publicly available on the internet may be indexed by Google. Linked to or not. Of course, if you don't link to it the chances of it being indexed go way down. However, Google uses a multitude of tools to gather URLs for indexing. Recently there was a news item about Dropbox links that ...


1

Think of the linking as a chain reaction. Google won't link to domains if it has no way of accessing it or even finding it. If a friend advertises your URL on a popular forum site that Google always indexes, then there's a chance Google will scan your URL and possibly index it, thinking the link may be part of the site. Is the fact I've installed ...


1

Usually it is supposed to block GoogleBot if you don't want it to crawl. You can simply do it by editing your robots.txt and blocking access to the pdf folder. For example: User-agent: * Disallow: listings/index/getpdf/* If you want Google to access and index it, you should let him see your files without redirecting it or asking for download. You can do ...


2

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why? I can honestly see zero benefit in doing this, so therefore, my answer to your question is Bad. I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Any ...



Top 50 recent answers are included