Hot answers tagged

22

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty ...


12

Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site being indexable by search engines other than Google, you may want to still support this scheme. Certainly, if you already have set up support for ...


11

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


10

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...


9

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


9

You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules. RewriteRule ^robots\.txt$ /robots.php [L] That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...


6

No. Google may sometimes execute Javascript to determine content, but it's a very bad idea to rely on Javascript for your site to be crawled as you want it to be.


6

Here is what Google's John Mueller says about Noindex: in robots.txt: We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don't think other search engines are using that at all. deepcrawl.com has done some testing of the feature and discovered that: It still works ...


5

So, the solution seems to be that Amazon cloudfront also evaluates my robots.txt and somehow uses different syntax rules from google. The working version of my robots.txt is the following: User-agent: Googlebot-Image Disallow: / User-agent: * Disallow: /homepage Disallow: /uncategorized Disallow: /page Disallow: /category Disallow: /author Disallow: /feed ...


5

Okay. This is a new one (to me at least) and quite interesting so far. I will not get into the weeds on this. When I wrote this, I was working on little or no sleep. I missed a few things which @unor has kindly pointed out and so I must temper my answer and give credit where credit is due. Thank you @unor! Archive.is is registered to Denis Petrov who is ...


4

The pages will be found and most likely be crawled. You're not far from the truth if you assume bots don't use Javascript. A crawler works similar to this: Go to a webpage and get it's contents Get all the information from the page, like keywords and all other SEO checks Get all internal and external links from the source Per link, go to step 1. It ...


4

Yes, the same way any request can be "dynamic". However, you would not redirect (as in your example code), you should internally rewrite using mod_rewrite. (The same as what Drupal is probably already doing.) For example, in your root .htaccess file: RewriteEngine On RewriteRule ^robots\.txt$ robots.php [L] RewriteEngine should only occur once (although ...


4

Your robots.txt should be in the root directory and should not have any other name. According to the standard specification: This file must be accessible via HTTP on the local URL "/robots.txt".


3

The space is a delimiter (ie. a special character) in .htaccess so must be backslash escaped if you want to match a literal space in the regex. Eg. DV\ CRAWLER. (Otherwise you are likely to get a less than helpful 500 Internal Server error.) Or, you can use the shorthand character class \s which matches any white space character (space, tab or new line / ...


3

Yes, Google can do that and it is best to assume that anything that is publicly available on the internet may be indexed by Google. Linked to or not. Of course, if you don't link to it the chances of it being indexed go way down. However, Google uses a multitude of tools to gather URLs for indexing. Recently there was a news item about Dropbox links that ...


3

Since you use tags that start with web, I assume you are building a system that requires HTML to be downloaded in order for a paid impression to work. What you need to do is learn robot behaviour by looking at the server access log files. On a server with apache installed, the file is typically named access_log. If your server is used frequently, you will ...


3

No, the last two lines of your robots.txt file take precedence over the first four as the syntax of those first four lines is incorrect. As a result Google is blocked from viewing your website. To allow Google and Bing you must specifically and individually allow each crawler: User-agent: googlebot Disallow: User-agent: bingbot Disallow: User-agent: * ...


3

Your user agent for Google wrong. It should be Googlebot. A user agent of Google is probably not recognized. See Google's "robots.txt" section of their help article Monitor crawling activity and errors.


3

robots.txt disallow rules are all "starts with" rules, not substring rules. MediaWiki suggests using this in robots.txt for a case like yours: User-agent: * Disallow: /index.php? Disallow: /index.php/Help Disallow: /index.php/MediaWiki Disallow: /index.php/Special: Disallow: /index.php/Template Disallow: /skins/ Google says that it supports more advanced ...


3

robots.txt Archive.is does not use a bot that autonomously crawls pages (e.g., by following hyperlinks), so robots.txt does not apply, because it’s always a user that gives the command to archive a certain page. For the same reason, services like Google’s Feedfetcher (Why isn't Feedfetcher obeying my robots.txt file?) and W3C’s Validator (details) don’t ...


3

If you really want to prevent a link from being indexed or followed, you can go extreme as follows: If you're using a server-side scripting language or have sufficient apache access, then modify code so that the page to not be indexed will have an HTTP 410 status code attached to it, meaning the page is gone for good. This will effectively cause previous ...


2

Disallow: *?s= Bots following the original robots.txt specification would not be allowed to crawl URLs like these: http://example.com/*?s= http://example.com/*?s=foo http://example.com/*?s=/ So they interpret *, ? and = literally (i.e., these characters have to appear at the beginning of the URL path). But many bots use (their own) extensions to the ...


2

I would highly recommend the Screaming Frog SEO Spider. He gets you all URLs, Status Codes, Titles, Descriptions, Images on your Website, etc. It's really a great SEO tool, as far as I know most SEOs use it. You can export csv- and Excel-files, as you requested, and also you can create sitemaps, image-sitemaps, and so on. The problem here is that the free ...


2

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why? I can honestly see zero benefit in doing this, so therefore, my answer to your question is Bad. I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Any ...


2

I will answer your question exactly as asked in the hope that it will persuade you to reconsider if the root cause of your problem is not better resolved in some other way. Is it possible to completely block a page from being crawled under any circumstance? Not in a typically configured web server. The robots.txt file is just a polite request by your ...


2

A few more methods of bot traffic detecting: Verification of user agent (it is a client application that uses a particular network protocol). Looking for a highly specific match to something like a malware signature or specific executable or C&C connection address. Examining such behavioral parameters as a depth of view, the duration of visit, ...


2

To prevent your whole site from being crawled, then No. 2: User-agent: * Disallow: / This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path ...


2

According to this we are still waiting to see what Bing will do, so I would assume (I know) not. Here is an excerpt regarding Bing: What Can We Expect? Google’s disregard for JavaScript in the past has also been used by many websites to their advantage. It will be important to understand the repercussions of Google being able to index all of ...


2

So I set up a test page here. Using Webmaster Tools' Fetch As Google feature, I saw that Google doesn't pick up the image declared in the img's srcset attribute: However, adding the JavaScript Polyfill Picturefill I was very surprised to see that Google now does pick up the image declared in the srcset attribute. This means Google is running the ...


2

This is not a good setup. Canonical pointing to page with different canonical will surely lead to problems (including indexation and rankings), sooner or later. The best scenario in your particular example is to 301 redirect the http://www.example.com/phone/great-page.html to http://www.example.com/phone/even-greater-page.html. The best practice is to use ...



Only top voted, non community-wiki answers of a minimum length are eligible