Hot answers tagged

12

Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site being indexable by search engines other than Google, you may want to still support this scheme. Certainly, if you already have set up support for ...


11

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


10

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...


10

You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules. RewriteRule ^robots\.txt$ /robots.php [L] That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...


6

Here is what Google's John Mueller says about Noindex: in robots.txt: We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don't think other search engines are using that at all. deepcrawl.com has done some testing of the feature and discovered that: It still works ...


5

Yes, the same way any request can be "dynamic". However, you would not redirect (as in your example code), you should internally rewrite using mod_rewrite. (The same as what Drupal is probably already doing.) For example, in your root .htaccess file: RewriteEngine On RewriteRule ^robots\.txt$ robots.php [L] RewriteEngine should only occur once (although ...


5

Okay. This is a new one (to me at least) and quite interesting so far. I will not get into the weeds on this. When I wrote this, I was working on little or no sleep. I missed a few things which @unor has kindly pointed out and so I must temper my answer and give credit where credit is due. Thank you @unor! Archive.is is registered to Denis Petrov who is ...


4

The pages will be found and most likely be crawled. You're not far from the truth if you assume bots don't use Javascript. A crawler works similar to this: Go to a webpage and get it's contents Get all the information from the page, like keywords and all other SEO checks Get all internal and external links from the source Per link, go to step 1. It ...


4

Your robots.txt should be in the root directory and should not have any other name. According to the standard specification: This file must be accessible via HTTP on the local URL "/robots.txt".


4

If you're already 301 redirecting HTTP traffic to HTTPS then you've got the biggest step already done. Any links to HTTP pages will get redirected to the relevant HTTPS page, and search engines will follow those the same as real users. As long as your sitemap refers to the HTTPS version as well you should be fine. As for more niggly bits, I'd also suggest ...


4

Looks like it. It even looks like Google recommends using pushShate and prefers it over hash bangs.


4

You are right, rel=nofollow is only for links. <img> tag can't have rel=nofollow attribute. What do you mean by disallow google to crawl out of my website through images? Google bot doesn't crawl out when it encounters an external image. It just gives a positive (or sometimes negative) recognition to it and may index it - Google can't crawl an ...


3

Since you use tags that start with web, I assume you are building a system that requires HTML to be downloaded in order for a paid impression to work. What you need to do is learn robot behaviour by looking at the server access log files. On a server with apache installed, the file is typically named access_log. If your server is used frequently, you will ...


3

Yes, Google can do that and it is best to assume that anything that is publicly available on the internet may be indexed by Google. Linked to or not. Of course, if you don't link to it the chances of it being indexed go way down. However, Google uses a multitude of tools to gather URLs for indexing. Recently there was a news item about Dropbox links that ...


3

My approach involves two passes: Filter only web browsers and consoles by matching the start of the user agent string with Mozilla|Opera|PSP|Bunjalloo|wii. Thanks to the user agent spoofing this check will detect almost all browsers Exclude bots by common stop strings bot|crawl|slurp|spider So if the first step is passed we assume that it is a browser ...


3

No, the last two lines of your robots.txt file take precedence over the first four as the syntax of those first four lines is incorrect. As a result Google is blocked from viewing your website. To allow Google and Bing you must specifically and individually allow each crawler: User-agent: googlebot Disallow: User-agent: bingbot Disallow: User-agent: * ...


3

Your redirects for users without cookies are a bad idea. Googlebot never sends cookies, but Google has data centers around the world. When Googlebot crawls from a data center in France, it won't be able to get to your English website and your English website could get dropped from the Google index. Location based redirects are also problematic for real ...


3

Will the Disallow code above stop robots from crawling all of my search result pages? Yes, it will stop the (good) bots that obey the robots.txt "standard". However, you don't need the trailing *. robots.txt is prefix matching, so the "wildcard" * at the end can simply be omitted. (Wildcard type matches are an extension of the original standard ...


3

Your user agent for Google wrong. It should be Googlebot. A user agent of Google is probably not recognized. See Google's "robots.txt" section of their help article Monitor crawling activity and errors.


3

robots.txt disallow rules are all "starts with" rules, not substring rules. MediaWiki suggests using this in robots.txt for a case like yours: User-agent: * Disallow: /index.php? Disallow: /index.php/Help Disallow: /index.php/MediaWiki Disallow: /index.php/Special: Disallow: /index.php/Template Disallow: /skins/ Google says that it supports more advanced ...


3

robots.txt Archive.is does not use a bot that autonomously crawls pages (e.g., by following hyperlinks), so robots.txt does not apply, because it’s always a user that gives the command to archive a certain page. For the same reason, services like Google’s Feedfetcher (Why isn't Feedfetcher obeying my robots.txt file?) and W3C’s Validator (details) don’t ...


3

If you really want to prevent a link from being indexed or followed, you can go extreme as follows: If you're using a server-side scripting language or have sufficient apache access, then modify code so that the page to not be indexed will have an HTTP 410 status code attached to it, meaning the page is gone for good. This will effectively cause previous ...


3

It look like a spam to me, sometimes your site get targeted by spam bots, you would want to fix that, but the solution wouldn't be to exclude them from GA but to exclude them from your website. This might be the least you can do about it : How to track user-agents who visit your website With a simple linux command you can track all user-agents that crawl ...


3

First, there's no reason to nofollow outbound links unless they're from unmoderated user-generated content (i.e. people could link to things you'd rather they didn't) or they've been paid for. Second, only this week a fairly thorough study established that not only is linking out not damaging to your SEO, it's beneficial.


3

All outbound links rel=nofollow? This is common question that arouses in mind of those who avail SEO marketing services There can be only one reason to do so and that is you are linking to all spammy sites on which you do not trust. So this may give a hint to Google that you are doing spam. Which i think will not be good for your SEO. So use rel=nofollow ...


3

I'm pretty sure Googlebot does not send HEAD requests - at least not with respect to crawling and indexing a website. My access logs from January and February 2016 show no HEAD requests from the Googlebot. Even if Googlebot did use HEAD requests I would be very surprised if these resulted in a 404 error in Google Search Console. Google should only report a ...


3

An internal web crawler can be used for the following purposes: Creating a Localized Search Engine. Scanning Pages to Detect Dead Links, where the target page was deleted or unpublished from a CMS, but the links to that page still exist in other pages. Finding Breadcrumb Click Trail Paths in massive websites. Finding Orphaned Pages in CMS generated ...


2

You can use an image as a counter, in this case it won't count bots and page name is passed as a query with image name I'm using this at img.php which update page view in database: <?php $xnt = $_GET["ID"]; if (isset($xnt) && is_numeric($xnt)) { $DBServer = "localhost"; $DBUser = "xxx"; $DBPass = "xxx"; $DBaze = "xxx"; $conn = ...


2

It is a very important factor for your website. When Google crawls your site daily it means your site matters a lot or it contains a very good information or content. When the Google crawler does not crawl your site daily. There are mainly two reasons: The site does not contain new and unique content Your server is slow. Please check it and solve the ...


2

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why? I can honestly see zero benefit in doing this, so therefore, my answer to your question is Bad. I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Any ...



Only top voted, non community-wiki answers of a minimum length are eligible