Tag Info

Hot answers tagged

35

This is not uncommon as there is no time frame for Google to index a website. The best thing you can do is to give Google every opportunity to find you. Try all of the following: To submit your site to Google go here. Submit an XML sitemap. You can do that from within your Webmaster Control panel. Here's a drupal add on to make this easy for you. Get ...


27

Lack of a robots.txt file will not be harmful. From the robotstxt.org website: To allow all robots complete access User-agent: * Disallow: (or just create an empty "/robots.txt" file, or don't use one at all) However, even if you are not specifying anything in your robots.txt file, it is a good way of informing search engines of the ...


15

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful ...


13

There are 2 main ways to prevent search engines from indexing specific pages: A Robots.txt file for your domain. The Meta Robots tag on each page. Robots.txt should be your first stop for URL patterns that match several files. You can see the syntax here and more detailed here. The robots.txt file must be placed in the root folder of your domain, i.e. at ...


13

I would strongly recommend registering your site with Webmaster Tools. There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where you are appearing in queries on Google. ...


11

You don't want the page to appear in the SERPs at all... Don't disallow in robots.txt. Add a noindex meta tag (or X-Robots-Tag HTTP header) to your pages instead. As j0k suggests, your pages could be found somehow. Stats reports, directory listings, etc... Disallowing in robots.txt prevents the page from being crawled, but could still be indexed and could ...


11

It takes some time for Googlebot to crawl your site. This amount of time can vary depending on the number of links to your site, among other factors. After your site is crawled, the URLs in the robot.txt file will be updated.


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


10

Hard to say for sure why you aren't indexed yet but: 1) Number of visitors has no bearing whatsoever on your indexing or rankings. Google doesn't know this information and, even if it they did, it really offers nothing in terms of relevance of any page for search. 2) 20 backlinks is hardly a lot. Even then, unless Google knows about those backlinks they ...


10

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.


10

Maybe someone didn't want to pay for spider traffic? Regardless, you are reading it correctly: http://www.robotstxt.org/robotstxt.html Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say ...


10

It seems that Google deliberately includes URLs disallowed in robots.txt in their index if there are links to those URLs from other pages they've crawled. To quote their Webmaster Tools help pages: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a ...


10

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed. User-agent: * Disallow: /templates_c But to answer your question, according to Google: At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the ...


10

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


8

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.


8

It's very simple to spoof a User-Agent header and pretend to be a Google Bot. But much harder to fake the I.P. from which the request is coming. Check that the I.Ps making these request are owned by Google.


8

Robots.txt is not used for preventing users to browse directories. Directory browsing is handled by the web server settings. All robots.txt does is asking search engines to not index the content within the directories that you specify.


8

You can't make them re-download your robots.txt when you want them to. Google will re-crawl it and use the new data whenever they feel it is appropriate for your site. They tend to crawl it regularly so I wouldn't expect it to take long for your updated file to be found and your pages re-crawled and re-indexed. Keep in mind that it may take some time after ...


8

It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL. I guess google just hasnt updated its cache. There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...


8

No. See the Google Robots.txt specifications, specifically the File Formats section. The expected file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF. Only valid records will be considered; all other content will be ignored. For example, if the resulting document is a HTML page, only valid ...


8

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located ...


8

There is no way to do it in robots.txt itself as served over HTTP. You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in your .htaccess file: RewriteEngine On RewriteCond %{HTTPS} =on RewriteRule ^robots.txt$ robots-deny-all.txt [L] Where robots-deny-all.txt has the contents: ...


7

For Google, in particular, the following rules will do the trick: User-Agent: * Allow: /$ Disallow: / For details, see Google's documentation of their supported robots.txt syntax. However, note that the middle line is non-standard for two reasons: first, it's an Allow directive (the basic robots.txt standard only supports Disallow) and second, it uses ...


7

In the root of the directory of the subdomain website, add a file called robots.txt containing: User-agent: * Disallow: / This will tell web crawlers not to index the site at all. They do not have to obey, but the main ones will.


7

Robots.txt just tells search engines not to crawl your pages. It does not tell them to not index your pages. So if your pages have links from other websites the search engines will know they exist. And because off-page factors affect rank, sometimes greatly, your pages can rank well for long-tail search terms without ever being crawled. To actually prevent ...


7

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


6

It doesn't matter. According to the spec: The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). CR = Carriage Return (i.e. \r) CR/NL = Carriage Return/New Line (i.e. \r\n) NL = New Line (i.e. \n) This means you can use \r. \n, or \r\n and it will work.



Only top voted, non community-wiki answers of a minimum length are eligible