Tag Info

Hot answers tagged

35

This is not uncommon as there is no time frame for Google to index a website. The best thing you can do is to give Google every opportunity to find you. Try all of the following: To submit your site to Google go here. Submit an XML sitemap. You can do that from within your Webmaster Control panel. Here's a drupal add on to make this easy for you. Get ...


15

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful ...


13

There are 2 main ways to prevent search engines from indexing specific pages: A Robots.txt file for your domain. The Meta Robots tag on each page. Robots.txt should be your first stop for URL patterns that match several files. You can see the syntax here and more detailed here. The robots.txt file must be placed in the root folder of your domain, i.e. at ...


13

I would strongly recommend registering your site with webmaster tools. There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where you are appearing in queries on Google. ...


11

You don't want the page to appear in the SERPs at all... Don't disallow in robots.txt. Add a noindex meta tag (or X-Robots-Tag HTTP header) to your pages instead. As j0k suggests, your pages could be found somehow. Stats reports, directory listings, etc... Disallowing in robots.txt prevents the page from being crawled, but could still be indexed and could ...


10

Hard to say for sure why you aren't indexed yet but: 1) Number of visitors has no bearing whatsoever on your indexing or rankings. Google doesn't know this information and, even if it they did, it really offers nothing in terms of relevance of any page for search. 2) 20 backlinks is hardly a lot. Even then, unless Google knows about those backlinks they ...


10

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


9

Maybe someone didn't want to pay for spider traffic? Regardless, you are reading it correctly: http://www.robotstxt.org/robotstxt.html Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say ...


9

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed. User-agent: * Disallow: /templates_c But to answer your question, according to Google: At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the ...


8

It seems that Google deliberately includes URLs disallowed in robots.txt in their index if there are links to those URLs from other pages they've crawled. To quote their Webmaster Tools help pages: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a ...


8

No. See the Google Robots.txt specifications, specifically the File Formats section. The expected file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF. Only valid records will be considered; all other content will be ignored. For example, if the resulting document is a HTML page, only valid ...


8

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


7

You can't make them re-download your robots.txt when you want them to. Google will re-crawl it and use the new data whenever they feel it is appropriate for your site. They tend to crawl it regularly so I wouldn't expect it to take long for your updated file to be found and your pages re-crawled and re-indexed. Keep in mind that it may take some time after ...


7

For Google, in particular, the following rules will do the trick: User-Agent: * Allow: /$ Disallow: / For details, see Google's documentation of their supported robots.txt syntax. However, note that the middle line is non-standard for two reasons: first, it's an Allow directive (the basic robots.txt standard only supports Disallow) and second, it uses ...


7

It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL. I guess google just hasnt updated its cache. There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...


7

Robots.txt just tells search engines not to crawl your pages. It does not tell them to not index your pages. So if your pages have links from other websites the search engines will know they exist. And because off-page factors affect rank, sometimes greatly, your pages can rank well for long-tail search terms without ever being crawled. To actually prevent ...


6

It doesn't matter. According to the spec: The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). CR = Carriage Return (i.e. \r) CR/NL = Carriage Return/New Line (i.e. \r\n) NL = New Line (i.e. \n) This means you can use \r. \n, or \r\n and it will work.


6

Google periodically scan web. Might be your site is not scanned till now. Checklist You have proper robots.txt Some of posts at your site are linking to other posts of your site. sitemap.xml is placed on proper path. And its entry is mentioned in robots.txt. You had successfully submitted your site sitemap to google and other web search engines. I'll ...


6

Remeber that no one from internet can see your directory tree, so stackoverflow.com/ and test.stackoverflow.com are completely diffrent sites for us and search robots. You can do that by checing http host in query. RewriteCond %{HTTP_HOST} ^test RewriteRule robots.txt someotherrobots.txt


6

This is a bad idea. This practice, of showing one thing to search engines and another to users, is called cloaking: http://www.google.com/support/webmasters/bin/answer.py?answer=66355 It will cause your site more harm than good, with respect to search engine results; it "may cause your site to be perceived as deceptive and removed from the Google index", ...


6

You can see what the file contains here: http://cdn.attracta.com/sitemap/728687/0.xml.gz If that doesn't reflect your sites structure I would delete it. However see answer 2 - you may be able to update it to make it accurate. Probably. An Attracta Starter Account ($120/year value) is bundled and integrated into the HostMonster Control Panel of ...


6

Yes, you can have a sitemap index containing references to other sitemap indexes [source], as long as they're all on the same domain. Each individual sitemap index can include up to 1,000 references to another <sitemap>. If you need to reference more than that, create multiple sitemap indexes and submit each one. Each individual sitemap can include ...


6

Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them. This will depend a bit on the exact implementation of the crawlers robot.txt ...


6

The robots.txt spec says prefix matching is used. This means if you don't want the excluded URLs to be visible in robots.txt, you can simply abbreviate them just enough to not match any allowed URLs. For example, if you want to disallow /very/secret, you could simply use: Disallow: /ve In your robots.txt file, and as long as you don't have other URLs ...


5

I'll have to go looking for the source of this information but apparently robots.txt will not necessarily prevent a page from being indexed. But the HTTP x-robots-tag header does apparently work. If you're using Apache you can block pages in bulk using this line in an .htaccess file: Header set x-robots-tag: noindex Give that a try and see what happens. ...


5

There are various theories as to how Google knows what to crawl. It could be that someone linked to your mobile version. It could be that Google tried random urls and came across the /m version of your site. I'm not aware that they say they won't use URLs from their analytics data. Yes they do follow those rules: ...


5

robots.txt only specifies how Google should or should not crawl your site and as Mike said if you solely rely on this vector it will take a long time to have the pages removed from the index. So you will also need to make a removal request if you want it to happen faster and have Google remove the pages from the index. When you are in Webmaster Tools go to ...



Only top voted, non community-wiki answers of a minimum length are eligible