2
votes
2answers
71 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
0
votes
1answer
24 views

Remove page from Search Result Bunch [duplicate]

When I do a branded search for a website the structure shows like ( see image ). I would love to remove Privacy Policy page or Manipulate the structure I see now ( showing pages I want to ). How I can ...
1
vote
4answers
155 views

Does Google ignore robots.txt

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
2
votes
3answers
91 views

Is it safe to block redirected (but still linked) URLs with robots.txt?

I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout the site the unclean URLs are linked in menus, content, products, etc. ...
1
vote
3answers
99 views

robots.txt file Question

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google
-1
votes
1answer
213 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...
3
votes
1answer
534 views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...
0
votes
0answers
152 views

Strange Google crawler 404s on Magento store involving “image” and “id” in URL

We're hit with over 10,000 404s on Google Webmaster all of a sudden. We've been getting them a few now and then, but not even close to this magnitude. Upon examining the items in this list of errors, ...
1
vote
1answer
75 views

Best way to prevent Google from indexing a directory [duplicate]

I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've seen are: Adding it into the robots.txt file: Disallow: ...
0
votes
1answer
65 views

Google indexed page a day before also reflecting in search but today everything vanish

We had robots.txt which disallow all robots as we were in development. We are live now. We change robots.txt as per our requirement a day before. Submit indexes using Google Webmaster Tools index ...
1
vote
3answers
211 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
0
votes
1answer
74 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
91 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
245 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
1
vote
3answers
2k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
5
votes
3answers
752 views

Google indexing page with parameters but page is Disallowed in robots.txt

I have the following in robots.txt: User-agent: * Disallow: /refer.php User-agent: NinjaBot Allow: / Sitemap: http://www.mysite.com/sitemap.xml The refer.php file does various things depending on ...
3
votes
1answer
105 views

Handling SEO for Infinite pages that cause external slow API calls

I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
0
votes
1answer
341 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...
2
votes
1answer
214 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
7
votes
4answers
11k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
0
votes
1answer
117 views

Hiding a particulat page from search engines not to index [duplicate]

Possible Duplicate: Blocking Search Engines to Crawl Parts of a Website? I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
0
votes
1answer
339 views

Is “robots.txt” necessarily needed? [duplicate]

Possible Duplicate: Do I need robots.txt? From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
1
vote
2answers
380 views

How do I have Google not index any of my site, except my domain?

Disallow: / I want Google to index the index page, but nothing else. Will this achieve it?
0
votes
2answers
2k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
0
votes
1answer
460 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
1
vote
2answers
86 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
2
votes
2answers
129 views

Pointing to Google that a specific website is for mobiles

I have 2 websites: let's call them A and B. A is a Desktop optimized website. B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A How ...
1
vote
1answer
683 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
3
votes
1answer
666 views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
1answer
76 views

Search and Tags for robots.txt

After setting up a blog at blogspot.com I added the site to my webmasters tools of google. I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
3
votes
6answers
148 views

Stopping Google index some web pages I have

I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?
2
votes
3answers
276 views

After adding *.xls to robots.txt how long will the results continue to show up in google

Is there an estimated amount of time before the document will not appear in google results? We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
3
votes
3answers
438 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...
10
votes
3answers
435 views

Does Google Preview obey Robots.txt?

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky. Is this the case and is there a way to ...
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
3
votes
3answers
6k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
10
votes
3answers
3k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
4
votes
3answers
602 views

600 visitors per day, 20 backlinks but still not referenced by Google

i've launch a website on wednesday (9 August) There's already in 4 days 12,000 viewed pages 1,400 visits 20 to 25 backlincks a sitemap.xml (130 URL) english language / french language - url like ...