4
votes
5answers
140 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
1
vote
1answer
39 views

My site's robots.txt was blocking everyone for a year. What to do now that I fixed that?

My site's robots.txt was blocking everyone for a year (The "Discourage search engines from indexing this site" option was checked in WordPress). I guess Google will re-try to crawl it in the upcoming ...
1
vote
3answers
55 views

Google is not crawling and indexing my site after updating my robots.txt file

I have an issue with Google not being able to properly crawl my site. I have read other questions where people have had the same issue. I've tried to follow their solution of using this in my ...
1
vote
3answers
52 views

How do I remove a LOT of indexed pages from Google?

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
5
votes
4answers
294 views

Should I set NOINDEX header for my JS, CSS and image files?

Are there any harms if my site send NOINDEX headers for all my static assets? For image files, I refer to those valueless, e.g. background images, button images, etc. Update: more background ...
2
votes
2answers
81 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
2
votes
1answer
53 views

Robots and Sitemaps

I have some problems regarding indexing. Google indexed so much in the list. Now I wanted to remove everything from my site. If I have disallow all in robots.txt and send my sitemap.xml. Would ...
0
votes
1answer
40 views

Hiding page content from Google when it is duplicated across several sites

I would like to hide some content from Google. Because a client has multiple websites with the same content then he gets a warning about duplicate content on some of them (multiple countries, same ...
3
votes
2answers
321 views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
2
votes
2answers
104 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
0
votes
1answer
36 views

Remove page from Search Result Bunch [duplicate]

When I do a branded search for a website the structure shows like ( see image ). I would love to remove Privacy Policy page or Manipulate the structure I see now ( showing pages I want to ). How I can ...
1
vote
4answers
306 views

Does Google ignore robots.txt

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
2
votes
3answers
105 views

Is it safe to block redirected (but still linked) URLs with robots.txt?

I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout the site the unclean URLs are linked in menus, content, products, etc. ...
1
vote
3answers
103 views

robots.txt file Question

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google
-1
votes
1answer
228 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...
3
votes
1answer
762 views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...
1
vote
1answer
79 views

Best way to prevent Google from indexing a directory [duplicate]

I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've seen are: Adding it into the robots.txt file: Disallow: ...
0
votes
1answer
66 views

Google indexed page a day before also reflecting in search but today everything vanish

We had robots.txt which disallow all robots as we were in development. We are live now. We change robots.txt as per our requirement a day before. Submit indexes using Google Webmaster Tools index ...
1
vote
3answers
274 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
0
votes
1answer
76 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
103 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
257 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
1
vote
3answers
3k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
5
votes
3answers
880 views

Google indexing page with parameters but page is Disallowed in robots.txt

I have the following in robots.txt: User-agent: * Disallow: /refer.php User-agent: NinjaBot Allow: / Sitemap: http://www.mysite.com/sitemap.xml The refer.php file does various things depending on ...
5
votes
1answer
109 views

Handling SEO for Infinite pages that cause external slow API calls

I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
0
votes
1answer
354 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...
2
votes
1answer
229 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
7
votes
4answers
13k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
0
votes
1answer
119 views

Hiding a particulat page from search engines not to index [duplicate]

Possible Duplicate: Blocking Search Engines to Crawl Parts of a Website? I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
0
votes
1answer
357 views

Is “robots.txt” necessarily needed? [duplicate]

Possible Duplicate: Do I need robots.txt? From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
1
vote
2answers
420 views

How do I have Google not index any of my site, except my domain?

Disallow: / I want Google to index the index page, but nothing else. Will this achieve it?
0
votes
2answers
2k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
0
votes
1answer
478 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
1
vote
2answers
96 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
2
votes
2answers
133 views

Pointing to Google that a specific website is for mobiles

I have 2 websites: let's call them A and B. A is a Desktop optimized website. B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A How ...
1
vote
1answer
736 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
3
votes
1answer
827 views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
1answer
77 views

Search and Tags for robots.txt

After setting up a blog at blogspot.com I added the site to my webmasters tools of google. I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
3
votes
6answers
149 views

Stopping Google index some web pages I have

I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?
2
votes
3answers
277 views

After adding *.xls to robots.txt how long will the results continue to show up in google

Is there an estimated amount of time before the document will not appear in google results? We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
3
votes
3answers
461 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...
10
votes
3answers
442 views

Does Google Preview obey Robots.txt?

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky. Is this the case and is there a way to ...
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
3
votes
3answers
7k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
11
votes
5answers
4k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
4
votes
3answers
614 views

600 visitors per day, 20 backlinks but still not referenced by Google

i've launch a website on wednesday (9 August) There's already in 4 days 12,000 viewed pages 1,400 visits 20 to 25 backlincks a sitemap.xml (130 URL) english language / french language - url like ...