0

I have a proxy at http://rahul2001.com/proxy/

I site can be 'proxied' by like this: http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.example.com/

The problem is that Google seems to be crawling pages through my proxy :/

uh oh..

I do NOT link a proxied version of Yahoo food or Google Careers or Google Search Hindi Help, but these still turn up in the search results...

PROBLEMS:

-I DO NOT want to block my website/proxy from search engines entirely, I still want the proxy service itself to be able to be located in search.

-I DO NOT want Google to use up my bandwidth by crawling useless sites.

-I DO NOT want to use captcha since a few of my apps use this proxy.

-I DO NOT want Google to spoil the search results of my website in this manner.

What do I do??

ALSO, why is Google entering random URLs into the form??

EDIT After adding the meta tag, I get an error :(

proxy.php (first few lines):

<head><meta name="robots" content="noindex, nofollow" />
</head>
<?php
/*
miniProxy - A simple PHP web proxy. <https://github.com/joshdick/miniProxy>
Written and maintained by Joshua Dick <http://joshdick.net>.
miniProxy is licensed under the GNU GPL v3 <http://www.gnu.org/licenses/gpl.html>.
*/

Error:

Warning: Cannot modify header information - headers already sent by (output started at /home/rahulcom/public_html/proxy/proxy.php:3) in /home/rahulcom/public_html/proxy/proxy.php

HELL BREAKS LOOSE - GOOGLE IS CRAWLING THE INTERNET USING MY PROXY!!

https://www.google.co.in/search?q=site:rahul2001.com+proxy

  • I realize that I want rather a lot... – rahuldottech Jan 5 '16 at 6:40
  • You should add meta tag <meta name="robots" content="noindex, nofollow" /> on these pages that you do not want search engines to index or list in SERP. – Robert hue Jan 5 '16 at 6:57
  • @Roberthue Read point 1 : I DO NOT want to block my website/proxy from search engines.... – rahuldottech Jan 5 '16 at 7:01
  • 1
    You should add this in proxy.php. So all links with proxy.php will not be indexed or followed. It will not block Google from indexing other part of your website but only links that have proxy.php. – Robert hue Jan 5 '16 at 7:02
  • @Roberthue HELL... It breaks the proxy for some reason.. – rahuldottech Jan 5 '16 at 7:15
1

Add this in robots.txt (Just places this file in root folder, mostly under public_html or where your home page sits)

User-agent: *

Disallow: /proxy/*

  • BUT... I want people to find my proxy...! – rahuldottech Jan 5 '16 at 7:00
  • robots.txt does not block search engine completely. Google will still list unwanted links in SERPs. – Robert hue Jan 5 '16 at 7:36
  • I think, It will solve your purpose. – Mani Jan 5 '16 at 7:48
  • The * in Disallow: /proxy/* is not needed. The rule means the same thing without it and will be understood by more robots. Only some robots understand patterns with wild cards. – Stephen Ostermiller Jan 5 '16 at 15:00
  • I would actually suggest Disallow: /proxy/proxy.php because that would still allow the proxy home page (/proxy/) but disallow all the proxied pages themselves. – Stephen Ostermiller Jan 5 '16 at 15:01
1

OK, so, you do not want to block the proxy from search engines, but you don't want the result to show up on search engines? Sorry, I don't get it :) Is it that you want the original site to show up on top on the proxies? Google decides which pages are more important I'm afraid.

Also be careful with duplicate content. The same information should only be on one URL. For duplicates you should use canonical links (http://speckyboy.com/2012/07/16/what-a-canonical-link-is-and-how-to-use-it-properly/) to tell Google which of the page is the original one. This is probably the one that will show up on Google.

There is not need for two URLs to show up on Google with the same information in it?

EDIT

From comments i was lead to believe that you want to show /proxy/ but not anything under this page, like /proxy/subpage. In this case, use robots.txt like this:

User-agent: *

Disallow: /proxy/

Allow: /proxy/$

  • Actually he wants Google to index, crawl and list his website in SERPs but not results. – Robert hue Jan 5 '16 at 7:38
  • The R in SERP is for Result :) Explain. – Tony Gustafsson Jan 5 '16 at 7:54
  • I mean he wants Google to index http://rahul2001.com/proxy/ and other web pages but he does not want Google to index and list pages with URLs like this http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.com/ – Robert hue Jan 5 '16 at 7:57
  • For example if you have a website http://www.example.com/ and want Google to index all website except this page. http://www.example.com/about/ – Robert hue Jan 5 '16 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.