0

I have a proxy at http://rahul2001.com/proxy/

I site can be 'proxied' by like this: http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.example.com/

The problem is that Google seems to be crawling pages through my proxy :/

uh oh..

I do NOT link a proxied version of Yahoo food or Google Careers or Google Search Hindi Help, but these still turn up in the search results...

PROBLEMS:

-I DO NOT want to block my website/proxy from search engines entirely, I still want the proxy service itself to be able to be located in search.

-I DO NOT want Google to use up my bandwidth by crawling useless sites.

-I DO NOT want to use captcha since a few of my apps use this proxy.

-I DO NOT want Google to spoil the search results of my website in this manner.

What do I do??

ALSO, why is Google entering random URLs into the form??

EDIT After adding the meta tag, I get an error :(

proxy.php (first few lines):

<head><meta name="robots" content="noindex, nofollow" />
</head>
<?php
/*
miniProxy - A simple PHP web proxy. <https://github.com/joshdick/miniProxy>
Written and maintained by Joshua Dick <http://joshdick.net>.
miniProxy is licensed under the GNU GPL v3 <http://www.gnu.org/licenses/gpl.html>.
*/

Error:

Warning: Cannot modify header information - headers already sent by (output started at /home/rahulcom/public_html/proxy/proxy.php:3) in /home/rahulcom/public_html/proxy/proxy.php

HELL BREAKS LOOSE - GOOGLE IS CRAWLING THE INTERNET USING MY PROXY!!

https://www.google.co.in/search?q=site:rahul2001.com+proxy

Improve this question
9
  • I realize that I want rather a lot...
    – undo
    Commented Jan 5, 2016 at 6:40
  • You should add meta tag <meta name="robots" content="noindex, nofollow" /> on these pages that you do not want search engines to index or list in SERP.
    – Robert hue
    Commented Jan 5, 2016 at 6:57
  • @Roberthue Read point 1 : I DO NOT want to block my website/proxy from search engines....
    – undo
    Commented Jan 5, 2016 at 7:01
  • 1
    You should add this in proxy.php. So all links with proxy.php will not be indexed or followed. It will not block Google from indexing other part of your website but only links that have proxy.php.
    – Robert hue
    Commented Jan 5, 2016 at 7:02
  • @Roberthue HELL... It breaks the proxy for some reason..
    – undo
    Commented Jan 5, 2016 at 7:15

2 Answers 2

Reset to default
1

Add this in robots.txt (Just places this file in root folder, mostly under public_html or where your home page sits)

User-agent: *

Disallow: /proxy/*

Improve this answer
5
  • BUT... I want people to find my proxy...!
    – undo
    Commented Jan 5, 2016 at 7:00
  • robots.txt does not block search engine completely. Google will still list unwanted links in SERPs.
    – Robert hue
    Commented Jan 5, 2016 at 7:36
  • I think, It will solve your purpose.
    – Mani
    Commented Jan 5, 2016 at 7:48
  • The * in Disallow: /proxy/* is not needed. The rule means the same thing without it and will be understood by more robots. Only some robots understand patterns with wild cards.
    – Stephen Ostermiller ♦
    Commented Jan 5, 2016 at 15:00
  • I would actually suggest Disallow: /proxy/proxy.php because that would still allow the proxy home page (/proxy/) but disallow all the proxied pages themselves.
    – Stephen Ostermiller ♦
    Commented Jan 5, 2016 at 15:01
1

OK, so, you do not want to block the proxy from search engines, but you don't want the result to show up on search engines? Sorry, I don't get it :) Is it that you want the original site to show up on top on the proxies? Google decides which pages are more important I'm afraid.

Also be careful with duplicate content. The same information should only be on one URL. For duplicates you should use canonical links (http://speckyboy.com/2012/07/16/what-a-canonical-link-is-and-how-to-use-it-properly/) to tell Google which of the page is the original one. This is probably the one that will show up on Google.

There is not need for two URLs to show up on Google with the same information in it?

EDIT

From comments i was lead to believe that you want to show /proxy/ but not anything under this page, like /proxy/subpage. In this case, use robots.txt like this:

User-agent: *

Disallow: /proxy/

Allow: /proxy/$

Improve this answer
4
  • Actually he wants Google to index, crawl and list his website in SERPs but not results.
    – Robert hue
    Commented Jan 5, 2016 at 7:38
  • The R in SERP is for Result :) Explain.
    – Tony Gustafsson
    Commented Jan 5, 2016 at 7:54
  • I mean he wants Google to index http://rahul2001.com/proxy/ and other web pages but he does not want Google to index and list pages with URLs like this http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.com/
    – Robert hue
    Commented Jan 5, 2016 at 7:57
  • For example if you have a website http://www.example.com/ and want Google to index all website except this page. http://www.example.com/about/
    – Robert hue
    Commented Jan 5, 2016 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.