3

We have a system hosting many websites for our customers, and inside that system there is a method that non-live customers can view their sites before we turn them on. Say the link is something like this: ourbigcompany.com/customer/domain=thisisanewsiteurl

Those links are not linked to anywhere outside a secure login - they are only sent to the customer via email. They are publicly viewable, as they have to be, but that's not the real problem. The real problem is that somehow Bing is getting hold of them and trying to crawl the sites. I know how to stop the crawling, but that would be like treating the symptoms without fixing the problem.

We log the traffic and there is no referrer - so that is not helpful.

If I change the querystring value for a site, Bing has it within hours. I need to figure out where Bing is getting the links from so that I can close what is obviously a security hole, but I am not sure how. Any ideas on how to figure that out?

Improve this question
5
  • 1
    More info: The IP does resolve to Microsoft, so we are certain it is Bing. Bing Webmaster Tools does not report anything linking to the page in question, though they are clearly finding it somehow
    – Jen R
    Mar 4, 2015 at 21:50
  • 1
    Someone may have posted a link to it somewhere public. If there's test sites, someone may have even just attempted Facebook Liking or Tweeting one of the pages as a test, the bots would find that.
    – nathangiesbrecht
    Mar 4, 2015 at 21:54
  • 1
    Are they being displayed in a sitemap somewhere? If its a WP site then there could be several XML SMs which are public. The main thing to worry about is your disallow rule in the robots.txt rather than how the crawlers is actually getting there.
    – Ben Racicot
    Mar 4, 2015 at 23:20
  • @JenR As nathangiesbrecht points out, you can't really control how URLs might get shared (e.g., browsers can also track them via toolbars and extensions when clicked on in emails). The best you can do is to make sure they're not crawlable as Ben Racicot indicated.
    – dan ♦
    Mar 5, 2015 at 1:49
  • See also: Can secret URLs be used to protect files?
    – Stephen Ostermiller ♦
    Mar 5, 2015 at 15:26

1 Answer 1

Reset to default
1

You won't be able to know for sure how search engines got the URL. They don't tell you that information. There are several possible ways that it could have happened:

  • The user shares or publishes the link themselves
  • The site has a link to another site. When that link is clicked, the secret URL is sent as a referrer. Some sites publish referrer URLs in places that search engines can find them.
  • Some browsers send information about every page you visit directly to the companies that run search engines. Google at least says they do not rely on any sent data to feed their crawler. Some browser features that rely on this are:
    • Safe browsing features that flag malware pages as you surf
    • Pagerank indicator toolbars
  • Usage of social buttons on the page such as Google +1 buttons
  • Usage of analytics software
  • Inclusion of advertisements on the site
  • Any 3rd party JavaScript, CSS, or image usage
  • The email you send with a link traverses through an email server owned by the search engine (Gmail, Hotmail). Links in such an email could be harvested for crawling.

As Google says:

It's almost impossible to keep a web server secret by not publishing links to it. As soon as someone follows a link from your "secret" server to another web server, your "secret" URL may appear in the referrer tag and can be stored and published by the other web server in its referrer log...

If you want to prevent Googlebot from crawling content on your site, you have a number of options, including using robots.txt to block access to files and directories on your server.

Improve this answer
5
  • I am guessing that at least one of these is true. I know that there has been no other (non-Bing) traffic to the specific site with domain=thisisanewsiteurl - however the sites do have external analytics software, share buttons, and some 3rd party images.
    – Jen R
    Mar 5, 2015 at 16:08
  • Un-accepted the answer... because someone would have to actually view the site in order to cause the 3rd party code, etc to report a referrer. :-(
    – Jen R
    Mar 5, 2015 at 16:11
  • I added one more possibility regarding email.
    – Stephen Ostermiller ♦
    Mar 5, 2015 at 16:13
  • OK, I'm still not confident, but I think we have figured it out. It has to be a combination of things... (1) the reasons in this answer give the search engine access to the original (poorly) hidden URL of ourbigcompany.com/customer/domain=thisisanewsiteurl , (2) We have code in place so that if a domain name is changed, visitors to thisisanewsiteurl.com would get a 301 redirect to evennewersiteurl.com (3) so the search engine gets hold of ourbigcompany.com/customer/domain=thisisanewsiteurl , and gets redirected to ourbigcompany.com/customer/domain=evennewersiteurl
    – Jen R
    Mar 5, 2015 at 16:29
  • To follow up, we will change our auto-redirect code to only redirect in the case of a "live" url (thisisanewsiteurl.com) and not our "test" URLs (ourbigcompany.com/customer/domain=thisisanewsiteurl). That should at least reduce the chances of the links getting out there. We'll also use the usual ways to prevent indexing on the test URLs (although that won't stop the "bad spiders", it should stop Bing). There is nothing really secret about what is on the sites, so there is no need for further security; we just wanted to understand how the links were being discovered so quickly.
    – Jen R
    Mar 5, 2015 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.