5

I've gotten some visits to pages that are supposed to be completely hidden from everyone via Yandex and I'd love to find out:

  1. what search term led them to it
  2. the referring page on my site

The problem is that in spite spending over an hour trying to utilize the vars in the URL, I can't get a legitimate search page on Yandex. Here's the URL in question:

http://yandex.ru/clck/jsredir?from=yandex.ru%3Bsearch%3Bweb%3B%3B&text=&etext=1271.RJS9ZfLhVdj6nXam87qy4e0e-DG9BQd_KlyA1gFVBu1uuZOuUSRTgOEasX71Cupm.fe839c38b17c539463c0b2f7d01d86940f4b3320&uuid=&state=_BLhILn4SxNIvvL0W45KSic66uCIg23qh8iRG98qeIXmeppkgUc0YL_nDC5hqtEQ6WayFoZKRZE&data=UlNrNmk5WktYejY4cHFySjRXSWhXUFJiWDhna1NqZnBmd1YzNG43VS13RUpmdUZXdnBLOHdkMFlqUzVDamF1OVBVb2xkMmtvMUxXWUxJM1hSVW5hS2x5R1R6LVpCcGVXZFZZNkprR0JOSUVPc3d0ZnBVOXpDV295ckZDdFpqS3l4WkZSOFF3c0RmVTN2ZkhIYWIwT0JzNVQyWko5ME9vMw&b64e=2&sign=08505d8afebc7cb1b4568d3e92c11ecb&keyno=0&cst=AiuY0DBWFJ7IXge4WdYJQXbYQp9t5VF6sf_IfF4r6pdt0ojCe4cFQNegojWnJn8UToJJyLyR96RrC_bl9mqJxfCjbo3nl3EPqUjNd2ADc0Zxar8tKC1hQd4R3WTMI1AD3dVkg_IhwheNgkWXjuLnig&ref=orjY4mGPRjk5boDnW0uvlrrd71vZw9kp5uQozpMtKCXdCnh-_wii4V8gT36dWFhYdLgT8HVc5IPL1yluhUPYHlzmn9nr8Aaa3y8eC13fJRd5RgTTAPeGmg&l10n=ru&cts=1481853806438&mc=4.32492874929

I took the code after the redir URL and tried adding the var and value etext on to the Yandex search URL but nothing has worked so far.

I've already downloaded the entire site via wget while using both a browser and Yandex search UA (it's how my site distinguishes bots to hide logins and human-specific content). Doing a search through all the downloaded content, I was unable to find any instance of the URLs in question. The sitemap has no mention of the URLs in question.

  • Cross post: webapps.stackexchange.com/questions/101214/… – MrWhite Dec 16 '16 at 22:09
  • Exactly the same problem here. Did you share the link on any non-public channels like E-Mail, Telegram, Skype? I think some of the channels is leaking the link... I can't explain how yandex got the link otherwise. – Aykut Çevik Dec 27 '16 at 10:03
  • These links wouldn't have been shared via any means. For instance, one of the links is part of a chain of links visited by me to ban an IP so it seems they're just visiting random links found in a pool of available lnks. – Schwim Dandy Dec 28 '16 at 13:00
1

Trying to check the search term won't work to identify the source of the link to begin with. Search engines identify pages to crawl using other pages that have already been crawled and the links on those pages. In other words if you have a link on your page which you click on to go to the protected page then the crawler will know about it, this also works if the link is added to the page using javascript. Most legitimate crawlers will respect the robots.txt specification to exclude pages which you don't want crawlers to access but in case of bad crawlers that don't respect the robots.txt exclusion restrict access to these links behind a login form. Quickest and easiest way is with .htaccess and .htpasswd to require authentication for the admin pages and put them into a restricted access directory.

  • Thanks very much for your post on the matter. In the month since I made this plea for assistance, I've managed to handle the issue via other means. – Schwim Dandy Jan 24 '17 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.