Tag Info

New answers tagged

2

One can spoof a referrer relatively easily in order to hide scans, harvesting, crawls, etc. Makes sense that a spoof would appear as an innocent search -- ie google, yahoo, bing, etc. Generally they hit the link once, so it may be 20 crawls. Also, they can sometimes spoof for advertising reasons. Think if you clicked a couple of those weird referrers, thats ...


1

You might point out that if you were actually using some illicit method of traffic generation it would be an extremely inefficient means of doing so with little benefit to you. Twenty clicks is only 0.15%, i.e., a tiny fraction of a percent, of 13,000, so there would be little incentive for you to do that. For the ones that point to specific webpages rather ...


0

If you can specify css in the wrapper do it there, if not, inline seems a good option, it works. <div style="font-size:150%"></div>


1

I ended up using iptables to block the ips. From the root linux shell I typed iptables -A INPUT -s 217.69.133.13 -j DROP iptables -A INPUT -s 217.69.133.12 -j DROP iptables -A INPUT -s 217.69.133.10 -j DROP iptables -A INPUT -s 217.69.133.11 -j DROP iptables -A INPUT -s 217.69.133.14 -j DROP iptables -A INPUT -s 217.69.133.15 -j DROP iptables ...


2

You are okay. Some may argue, but this is not a bad bot. I research these things as part of my research and while I do have a lot of activity from mail.ru, I do not have any bad bot activity doing a quick search. Mail.ru operates a search engine. This is a web crawler for their search engine. The page referred to in the agent string is ...


1

Old question, but since it was asked a new project has emerged called Doctored which appears to meet the requirements, example is here: http://holloway.github.io/doctored/ The project is open source and allows the end user to not only edit xml but also validates it against a schema. Disclaimer: I am not the author of the above project.


1

When you have the same content available at many different URLs, you can use URL canonicalization to tell which set of URLs are the preferred ones for search engines to crawl and index. Google introduced a link rel canonical meta tag for this purpose. Lets say that you want to make 500 photos per page the canonical URL. On the other URLs ...


1

Yes use HTTPS everywhere, add a HSTS policy, use only TLS, etc. HTTPS already is "the standard" and it's plenty fast enough with a proper server setup...it's just that most folks don't take the time or wanna spend the money to use it. If you consider time to encode a drawback, consider the results of the encoding (good encryption) as a drawback too. I ...


1

Blocking time is basically a "buffer" in browsers. Upon startup, especially, Chrome blocks most connections to decrease loading time. Eventually, the blocking time is completely removed after browsing many pages because it become impractical with Chrome fully "warmed up" on your computer. The blocking time settings are reset upon Chrome restart. The ...



Top 50 recent answers are included