Tag Info

Hot answers tagged

17

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful ...


13

This is Vincent from Bing Webmaster Tools. This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details: Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog You ...


11

It is impossible to configure logging of the #fragment portion of the URL because your web server never sees it. From RFC3986 the fragment identifier is separated from the rest of the URI prior to a dereference, and thus the identifying information within the fragment itself is dereferenced solely by the user agent, regardless of the URI scheme. ...


5

As long as your privacy policy properly disclose what you track, and you don't misuse the information, there shouldn't be any problem.


4

Regarding your second question: ::1 is localhost in ipv6. Regarding your third question: # Mark requests for the robots.txt file SetEnvIf Request_Method "^OPTIONS$" dontlog # Log what remains CustomLog logs/custom.log common env=!dontlog see also http://httpd.apache.org/docs/2.4/logs.html#accesslog


4

From here It's Apache polling its child processes to verify they're responding correctly. 31-2 - 0/0/44 . 0.00 41 0 0.0 0.00 0.92 ::1 mxx1.xx.com OPTIONS * HTTP/1.0 The second field, " - ", shows that this isn't an active connection.. It's the last connection that took place for this particular thread/process. Since Apache polls ...


4

Try adding something like the following into your VirtualHost config: SetEnvIf User-agent .*Googlebot.* googlebot CustomLog /var/log/apache2/access_googlebot.log combined env=googlebot CustomLog /var/log/apache2/access.log combined env=!googlebot


4

But what's the point of this type of activity? Your guess is as good as anyone's - perhaps the operator is mistakenly trying to access a set of spam accounts created elsewhere? (or ones which were supposed to be created at your site but failed for whatever reason) Edit: ... speaking to your initial question: How should a site respond to automated ...


4

Have a look at the following documentation on the Google help site for tracking file downloads: http://www.google.com/support/analytics/bin/answer.py?answer=55529 The only true way to analyze web traffic on your site is to look at the actual log files on your web server with something like WebTrends, SmarterStats, AWStats, etc.


4

It turns out the UserAgent is attached when using the web site monitoring / uptime provider Site Confidence: http://www.siteconfidence.com.au/


4

Completely agree that using a SQL query is not the most effective approach to this. However, if the logs you happen to need are in SQL already, and it's a one off or data sizes a not large enough to warrant improving the approach, here's a handy expression that can be used as a starting point... Based on Sarath's post here: ...


3

The transfer log is a log of requests, and even though there were no files (which is why there's a 404) they still get logged. It looks like a bot trying to find a vulnerability in your site using known loopholes.


3

Any type of logging software such as Awstats and Webalizer do not contain the information you are after, you should take a look at various Analytics platforms and see what one is best suited for your business. I personally recommend Google Analytics but there are many more out their to test. Please note that all links listed below and are in no particular ...


3

GET http://www.example.com/path/file.html HTTP/1.1 GET http://www.othersite.net/ HTTP/1.1 These are attempts to use your server as a HTTP proxy; that's what a full URL rather than a path in the request means. These are attempts to misuse your server as an open proxy, which it is properly rejecting. Giving the full URL while requesting a page from your own ...


3

It's always bothered me too that Google Analytics doesn't track file downloads out-of-the-box. One would think this is a common scenario... I had the same issue and Googled around. Learned two things. First, make sure you're using the newer Asynchronous Google Analytics. The asynchronous GA will make your website load faster. Finally, track file downloads ...


3

Google Analytics can't analyse your log files, for that you'd need the paid version called Urchin. What you can do though is track clicks/interactions on links to files. You can track clicks in two ways, as virtual page views or as events. Generally speaking, event tracking would be the recommended way to track downloads.


3

Two of the best opensource web analytics are OWA and Piwik OWA is supporting click-stream and the demo is here : http://demo.openwebanalytics.com/owa Piwik is offering a clickheat plugin demo : http://demo.piwik.org/ Worth mentionning: http://greensweater.wordpress.com/2006/05/01/clickstream-tracking-with-apache/ https://github.com/riivo/pwum


3

Is it definitely a bot? Do you have a user-agent? And is there a referer [sic] header for those requests? A Google search doesn't bring up anything remotely significant so I don't think it's related to robots.txt, nor anything like a bot polling for security flaws. It is likely to be something simple like a mistyped link to your site - for example you have ...


3

There is no "recommendation" for specifically the center column, that would be odd. Many sites nowadays stick to a maximum of 1000 pixels for the entire site width (for 1024x768 resolution screens). IMO wider than that is not all that useful anyway, since lines of text can before too long to read comfortably (like Wikipedia, which uses a fluid layout). The ...


3

As others have said, page anchors are something that the browser operates on rather than the web server. You could use some code to pass the anchor info on to your Analytics program. For example you could extract the location.hash value and pass it to Google Analytics as a custom variable. More usefully you could run code that checks the page html for ...


3

What is the reason behind the different numbers, even when the data is quite trivial like unique visitors and page loads? Google Analytics requires a Javascript callback to Google's domain and will not record traffic from users with Javascript disabled or the majority of bots. AWStats is literally reading your webserver's logs and will record (in one ...


3

I would be suspicious: 208.115.113.90 - - [09/Aug/2012:00:58:17 -0400] "GET /?p=unauthmanagement HTTP/1.0" 200 8998 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)" ‹ùU#Pmoo.tigertutorialscom First: GET /?p=unauthmanagement What is this? Does this page exist? "unauthorized management" could be a script that someone uploaded to do ...


3

MixPanel is reasonably priced, has an excellent feature set and has emailed reports.


3

Apache does provide best practice advice in the documentation. Apache Virtual Host documentation Apache > HTTP Server > Documentation > Version 2.4 On this link youll find a list of documentation pages which explain all details of virtual host support in Apache HTTP Server. https://httpd.apache.org/docs/2.4/vhosts/ --> ...


3

That kind of information is collected for website usability testing. There is a strong correlation between where the mouse moves and where users read (many users move the mouse near what they are reading). Knowing where the mouse goes and what users click on gives webmasters a good sense of what users find important and what most draws their attention on ...


2

Amazon recommends using AWStats and includes instructions on how to set up an AWStats configuration file. You will also need to set up some method to automate downloading the logs to where your AWStats installation is. Personally I use cron to set off a Python script that uses boto to grab the files. """s3logDL.py Downloads S3 logs and deletes old logs ...


2

As discussed in this question it's not monitor size that matters. It's browser size/resolution. Someone may have a very wide monitor but only keep their browser open to half its width. Your best bet is to design the middle column to grow and shrink depending on browser size. Using the min/max-width CSS properties should give it the constraints you're ...


2

On a *nix system you could use syslog-ng to store log messages on a dedicated log server for your load-balanced cluster(s) and then use a log analysis solution like Splunk to keep tabs on things - as for what the StackExchange sites actually run, may be a good question for StackOverflow Meta.


2

There is much information that web server logs contain which will never be available to Google Analytics, two things I can think of: Errors like 404, etc. Access on media files, like pictures, etc. (including external websites linking to your images) IP addresses, although an answer to another question pointed out that it can be set as a user variable Full ...


2

The best tool that i have used is Splunk(http://www.splunk.com/download/) It's not cheap, but real cool and give best speed and functionality that you ever need. Also they have freeware license for aggregation 500mb of data per day. I used this licence on my production game servers(about 3m users) and haven't any problem with this limit



Only top voted, non community-wiki answers of a minimum length are eligible