Tag Info

Hot answers tagged

15

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful ...


12

This is Vincent from Bing Webmaster Tools. This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details: Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog You ...


11

It is impossible to configure logging of the #fragment portion of the URL because your web server never sees it. From RFC3986 the fragment identifier is separated from the rest of the URI prior to a dereference, and thus the identifying information within the fragment itself is dereferenced solely by the user agent, regardless of the URI scheme. ...


4

Have a look at the following documentation on the Google help site for tracking file downloads: http://www.google.com/support/analytics/bin/answer.py?answer=55529 The only true way to analyze web traffic on your site is to look at the actual log files on your web server with something like WebTrends, SmarterStats, AWStats, etc.


4

Regarding your second question: ::1 is localhost in ipv6. Regarding your third question: # Mark requests for the robots.txt file SetEnvIf Request_Method "^OPTIONS$" dontlog # Log what remains CustomLog logs/custom.log common env=!dontlog see also http://httpd.apache.org/docs/2.4/logs.html#accesslog


4

From here It's Apache polling its child processes to verify they're responding correctly. 31-2 - 0/0/44 . 0.00 41 0 0.0 0.00 0.92 ::1 mxx1.xx.com OPTIONS * HTTP/1.0 The second field, " - ", shows that this isn't an active connection.. It's the last connection that took place for this particular thread/process. Since Apache polls ...


4

But what's the point of this type of activity? Your guess is as good as anyone's - perhaps the operator is mistakenly trying to access a set of spam accounts created elsewhere? (or ones which were supposed to be created at your site but failed for whatever reason) Edit: ... speaking to your initial question: How should a site respond to automated ...


3

Two of the best opensource web analytics are OWA and Piwik OWA is supporting click-stream and the demo is here : http://demo.openwebanalytics.com/owa Piwik is offering a clickheat plugin demo : http://demo.piwik.org/ Worth mentionning: http://greensweater.wordpress.com/2006/05/01/clickstream-tracking-with-apache/ https://github.com/riivo/pwum


3

Completely agree that using a SQL query is not the most effective approach to this. However, if the logs you happen to need are in SQL already, and it's a one off or data sizes a not large enough to warrant improving the approach, here's a handy expression that can be used as a starting point... Based on Sarath's post here: ...


3

Any type of logging software such as Awstats and Webalizer do not contain the information you are after, you should take a look at various Analytics platforms and see what one is best suited for your business. I personally recommend Google Analytics but there are many more out their to test. Please note that all links listed below and are in no particular ...


3

GET http://www.example.com/path/file.html HTTP/1.1 GET http://www.othersite.net/ HTTP/1.1 These are attempts to use your server as a HTTP proxy; that's what a full URL rather than a path in the request means. These are attempts to misuse your server as an open proxy, which it is properly rejecting. Giving the full URL while requesting a page from your own ...


3

It's always bothered me too that Google Analytics doesn't track file downloads out-of-the-box. One would think this is a common scenario... I had the same issue and Googled around. Learned two things. First, make sure you're using the newer Asynchronous Google Analytics. The asynchronous GA will make your website load faster. Finally, track file downloads ...


3

Google Analytics can't analyse your log files, for that you'd need the paid version called Urchin. What you can do though is track clicks/interactions on links to files. You can track clicks in two ways, as virtual page views or as events. Generally speaking, event tracking would be the recommended way to track downloads.


3

Is it definitely a bot? Do you have a user-agent? And is there a referer [sic] header for those requests? A Google search doesn't bring up anything remotely significant so I don't think it's related to robots.txt, nor anything like a bot polling for security flaws. It is likely to be something simple like a mistyped link to your site - for example you have ...


3

There is no "recommendation" for specifically the center column, that would be odd. Many sites nowadays stick to a maximum of 1000 pixels for the entire site width (for 1024x768 resolution screens). IMO wider than that is not all that useful anyway, since lines of text can before too long to read comfortably (like Wikipedia, which uses a fluid layout). The ...


3

As others have said, page anchors are something that the browser operates on rather than the web server. You could use some code to pass the anchor info on to your Analytics program. For example you could extract the location.hash value and pass it to Google Analytics as a custom variable. More usefully you could run code that checks the page html for ...


3

What is the reason behind the different numbers, even when the data is quite trivial like unique visitors and page loads? Google Analytics requires a Javascript callback to Google's domain and will not record traffic from users with Javascript disabled or the majority of bots. AWStats is literally reading your webserver's logs and will record (in one ...


3

I would be suspicious: 208.115.113.90 - - [09/Aug/2012:00:58:17 -0400] "GET /?p=unauthmanagement HTTP/1.0" 200 8998 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)" ‹ùU#Pmoo.tigertutorialscom First: GET /?p=unauthmanagement What is this? Does this page exist? "unauthorized management" could be a script that someone uploaded to do ...


3

That kind of information is collected for website usability testing. There is a strong correlation between where the mouse moves and where users read (many users move the mouse near what they are reading). Knowing where the mouse goes and what users click on gives webmasters a good sense of what users find important and what most draws their attention on ...


2

Amazon recommends using AWStats and includes instructions on how to set up an AWStats configuration file. You will also need to set up some method to automate downloading the logs to where your AWStats installation is. Personally I use cron to set off a Python script that uses boto to grab the files. """s3logDL.py Downloads S3 logs and deletes old logs ...


2

As discussed in this question it's not monitor size that matters. It's browser size/resolution. Someone may have a very wide monitor but only keep their browser open to half its width. Your best bet is to design the middle column to grow and shrink depending on browser size. Using the min/max-width CSS properties should give it the constraints you're ...


2

On a *nix system you could use syslog-ng to store log messages on a dedicated log server for your load-balanced cluster(s) and then use a log analysis solution like Splunk to keep tabs on things - as for what the StackExchange sites actually run, may be a good question for StackOverflow Meta.


2

There is much information that web server logs contain which will never be available to Google Analytics, two things I can think of: Errors like 404, etc. Access on media files, like pictures, etc. (including external websites linking to your images) IP addresses, although an answer to another question pointed out that it can be set as a user variable Full ...


2

The best tool that i have used is Splunk(http://www.splunk.com/download/) It's not cheap, but real cool and give best speed and functionality that you ever need. Also they have freeware license for aggregation 500mb of data per day. I used this licence on my production game servers(about 3m users) and haven't any problem with this limit


2

Using the programming functions in SQL is a pretty bad way to deal with this. A decent method would probably be to add an extra field to the table for the browser name (or ID from another table), and every time you store a row in the database, use your programming language there (e.g. PHP) to decide on the user agent and store that. Then you can do a select ...


2

"Bad" is a somewhat subjective and relative term. But if you want to adhere to best practices regarding respecting user privacy, then, in addition to KoKo's answer, you can simply track the data for the minimum amount of time needed to do what you need to do, and then discard or anonymize it afterwards. This is what Google and most other major online ...


2

Splunk can import Apache logs http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Apachelogslocal Or review this answer on Stack Overflow A relatively recent option is to use Flume to collect the logs and use the MongoDB sink plugin for Flume to write the events to MongoDB.


2

It may have to do with how XAMP is configured with your local server and how routing is setup to send web traffic to your server. It's actually common to see bots scanning your server for vulnerabilities and common scripts such as phpMyAdmin. On linux I would add them to hosts.deny, in apache you can ban them with your .htaccess file. If the attacks are ...


2

According to this knowledge base article, MatrixStats is owned by FastHosts.co.uk and is available only to their customers. You could use Google Analytics, or try one of the following programs: IIS Log Analyzer Weblog Expert Easy AWstats for IIS instructions


2

Apache does provide best practice advice in the documentation. Apache Virtual Host documentation Apache > HTTP Server > Documentation > Version 2.4 On this link youll find a list of documentation pages which explain all details of virtual host support in Apache HTTP Server. https://httpd.apache.org/docs/2.4/vhosts/ --> ...



Only top voted, non community-wiki answers of a minimum length are eligible