I m doing some work to analyse the access logs from a Catalyst web application. The data is from the load balancers in front of the web farm and totals about 35Gb per day. It s stored in a Hadoop HDFS filesystem and I use MapReduce (via Dumbo, which is great) to crunch the numbers.
The purpose of the analysis is try to establish a usage profile -- which actions are used most, what the average response time for each action is, whether the response was served from a backend or cache -- for capacity planning, optimisation and to set thresholds for monitoring systems. Traditional tools like Analog will give me the most-requested URL or most-used browser but none of that s useful for me. I don t need to know that /controller/foo?id=1984 is the most popular URL; I need to know what hit rate and response time for all hits to /controller/foo is so I can see if there s room for optimisation or caching and try to estimate what might happen if hits for this action suddenly double.
I can easily break the data down into requests per action per period via MapReduce. The problem is displaying it in a digestable form and picking out important trends or anomalies. My output is of the form:
( 2009-12-08T08:30 , /ctrl_a/action_a ) (2440, 895)
( 2009-12-08T08:30 , /ctrl_a/action_b ) (2369, 1549)
( 2009-12-08T08:30 , /ctrl_b/action_a ) (2167, 0)
( 2009-12-08T08:30 , /ctrl_b/action_b ) (1713, 1184)
( 2009-12-08T08:31 , /ctrl_a/action_a ) (2317, 790)
( 2009-12-08T08:31 , /ctrl_a/action_b ) (2254, 1497)
( 2009-12-08T08:31 , /ctrl_b/action_a ) (2112, 0)
( 2009-12-08T08:31 , /ctrl_b/action_b ) (1644, 1089)
i.e., the keys are time periods and the values are tuples of (action, hits, cache hits) per time period. (I don t have to stick with this; it s just what I have so far.)
There are about 250 actions. They could be combined into a smaller number of groups but plotting the number of requests (or response time, etc) for each action over time on the same graph probably won t work. Firstly it ll be way too noisy and secondly the absolute numbers don t matter too much -- a 100 req/min rise in requests for a often-used, lightweight, cacheable response is much less important than a 100 req/min rise in a seldom-used but expensive (maybe hits the DB) uncacheable response. One the same graph we wouldn t see the changes in requests for the little-used action.
A static report isn t much good -- a huge table of numbers is hard to digest. If I aggregate by the hour we might miss important minute-by-minute changes.
Any suggestions? How re you handling this problem? I guess one way would be to somehow highlight significant changes in the rate of requests or response time per action. A rolling average and standard deviation might show this, but could I do something better?
What other metrics or figures could I generate?