XPost: comp.ai, comp.ai.neural-nets, comp.databases   
   XPost: sci.math   
   From: kirtu@earthlink.net   
      
   CMOS wrote:   
   > hi all,   
   > i need to build a system which will find any anomalies of a particular   
   > activity. The system will be feeded with events that are happening and   
   > it should be able to find any significant deviation from the normal   
   > operation. ...   
   > 1 ) webserver access   
   ...   
   > 5 ) access ( login ) patterns, frequency to a particular system   
   > ...   
   >   
   > basically i need to be alerted on   
   >   
   > deviation of events's frequency from normal   
   > deviation in pattern of events that are happening, etc   
   >   
   > i wonder whether there is any area in mathematics / computer science   
   > which deal with that kind of problems. So i really appreciate if some   
   > one can suggest me a good path for the project.   
      
   You have made a mistake common to this kind of problem.   
      
   Namely you are assuming that derivations from an observed norm is an   
   error.   
      
   With the domains listed you can get away with this.  But there are   
   domains where this assumption will be false.  Specifically this   
   assumption does not necessarily hold for computer log file analysis or   
   in fact in any domain in which misconfiguration and/or malfunctions   
   (i.e. bugs requiring patches or hotfixes applied to a large number of   
   systems [like the endless stream of hotfixes that Microsoft was putting   
   out before they started staging their hotfixes as upgrades]) are very   
   common and can be seen as normal.  In fact with such systems outliers   
   can actually be correctly configured systems! With these sorts of   
   domains the problem is compounded because there isn't generally a   
   universal baseline that can be established because different systems   
   have differing configurations because of different needs in different   
   environments.   
      
   Regarding even the domains listed, access is handled best by policy and   
   policy isn't always reflected in actually access patterns.  If you have   
   a large number of users and there have been long-term undetected   
   intrusion, then simply looking for outliers alone will not detect the   
   intrusion because the currently undetected intrusion is already   
   established as the norm (an extreme but possible condition).  Outlier   
   analysis like this can only detect new  intrusion.  So consider adding   
   policy checking.   
      
   BTW - these issues came up in a real life system I was working on where   
   people from a major defense contractor made the same naive assumption.   
      
   Kirt Undercoffer   
      
   [ comp.ai is moderated.  To submit, just post and be patient, or if ]   
   [ that fails mail your article to , and ]   
   [ ask your news administrator to fix the problems with your system. ]   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)