home bbs files messages ]

Forums before death by AOL, social media and spammers... "We can't have nice things"

   comp.ai      Awaiting the gospel from Sarah Connor      1,954 messages   

[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]

   Message 809 of 1,954   
   Ted Dunning to All   
   Re: Detecting Anomalies of events   
   24 Oct 05 10:50:19   
   
   XPost: comp.ai.neural-nets, comp.databases, comp.ai.fuzzy   
   XPost: sci.math   
   From: ted.dunning@gmail.com   
      
   I re-read the original posting and realized that I (and the other   
   posters) had missed the fact that all of the examples were cases of   
   time-embedded events with non-constant frequency.   
      
   As it turns out, I have done a fair bit of work on this and should have   
   been able to give a much better answer.   
      
   The simple and practical answer is to view each of the types of even as   
   a Poisson process with a non-linear time warp.  It is very easy to   
   build alarms for Poisson processes since you can build a model that has   
   specified false alarm/missed event rates just by looking at the delay   
   since the last event.  Some preprocessing may be necessary if accesses   
   from a single source are clustered as would often be the case with a   
   database.   
      
   Finding the correct time warp is as simple as estimating the average   
   rate of events.  For web sales, this is as easy as building a model   
   with time of day, day of week and a holiday flag.  Day of week is often   
   represented simply as a weekend flag.  Generalized linear models are   
   really the right tool for this, but you can just do hourly average   
   rates on the kinds of days and be pretty much in business (with a   
   little bit of linear smoothing).  I have built activity level alarms   
   based on this approach that worked very well indeed.   
      
   Sorry for being dense.   
      
   [ comp.ai is moderated.  To submit, just post and be patient, or if ]   
   [ that fails mail your article to , and ]   
   [ ask your news administrator to fix the problems with your system. ]   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)   

[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]


(c) 1994,  bbs@darkrealms.ca