From: ted.dunning@gmail.com   
      
   I can relate the history and evolution of expert systems in a few   
   areas.   
      
   In financial fraud detection, particularly the problem of finding   
   credit card transaction fraud, expert systems were an early candidate.   
   Based on experts' experience, rules were posed which characterized   
   known fraud and the output on unseen data was used to figure out how   
   the rules would go wrong.  These systems did as well as the state of   
   the art at the time which was considerably worse than the state of the   
   art now.  That state of the art was defined by supervised training   
   algorithms running against specially defined and somewhat intricate   
   features of transactions and history.   
      
   The interesting and important thing that happened is that the fraud   
   characteristics evolved over time as, indeed, things in the real world   
   tend to do.  The rule developers continued to add rules but as they   
   reached systems with hundreds to thousands of rules, the interactions   
   between the rules became the dominant determinant of the behavior of   
   the system.  As a result, these systems became harder and harder to   
   maintain.  The systems based on supervised learning had no such   
   difficulties; as new features were proposed or as the world shifted,   
   they simply retrained their model and performance was maintained and   
   the level of complexity did not increase.   
      
   Ultimately, the rules systems suffered so severely from bit-rot that it   
   became cheaper to buy the services of a trained system.   One company   
   in particular, HNC Software, did a good enough job in this area so that   
   they eventually dominated the market for credit card fraud detection   
   software and services.  Eventually, CITI was just about the only place   
   that a system based primarily on large numbers of human designed rules   
   persisted, and that was largely due to a large staff of modelers who   
   developed special techniques to allow them to retire old rules and   
   manage the interactions.   
      
   The really important lesson from all of this is not that rules are bad   
   and neural nets are good.  The HNC system in the end had a rule   
   post-processor which was critical in assuring that the system could   
   comply with various regulatory and business requirements.  For example,   
   there is a limitation period in the US after a first contact about a   
   possible fraud during which a customer cannot be contacted about the   
   same issue.  Getting a neural net to understand and comply with   
   arbitrary and stiff requirements like this is a losing battle.   
      
   The features that were inputs into the system were also examples of   
   very simple rules if you define any hand-coded processing as a rule.   
   For example, you could define apparent velocity as the ratio of   
   distance and time between transactions and set a hard or soft threshold   
   for what constitutes "high" velocity.  You could do the same to define   
   "large" amounts of cash advances.  These might make very good input   
   variables for a learning machine, but you could also make the case that   
   they are just rules with soft outputs.   
      
   The difference, then, is really one of emphasis rather than of essence.   
    The systems that ultimately failed depended on large numbers of fairly   
   complex rules with binary outputs that were combined using human   
   specified rules in the style of expert systems from the 1980's.  They   
   failed because the combination step quickly became unmaintainable when   
   they reached the software-engineering break-even where each bug fix   
   introduces as many new bugs as it removes old bugs.   
      
   The systems that succeeded used very simple rules with soft outputs   
   combined using a learning system that weighted these inputs and their   
   interactions so as to optimize performance.  The commonly repeated   
   nostrum about neural nets and similar learning algorithms being   
   "uninspectable" as compared to traditional rule-based systems was shown   
   to be completely misguided because supervised learning systems often   
   provide an ability to quantify the average benefit of any particular   
   input.  This ability to diagnose which inputs were useful and which   
   counter-productive combined with the ability to learn new patterns is   
   what really prevented bit-rot from making these systems progressively   
   less effective.   
      
   At this point, the lessons of history are pretty clear, at least with   
   respect to fraud detection.  You can't do without rules in such a   
   problem, at least in the form of feature detectors, but these rules   
   have to be kept exceedingly simple and must be rigorously evaluated for   
   an benefit.  You also absolutely must have a very sophisticated way of   
   combining the output of these rules that allows you to determine net   
   benefit and blame.  Finally, you really will need some sort of ability   
   to impose business rules on the outputs.   
      
   I should point out that the market has spoken in other ways as well.   
   According to Amazon, at least, Feigenbaum's book is now literally not   
   worth the paper it is printed on.  That is a pretty harsh judgement and   
   I think that he had some interesting things to say at the time, but we   
   have learned a lot since then.  Peter Norvig's book, on the other hand,   
   is still selling.  Conclude what you will, but I think that the fact   
   that he put in chapters on reasoning in the face of uncertainty and on   
   learning systems is a pretty major component of his success.   
      
   [ comp.ai is moderated.  To submit, just post and be patient, or if ]   
   [ that fails mail your article to , and ]   
   [ ask your news administrator to fix the problems with your system. ]   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)