From: felixcruxca@gmail.com   
      
   On Tue, 2008-01-29 at 02:35 +0000, talsegal@gmail.com wrote:   
   > Hi all,   
   >   
   > I have a general question, I hope you guys could help me.   
   >   
   > Suppose I have a classifier A that discriminates between two classes:   
   > class W and B (White balls and Black balls, respectively).   
   >   
   > Suppose I have to run the classifier on a vast set of balls (:= P), in   
   > which the distribution of White and Black balls is unknown (Which   
   > means I don't know the a-priori probability of getting a white or a   
   > black ball to examine).   
   >   
   > Now I would like to test the classifier. I choose a subset of P (:=N)   
   > that consists of N balls and run the experiment to get the ROC curve   
   > of the classifier.   
   >   
   > My question is: What is the best way to set the distribution of White   
   > and Black balls in N if the distribution of P is unknown? 0.5*N Black   
   > balls and 0.5*N White balls sounds right, but is it really right?! And   
   > how would the answer change if P can be determined?   
   >   
      
       I wouldn't think that a 50/50 distribution is the best way to test,   
   since it makes it  impossible to distinguish accurate classification   
   from random guessing. In other words, did your classifier actually   
   determine that half were white and half were black, or did it flip a   
   coin each time one came up? Try something like 75% of one and 25% of the   
   other. Cheers,   
      
   Felix   
      
   [ comp.ai is moderated ... your article may take a while to appear. ]   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)