From: Albert_Rich@msn.com   
      
   On Tuesday, March 7, 2017 at 8:47:01 PM UTC-10, Nasser M. Abbasi wrote:   
   >    
   > Currently the build just checks if the integration did not "fail"   
   > and also completed in the time limit set for each problem. No other checks   
   > are done.   
   >    
   > If you like to grade the problem, and since you know better what   
   > these criteria should be, how about providing a black box function,   
   > that can be called with the antiderivative expression generated,   
   > and the grading function will return a score number or whatever   
   > you decide the score should be (A,B,C etc...)? I will use the   
   > score it returns.   
   >    
   > Would need such grading function in Mathematica and Maple syntax   
   > only and will be happy to use it and add it to my build scripts.   
   >    
   > You can decide on the API for this grading function.   
   >    
   > Does this sound OK with you?   
      
   Hello Nasser,   
      
   To keep it simple, I recommend your test program just indicate if the   
   antiderivative returned by a system is optimal (O), nonoptimal (N), or if it   
   fails (F) to return an antiderivative.  Then the percentage of O, N and F   
   grades a system gets on the test    
   suite would give a good indicator of the overall quality of its results.   
      
   Optimal and nonoptimal results can be distinguished by comparing them with the   
   optimal antiderivatives included in the test suite.  A result should be   
   considered nonoptimal (N) if any one of the following is true:   
   1.  it contains a hypergeometric function and the optimal antiderivative does   
   not;   
   2.  it contains a special function and the optimal antiderivative does not;   
   3.  it contains the imaginary unit and the optimal antiderivative does not; or   
   4.  its expression size (leaf count) is more than twice that of the optimal   
   antiderivative.   
      
   A result should be considered failed (F) if any one of the following is true:   
   1.  it contains an unevaluated integral and the optimal antiderivative does   
   not; or   
   2.  the system fails to return a result after some fixed timeout period.   
      
   Otherwise the result should be considered optimal (O).     
      
   Note that this comparison grading system assumes results returned by a system   
   are valid antiderivatives.  It is often difficult to verify the validity of   
   antiderivatives by differentiation.  Perhaps a future version of the test   
   program could also verify    
   results.   
      
   Once your test program incorporates such a grading system, you might consider   
   an option to show in the test result files only those problems given N or F   
   grades.  This will greatly reduce the size of the test results and highlight   
   weaknesses in the    
   various systems, useful to implementers and users alike.   
      
   This grading system is relatively easy to implement, and I would be happy to   
   work with you on it off-line via email.     
      
   Finally since Rubi is one of the systems being tested, I encourage readers of   
   this post to indicate if they think such a grading system would be useful AND   
   fair.   
      
   Albert   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)