From: Albert_Rich@msn.com   
      
   On Thursday, October 26, 2017 at 4:57:33 PM UTC-10, Richard Fateman wrote:   
   > You should realize that once you publish your test set   
   > it is possible to game the benchmark with a program that simply looks up   
   > the problem and returns the optimal solution.  Not that   
   > I think that is what is going on, but   
   > "Lies, Damned Lies, and Statistics... and Benchmarks".,   
   >    
   > RJF   
      
   Ok, let's say a devious implementer front-ends its symbolic integrator to   
   return optimal antiderivatives for all 66,000+ problems in the test suite.    
   That's great!  It would be a major improvement, considering that current   
   versions of the commercial    
   systems tested by Nasser get A's on less than 71% of the problems.   
      
   By in large the integration test suite uses variables, rather than constants,   
   for coefficients and exponents.  Therefore, the "gamed" system would not only   
   get optimal antiderivatives for the 66,000+ problems in the test suite, but   
   also for the    
   infinitely many instances of those integrands where constants are substituted   
   for variables in the integrand.   
      
   But, of course, that would be a crazy thing to do.  A much easier and more   
   efficient way to improve a system's grade is to look up instances of the   
   right-side of the just 6200+ rules Rubi uses to integrate expressions.    
   They're freely available on Rubi's    
   website.   
      
   Albert   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)