From: jbb@notatt.com   
      
   On 9/3/2022 10:43 PM, Nasser M. Abbasi wrote:   
   > On 9/3/2022 11:25 PM, Jeff Barnett wrote:   
   >   
   >>>   
   >>> Random.seed!(12) #must do this in order to reproduce same result   
   >>>   
   >>> Then now same antiderivative is generated each time. So now I can add   
   >>> Julia integrator to CAS integration tests.   
   >>   
   >   
   >   
   >> That doesn't seem correct to me since users will get whatever first seed   
   >> the system generates (unless they choose their own as you are). The   
   >> chances that they will choose 12 as you have is small. I would either   
   >> give the system a fail because it is unreliable on that problem or I   
   >> would test it for, say, a 100 or so seeds and give the number (a   
   >> percent) that it gets correct. The fact that you had to search for a   
   >> seed that led to correct behavior voids the system's ability for that   
   >> example. It might be that you could find a seed for each problem in the   
   >> test suite that leads to correct behavior for the problem. But that   
   >> doesn't make the system perfect.   
   >>   
   >> Basically when you are testing a system that uses a "probabilistic"   
   >> approach, you need to develop scoring methods that account for that fact.   
   >   
   >   
   > But I have documented that this result is based on using seed `12`.   
   > Any other value could very well generate new result.   
   >   
   > So if someone wants to obtain the same result as I have obtained,   
   > they have to use same seed.   
      
   I've snipped everything below since I wasn't clear and I think there is   
   a misunderstanding. I assumed, tacitly, that people look at the results   
   of running test suites "to understand how well they can expect them to   
   work" they do not look at the results to learn how to solve exactly the   
   problems in that suite. What about a similar problem which is properly   
   solved with initial seed 32457689912 but fails with 12? They look at   
   your work to learn which systems are trustworthy and worth the price.   
      
   A zillion years ago I wrote a program to form fault trees from systems   
   where some of the components had circular dependencies. That fault tree   
   then was reduced to minimal fault sets. This gibberish meant find sets   
   of components such that if all failed, the system failed and that no   
   subset of fault set led to system failure. This is yet another   
   processing of and or expressions to get in a particular form and, of   
   course, this is one of those BP complete problems. The algorithm I   
   developed order all the variables and did some magic that allowed the   
   output tree to share, multiple times, most of its nodes dramatically   
   saving space. The exact amount of space but more importantly, the   
   runtime depended on the variable order chosen. There was no way to   
   guarantee a polynomial run time since it was an NP complete problem. For   
   engineering work I would try an ordering and after a while I'd start it   
   with another and I usually found solutions in very good time. The   
   problem was that one day it solved a problem in few seconds and the next   
   day it might run for hours and not terminate. Most of the work I did was   
   trying to develop heuristics to find good orders that led to good   
   performance. However, those heuristics often depended on what order they   
   examined parameters.   
      
   The problem with a test suite for the above is the users didn't care   
   about all these internal machinations; they just wanted to know if the   
   system would work. I could have published test cases and a good ordering   
   heuristic for each one but that would be useless information to an   
   engineer that was trying to do a reliability forecast. The work was for   
   the design of ultra performance aircraft and was to evaluate the   
   reliability of the plane and its safety and effectiveness. The joke in   
   all of this was that their current methods for dealing with circular   
   dependency were not correct; what I mean is you have in effect several   
   boolean equations in many variables and you substitute one of the   
   equations into another and an inconsistency pops out.   
      
   Let me finish by repeating a line from above: "What about a similar   
   problem which is properly solved with initial seed 32457689912 but fails   
   with 12?" Test suites must tell someone how well things work; They are   
   not to teach hacking.   
   --   
   Jeff Barnett   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)