From: pastgio@units.it   
      
   Il 15/12/23 05:22, Steven G. Kargl ha scritto:   
   > On Fri, 15 Dec 2023 09:59:56 +0800, Robinn wrote:   
   >   
   >> I got some old neural network code (written about 30 years ago).   
   >> It has several activation functions, which only change 2 lines, like so:   
   >>   
   >>         if (activation(1:2).eq.'SI' .or. activation(1:2).eq.'LO') then   
   >>            output(i,j) = 1.0/(1.0+EXP(-output(i,j)))       ! sigmoid   
   >>            slope(i,j) = output(i,j) * (1.0 - output(i,j)) ! sigmoid   
   >>         elseif (activation(1:2).eq.'TA') then   
   >>            output(i,j) = TANH(output(i,j))                 ! TANH   
   >>            slope(i,j) = 1.0 - output(i,j)*output(i,j)     ! TANH   
   >>         elseif (activation(1:2).eq.'AR') then   
   >>            y = output(i,j)   
   >>            output(i,j) = ATAN(y)                           ! arctan   
   >>            slope(i,j) = 1.0/(1.0 +y*y)                  ! arctan   
   >>         elseif (activation(1:5).eq.'SOFTP') then   
   >>            y = EXP(output(i,j))   
   >>            output(i,j) = LOG(1.0+y)                        ! softplus   
   >>            slope(i,j) = 1.0/(1.0+1.0/y)               ! softplus   
   >>         elseif (activation(1:5).eq.'SOFTS') then   
   >>            y = output(i,j)   
   >>            output(i,j) = y/(ABS(y)+1.0)                    ! softsign   
   >>            slope(i,j) = 1.0/(1.0+ABS(y))**2             ! softsign   
   >>   
   >> Now when running it, the tanh option is slowest, as expected.   
   >> But the sigmoid (using exp) is faster than softsign, which only needs   
   >> abs  and simple arithmetic. How can this be? Even if exp is using a   
   >> table lookup and spline interpolation, I would think that is slower.   
   >> Softsign would have an extra divide, but I can't see that tipping the   
   >> scales.   
   >   
   > There is insufficient information to provide much help.  First, what   
   > compiler and operating system?  Second, how did you do the timing?   
   > Third, is there a minimum working example that others can profile?   
   >   
      
   Fourth, what were the numbers of timing.   
      
   Giorgio   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)