From: sgk@REMOVEtroutmask.apl.washington.edu   
      
   On Fri, 15 Dec 2023 09:59:56 +0800, Robinn wrote:   
      
   > I got some old neural network code (written about 30 years ago).   
   > It has several activation functions, which only change 2 lines, like so:   
   >   
   >        if (activation(1:2).eq.'SI' .or. activation(1:2).eq.'LO') then   
   >           output(i,j) = 1.0/(1.0+EXP(-output(i,j)))       ! sigmoid   
   >           slope(i,j) = output(i,j) * (1.0 - output(i,j)) ! sigmoid   
   >        elseif (activation(1:2).eq.'TA') then   
   >           output(i,j) = TANH(output(i,j))                 ! TANH   
   >           slope(i,j) = 1.0 - output(i,j)*output(i,j)     ! TANH   
   >        elseif (activation(1:2).eq.'AR') then   
   >           y = output(i,j)   
   >           output(i,j) = ATAN(y)                           ! arctan   
   >           slope(i,j) = 1.0/(1.0 +y*y)                  ! arctan   
   >        elseif (activation(1:5).eq.'SOFTP') then   
   >           y = EXP(output(i,j))   
   >           output(i,j) = LOG(1.0+y)                        ! softplus   
   >           slope(i,j) = 1.0/(1.0+1.0/y)               ! softplus   
   >        elseif (activation(1:5).eq.'SOFTS') then   
   >           y = output(i,j)   
   >           output(i,j) = y/(ABS(y)+1.0)                    ! softsign   
   >           slope(i,j) = 1.0/(1.0+ABS(y))**2             ! softsign   
   >   
   > Now when running it, the tanh option is slowest, as expected.   
   > But the sigmoid (using exp) is faster than softsign, which only needs   
   > abs  and simple arithmetic. How can this be? Even if exp is using a   
   > table lookup and spline interpolation, I would think that is slower.   
   > Softsign would have an extra divide, but I can't see that tipping the   
   > scales.   
      
   There is insufficient information to provide much help.  First, what   
   compiler and operating system?  Second, how did you do the timing?   
   Third, is there a minimum working example that others can profile?   
      
   --   
   steve   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)