From: jahns@idontlikespam.dkrz.de   
      
   On 2023-12-15 02:59, Robinn wrote:   
   > I got some old neural network code (written about 30 years ago).   
   > It has several activation functions, which only change 2 lines, like so:   
   >   
   >        if (activation(1:2).eq.'SI' .or. activation(1:2).eq.'LO') then   
   >           output(i,j) = 1.0/(1.0+EXP(-output(i,j)))       !   
   sigmoid   
   >           slope(i,j) = output(i,j) * (1.0 - output(i,j)) ! sigmoid   
   >        elseif (activation(1:2).eq.'TA') then   
   >           output(i,j) = TANH(output(i,j))         
   Â Â Â Â Â Â Â Â Â  ! TANH   
   >           slope(i,j) = 1.0 - output(i,j)*output(i,j)     ! TANH   
   >        elseif (activation(1:2).eq.'AR') then   
   >           y = output(i,j)   
   >           output(i,j) = ATAN(y)              
   Â Â Â Â Â Â Â Â Â Â Â Â Â Â  ! arctan   
   >           slope(i,j) = 1.0/(1.0 +y*y)           
   Â Â Â Â Â Â Â Â  ! arctan   
   >        elseif (activation(1:5).eq.'SOFTP') then   
   >           y = EXP(output(i,j))   
   >           output(i,j) = LOG(1.0+y)             
    Â Â Â Â Â Â Â Â Â Â Â Â  ! softplus   
   >           slope(i,j) = 1.0/(1.0+1.0/y)               !   
   softplus   
   >        elseif (activation(1:5).eq.'SOFTS') then   
   >           y = output(i,j)   
   >           output(i,j) = y/(ABS(y)+1.0)           
    Â Â Â Â Â Â Â Â Â Â  ! softsign   
   >           slope(i,j) = 1.0/(1.0+ABS(y))**2             !   
   softsign   
   >   
   > Now when running it, the tanh option is slowest, as expected.   
   > But the sigmoid (using exp) is faster than softsign, which only needs   
   > abs  and simple arithmetic. How can this be? Even if exp is using a table   
   lookup   
   > and spline interpolation, I would think that is slower.   
   > Softsign would have an extra divide, but I can't see that tipping the scales.   
      
      
   You perhaps are not aware that the string comparisons (for which most compilers   
   call the strncmp function) you have in your conditionals are quite expensive on   
   todays CPUs. I would recommend to use an INTEGER constant to make the switch.   
      
   Thomas   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)