home bbs files messages ]

Forums before death by AOL, social media and spammers... "We can't have nice things"

   comp.databases.ms-sqlserver      Notorious Rube Goldberg contraption      19,505 messages   

[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]

   Message 19,372 of 19,505   
   Erland Sommarskog to Anton Shepelev   
   Re: Strage difference between precision    
   25 Apr 22 21:58:30   
   
   From: esquel@sommarskog.se   
      
   Anton Shepelev (anton.txt@gmail.com) writes:   
   > Hello, all   
   >   
   > Can you please tell me why this query:   
   >   
   >    SELECT     POWER(0.5, 2), POWER(0.50, 2)   
   >    Returns    0.3            0.25   
   >    Instead of 0.25           0.25   
   >   
      
   All say after me: "SQL Server is designed for maximum confusion!".   
      
   I know how it feels, I've run into exactly this thing myself. And   
   I thought, what!? Then I read the manual and learnt this:   
      
   The retun type of power() is the type of the first argument. The 0.3   
   has the type decimal(1, 1), and so the return type is numeric(1,1)   
   and you get 0.3. Intuitive? Maybe not. But this is what the manual   
   says.   
      
   By the way, this is how you can tell the type of a numeric literal:   
   DECLARE @s sql_variant = 0.3   
   SELECT sql_variant_property(@s, 'Basetype'),   
          sql_variant_property(@s, 'Precision'),   
          sql_variant_property(@s, 'Scale')   
      
   To avoid such surpises, say something like:   
      
     SELECT cast(power(cast @val as float) as decimal(10,3))   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)   

[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]


(c) 1994,  bbs@darkrealms.ca