home bbs files messages ]

Forums before death by AOL, social media and spammers... "We can't have nice things"

   comp.lang.visual.basic      MS Visual Basic discussions, NOT dot-net      10,840 messages   

[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]

   Message 9,459 of 10,840   
   Rick Rothstein to All   
   Re: Very strange thing in VB6   
   01 May 05 12:46:05   
   
   From: rickNOSPAMnews@NOSPAMcomcast.net   
      
   > Int performs the "return integer" stuff on all bits and   
   > bobs of the calculation not just the result, so int reads   
   > your calculation as (5 * 1)   
      
   NO! That is not how the Int function works! If it did work that way,   
   then   
      
        MsgBox Int(1.2 * 3.4)   
      
   would display 3; but that is NOT what is displayed... the value 4 is   
   displayed because the values are NOT made into integers before the   
   multiplication. The problem is due to the reasons discussed in the   
   several articles I cited links for in my previous postings to this   
   thread. You should go back and read them in order to correct your   
   mistaken idea of how the Int function works.   
      
      
   > The way around it is to pass values, not calculations   
   > to the Int function. You should have written   
   >   
   > lngResult = (5 * 1.2)   
   > Int(lngResult)   
      
   Since you named your variable lngResult, one has to assume you Dim'med   
   it as Long. If that is correct, then the Int function call is not needed   
   because the act of assigning a floating point number to a variable that   
   has been declared as type Long automatically forces the number to an   
   integer (non-floating point) number. So the Int function actually does   
   nothing in the above calculation. However, one needs to be aware of a   
   potential problem when assigning floating point values to a type Long   
   variable... VB uses Banker's Rounding on the floating point number to   
   make it into an integer value prior to the assignment. Unexpected   
   results can follow if your calculation produces a floating point value   
   that end in exactly ".5" because Banker's Rounding rounds these numbers   
   to the nearest EVEN preceding digit. Consider this code snippet...   
      
        Dim lngResult As Long   
        lngResult = 1 * 1.5   
        Debug.Print lngResult   
        lngResult = 2 * 1.5   
        Debug.Print lngResult   
        lngResult = 3 * 1.5   
        Debug.Print lngResult   
        lngResult = 4 * 1.5   
        Debug.Print lngResult   
        lngResult = 5 * 1.5   
        Debug.Print lngResult   
        lngResult = 6 * 1.5   
        Debug.Print lngResult   
        lngResult = 7 * 1.5   
        Debug.Print lngResult   
        lngResult = 8 * 1.5   
        Debug.Print lngResult   
        lngResult = 9 * 1.5   
        Debug.Print lngResult   
      
   It prints out the following values...   
      
         2   
         3   
         4   
         6   
         8   
         9   
         10   
         12   
         14   
      
   Notice that the numbers 5, 7, 11, 13 do not appear.   
      
   Okay, that takes care of the case where you Dim'med lngResult as Long.   
   If, on the other hand, you didn't declare it as anything (a very bad   
   practice in my opinion), then it is a Variant. In that case, the   
   calculation forces the variable to a data sub-type of Double (during the   
   assignment) Doubles have "quirks" of their own. Double are 16 digit   
   values, but only expose 15 digits of them. The 16th digit provides a   
   guard digit to help maintain accuracy during chained calculations. In   
   essence, VB uses this 16 decimal place to absorb the errors inherent in   
   floating point values not being able to be stored exactly using the IEEE   
   standard. Now I'm not 100% sure if the 16th place is lost during the   
   assignment, or if it is retained but ignored every time a value is   
   retrieved from a variable of type or sub-type Double, but the net effect   
   is the calculation 5*1.2 becomes the value of 6, exactly, when stored in   
   a Double type variable. You can see this like so...   
      
        Dim lngResult As Double   
        lngResult = 5 * 1.2   
        Debug.Print lngResult - 6   
        Debug.Print 5 * 1.2 - 6   
      
   The first debug print out displays 0 indicated that the value stored in   
   the Double type variable is (or is being treated as if it is) exactly 6   
   (to 15 digits of accuracy). The second debug statement, on the other   
   hand, displays   
      
        -2.22044604925031E-16   
      
   indicating that during chained calculations, the 16th decimal place is   
   still being used. Floating point numbers can be a real pain in the ass   
   to work with.   
      
      
   Rick - MVP   
      
   Rick - MVP   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)   

[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]


(c) 1994,  bbs@darkrealms.ca