Forums before death by AOL, social media and spammers... "We can't have nice things"
|    comp.lang.visual.basic    |    MS Visual Basic discussions, NOT dot-net    |    10,840 messages    |
[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]
|    Message 9,459 of 10,840    |
|    Rick Rothstein to All    |
|    Re: Very strange thing in VB6    |
|    01 May 05 12:46:05    |
      From: rickNOSPAMnews@NOSPAMcomcast.net              > Int performs the "return integer" stuff on all bits and       > bobs of the calculation not just the result, so int reads       > your calculation as (5 * 1)              NO! That is not how the Int function works! If it did work that way,       then               MsgBox Int(1.2 * 3.4)              would display 3; but that is NOT what is displayed... the value 4 is       displayed because the values are NOT made into integers before the       multiplication. The problem is due to the reasons discussed in the       several articles I cited links for in my previous postings to this       thread. You should go back and read them in order to correct your       mistaken idea of how the Int function works.                     > The way around it is to pass values, not calculations       > to the Int function. You should have written       >       > lngResult = (5 * 1.2)       > Int(lngResult)              Since you named your variable lngResult, one has to assume you Dim'med       it as Long. If that is correct, then the Int function call is not needed       because the act of assigning a floating point number to a variable that       has been declared as type Long automatically forces the number to an       integer (non-floating point) number. So the Int function actually does       nothing in the above calculation. However, one needs to be aware of a       potential problem when assigning floating point values to a type Long       variable... VB uses Banker's Rounding on the floating point number to       make it into an integer value prior to the assignment. Unexpected       results can follow if your calculation produces a floating point value       that end in exactly ".5" because Banker's Rounding rounds these numbers       to the nearest EVEN preceding digit. Consider this code snippet...               Dim lngResult As Long        lngResult = 1 * 1.5        Debug.Print lngResult        lngResult = 2 * 1.5        Debug.Print lngResult        lngResult = 3 * 1.5        Debug.Print lngResult        lngResult = 4 * 1.5        Debug.Print lngResult        lngResult = 5 * 1.5        Debug.Print lngResult        lngResult = 6 * 1.5        Debug.Print lngResult        lngResult = 7 * 1.5        Debug.Print lngResult        lngResult = 8 * 1.5        Debug.Print lngResult        lngResult = 9 * 1.5        Debug.Print lngResult              It prints out the following values...               2        3        4        6        8        9        10        12        14              Notice that the numbers 5, 7, 11, 13 do not appear.              Okay, that takes care of the case where you Dim'med lngResult as Long.       If, on the other hand, you didn't declare it as anything (a very bad       practice in my opinion), then it is a Variant. In that case, the       calculation forces the variable to a data sub-type of Double (during the       assignment) Doubles have "quirks" of their own. Double are 16 digit       values, but only expose 15 digits of them. The 16th digit provides a       guard digit to help maintain accuracy during chained calculations. In       essence, VB uses this 16 decimal place to absorb the errors inherent in       floating point values not being able to be stored exactly using the IEEE       standard. Now I'm not 100% sure if the 16th place is lost during the       assignment, or if it is retained but ignored every time a value is       retrieved from a variable of type or sub-type Double, but the net effect       is the calculation 5*1.2 becomes the value of 6, exactly, when stored in       a Double type variable. You can see this like so...               Dim lngResult As Double        lngResult = 5 * 1.2        Debug.Print lngResult - 6        Debug.Print 5 * 1.2 - 6              The first debug print out displays 0 indicated that the value stored in       the Double type variable is (or is being treated as if it is) exactly 6       (to 15 digits of accuracy). The second debug statement, on the other       hand, displays               -2.22044604925031E-16              indicating that during chained calculations, the 16th decimal place is       still being used. Floating point numbers can be a real pain in the ass       to work with.                     Rick - MVP              Rick - MVP              --- SoupGate-Win32 v1.05        * Origin: you cannot sedate... all the things you hate (1:229/2)    |
[   << oldest   |   < older   |   list   |   newer >   |   newest >>   ]
(c) 1994, bbs@darkrealms.ca