Basic maths - completely wrong

[tomhosking]

My program calculates the difference between two measured values, to see what the error is. When I ask VB to do this, it often comes out with absurd answers.

eg:
1921.25618 - 1921.25625

gives

-6.99999999

The code is:
facediff = targetSet(q).myMeasXYZFace1.X - targetSet(q).myMeasXYZFace2.X

Where facediff is a string, and the other two are double.

 

[kulrom]

I beleive you will find this thread for usefull ... Cheers

 

[tomhosking]

which thread?

 

[kulrom]

Oh my gosh ... sorry this is the link http://www.vbdotnetforums.com/showthread.php?t=3814&highlight=currency

Cheers

 

[tomhosking]

Worked it out:

If you do not explicitly convert a double to string, it doesnt give an outright error, it just gives wrong values!

 

[TechGnome]

Doubles are not an exact number. They are an approximated value data type. This stems from the fact there is no efficient way to store the number in memory accurately. This offen leads to the type of phenomenon you are seeing. We see it all the time here too. If you don't need more than 4 decimal places, the a currency data type is the best thing to to use.

Tg

 

[tomhosking]

I do need more than 4 dps Its for measurements of distances in metres to accuracies of microns...