This should be simple, right?

[greavis]

Hello,

I'm building a detailed physics modeling application and the known rounding errors of the floating-point data types are causing me problems. I've converted my calculations to use Decimal and things have worked out so far. However, in using the Math.Sqrt method (which takes in and returns a Double) I can not convert the return value from a Double to a Decimal without the value being distorted.

Take the following example...

Dim Distance As Double
Distance = System.Math.Sqrt(1000000)

Now "Distance" comes back with "1000" which is good. Now, how can I convert "Distance" to a Decimal? When I use "CDec" or "Convert.ToDecimal" (I assume they are the same) the value comes back as "1000.00000376832D". This has become the bain of my existence...

Any ideas/help?!!!

Thank you!
- Greavis

 

[ManicCW]

I've tested this and I see no problem the result is 1000. Here is the code:

[SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Private[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Sub[/COLOR][/SIZE][SIZE=2] Button1_Click([/SIZE][SIZE=2][COLOR=#0000ff]ByVal[/COLOR][/SIZE][SIZE=2] sender [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] System.Object, [/SIZE][SIZE=2][COLOR=#0000ff]ByVal[/COLOR][/SIZE][SIZE=2] e [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] System.EventArgs) [/SIZE][SIZE=2][COLOR=#0000ff]Handles[/COLOR][/SIZE][SIZE=2] Button1.Click
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] Distance [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Double
[/COLOR][/SIZE][SIZE=2]Distance = [/SIZE][SIZE=2][COLOR=#0000ff]CDec[/COLOR][/SIZE][SIZE=2](System.Math.Sqrt([/SIZE][SIZE=2][COLOR=#0000ff]Me[/COLOR][/SIZE][SIZE=2].TextBox1.Text))
[/SIZE][SIZE=2][COLOR=#0000ff]Me[/COLOR][/SIZE][SIZE=2].TextBox2.Text = Distance
[/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Sub
[/COLOR][/SIZE]

I've added two textboxes and button on form. You inssert value in first textbox and click button, it shows result in second textbox