AnsweredAssumed Answered

vrf Real?

Question asked by VRFuser on May 23, 2008
> I'm using a constant data real
> 32 with the value 2.2, but I get
> 2.200000047683716.

It's a rounding error, and every language suffers the same problem to some extent. There are several things you can do about it, but probably the easiest is to convert any constants explicitly. For instance, if you change the condition to asReal32(2.2)==A it works.

Also, Real64 works as expected without explicitly converting (though to what place that would hold true I don't know).
-SHAWN-




---
You are currently subscribed to vrf as: ming_meng@agilent.com To subscribe please send an email to: "vrf-request@lists.it.agilent.com" with the word subscribe in the message body.
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list,  email "vrf@agilent.com". 
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".
Search the "unofficial vrf archive" at "http://www.vrfarchive.com/vrf_archive".
Search the Agilent vrf archive at "http://vee.engineering.agilent.com".

Outcomes