AnsweredAssumed Answered

vrf reals

Question asked by rsb on Dec 6, 2001
> I am exporting data from VEE 6 to Exel. The problem is dat VEE has a
> different (strange) way of handling reals. (e.g. 0.7 becomes
> 0.699999999996).
>
> Is there a way to overcome this problem; e.g. so to get exact 0.7 in
> exel without making use of text-strings?


What you are seeing is simply computer math at work- no computer
represents numbers exactly since they are digital devices. In this case
IEEE 64-bit math cannot exactly represent 0.7. VEE by default
represents it as accurately as IEEE std allows- 0.699999999996
as you note.

Excell also cannot represent this as exact 0.7 (without
using strings) .

What you want , however, is just a number format issue.

When you write to Excel set formatting to the precision that you want
or over in Excell set the formatting as you want- then you
will see "0.7" on the screen as you want.

Stan

PS- note that this is true for ALL programs on all computers-
only the details change, such as number precision and default
formatting.
---------------------------------------------------------------------
This is the "vrf" maillist, managed by Majordomo.  To send messages to
this maillist, just email to "vrf@lvld.agilent.com".  Subscriptions and
unsubscriptions are done through the address "vrf-request@lvld.agilent.com".
If you need details, just send a message containing the text "help"
to "vrf-request@lvld.agilent.com".
---------------------------------------------------------------------

Outcomes