> I am exporting data from VEE 6 to Exel. The problem is dat VEE has a
> different (strange) way of handling reals. (e.g. 0.7 becomes
> 0.699999999996).
>
> Is there a way to overcome this problem; e.g. so to get exact 0.7 in
> exel without making use of text-strings?
What you are seeing is simply computer math at work- no computer
represents numbers exactly since they are digital devices. In this case
IEEE 64-bit math cannot exactly represent 0.7. VEE by default
represents it as accurately as IEEE std allows- 0.699999999996
as you note.
Excell also cannot represent this as exact 0.7 (without
using strings) .
What you want , however, is just a number format issue.
When you write to Excel set formatting to the precision that you want
or over in Excell set the formatting as you want- then you
will see "0.7" on the screen as you want.
Stan
PS- note that this is true for ALL programs on all computers-
only the details change, such as number precision and default
formatting.
---------------------------------------------------------------------
This is the "vrf" maillist, managed by Majordomo. To send messages to
this maillist, just email to "vrf@lvld.agilent.com". Subscriptions and
unsubscriptions are done through the address "vrf-request@lvld.agilent.com".
If you need details, just send a message containing the text "help"
to "vrf-request@lvld.agilent.com".
---------------------------------------------------------------------
> different (strange) way of handling reals. (e.g. 0.7 becomes
> 0.699999999996).
>
> Is there a way to overcome this problem; e.g. so to get exact 0.7 in
> exel without making use of text-strings?
What you are seeing is simply computer math at work- no computer
represents numbers exactly since they are digital devices. In this case
IEEE 64-bit math cannot exactly represent 0.7. VEE by default
represents it as accurately as IEEE std allows- 0.699999999996
as you note.
Excell also cannot represent this as exact 0.7 (without
using strings) .
What you want , however, is just a number format issue.
When you write to Excel set formatting to the precision that you want
or over in Excell set the formatting as you want- then you
will see "0.7" on the screen as you want.
Stan
PS- note that this is true for ALL programs on all computers-
only the details change, such as number precision and default
formatting.
---------------------------------------------------------------------
This is the "vrf" maillist, managed by Majordomo. To send messages to
this maillist, just email to "vrf@lvld.agilent.com". Subscriptions and
unsubscriptions are done through the address "vrf-request@lvld.agilent.com".
If you need details, just send a message containing the text "help"
to "vrf-request@lvld.agilent.com".
---------------------------------------------------------------------
different (strange) way of handling reals. (e.g. 0.7 becomes
0.699999999996).
Is there a way to overcome this problem; e.g. so to get exact 0.7 in
exel without making use of text-strings?
Thanks,
Hans van Zwet
____________________________________________________________________________
Ing. J.M. van Zwet TCA (Technological Control and
assessment)
Tel : +31(0)15-8881211 Brassersplein 2
mailto:jvzwet@office.casema.nl Postbus 345, 2600 AH Delft
____________________________________________________________________________
---------------------------------------------------------------------
This is the "vrf" maillist, managed by Majordomo. To send messages to
this maillist, just email to "vrf@lvld.agilent.com". Subscriptions and
unsubscriptions are done through the address "vrf-request@lvld.agilent.com".
If you need details, just send a message containing the text "help"
to "vrf-request@lvld.agilent.com".
---------------------------------------------------------------------