From: Greg Goebel[SMTP:firstname.lastname@example.org]
Sent: Friday, May 29, 1998 6:39 AM
Subject: Re: VRf need help for an rounding error Vee 4.0
Michael Janitz wrote:
> in order to pick up a measuring instrument I use
> exact parameters e.g. "87.5"...
> but generating these parameters for addition, substraction...
> there are rounding errors e.g. "87.499 999 999"
> which produce an "error" message on the instrument.
> What to do?
The errors are due to binary floating-point approximation
of decimal numbers ... if you are sending such values to an
instrument with Direct I/O, you can specify the number of
digits of precision with a WRITE TEXT REAL transaction.
You could use a TO STRING object in parallel to doublecheck
that you are getting the formatting you want.
[<>] regards -- gvg