AnsweredAssumed Answered

Driver for HP4277A LCR Meter

Question asked by VRFuser on Jun 2, 1998

From:  Greg Goebel[]
Sent:  Friday, May 29, 1998 6:39 AM
Subject:  Re: VRf need help for an rounding error Vee 4.0

Michael Janitz wrote:

>  in order to pick up a measuring instrument I use
> exact parameters e.g. "87.5"...
> but generating these parameters for addition, substraction...
> there are rounding errors e.g. "87.499 999 999"
> which produce an "error" message on the instrument.
> What to do?


The errors are due to binary floating-point approximation
of decimal numbers ... if you are sending such values to an
instrument with Direct I/O, you can specify the number of
digits of precision with a WRITE TEXT REAL transaction.

You could use a TO STRING object in parallel to doublecheck
that you are getting the formatting you want.

[<>] regards -- gvg