in order to pick up a measuring instrument I use
exact parameters e.g. "87.5"...
but generating these parameters for addition, substraction...
there are rounding errors e.g. "87.499 999 999"
which produce an "error" message on the instrument.
What to do?
exact parameters e.g. "87.5"...
but generating these parameters for addition, substraction...
there are rounding errors e.g. "87.499 999 999"
which produce an "error" message on the instrument.
What to do?
> in order to pick up a measuring instrument I use
> exact parameters e.g. "87.5"...
> but generating these parameters for addition, substraction...
> there are rounding errors e.g. "87.499 999 999"
> which produce an "error" message on the instrument.
> What to do?
Sir:
The errors are due to binary floating-point approximation
of decimal numbers ... if you are sending such values to an
instrument with Direct I/O, you can specify the number of
digits of precision with a WRITE TEXT REAL transaction.
You could use a TO STRING object in parallel to doublecheck
that you are getting the formatting you want.
[<>] regards -- gvg