AnsweredAssumed Answered

VRF- "real64" format errors

Question asked by VRFuser on Oct 30, 1996
Hi All

I've been doing some IO to an instrument using "WRITE BINARY REAL64"
transactions and expecting to get something like IEEE 754 floating
point std (64 bits total. 1 sign bit, 11 exp 52 mantissa). I believe
that this is about what is happening.

HOWEVER I seem to more-or-less randomly get an arithmetic error
when the transaction is executed, related to the values that I

In order to investigate I dredged up some IEE 854 VEE code that I wrote
a while back and started looking at various numbers run thru this
alogrithm. Sure enough, if I place a continuous slider in front of my
algorithm it errors out on some numbers. I calculate 4 16 bit words
and see an overflow or underflow on the math for one or more of the

I'm guessing that the issue here is that the numbers I input are simply
not realizable using the finite resolution of this format.

If so, how do I then filter the numbers in VEE to convert them to the
closest numer that can be represented in "REAL64" format?



Stan Bischof   Hewlett Packard Company   707-577-3994