Hi All

I've been doing some IO to an instrument using "WRITE BINARY REAL64"

transactions and expecting to get something like IEEE 754 floating

point std (64 bits total. 1 sign bit, 11 exp 52 mantissa). I believe

that this is about what is happening.

HOWEVER I seem to more-or-less randomly get an arithmetic error

when the transaction is executed, related to the values that I

input.

In order to investigate I dredged up some IEE 854 VEE code that I wrote

a while back and started looking at various numbers run thru this

alogrithm. Sure enough, if I place a continuous slider in front of my

algorithm it errors out on some numbers. I calculate 4 16 bit words

and see an overflow or underflow on the math for one or more of the

words.

I'm guessing that the issue here is that the numbers I input are simply

not realizable using the finite resolution of this format.

If so, how do I then filter the numbers in VEE to convert them to the

closest numer that can be represented in "REAL64" format?

thanks

Stan

----------------------------------------------------------------------

Stan Bischof Hewlett Packard Company 707-577-3994 stanb@sr.hp.com

----------------------------------------------------------------------

I've been doing some IO to an instrument using "WRITE BINARY REAL64"

transactions and expecting to get something like IEEE 754 floating

point std (64 bits total. 1 sign bit, 11 exp 52 mantissa). I believe

that this is about what is happening.

HOWEVER I seem to more-or-less randomly get an arithmetic error

when the transaction is executed, related to the values that I

input.

In order to investigate I dredged up some IEE 854 VEE code that I wrote

a while back and started looking at various numbers run thru this

alogrithm. Sure enough, if I place a continuous slider in front of my

algorithm it errors out on some numbers. I calculate 4 16 bit words

and see an overflow or underflow on the math for one or more of the

words.

I'm guessing that the issue here is that the numbers I input are simply

not realizable using the finite resolution of this format.

If so, how do I then filter the numbers in VEE to convert them to the

closest numer that can be represented in "REAL64" format?

thanks

Stan

----------------------------------------------------------------------

Stan Bischof Hewlett Packard Company 707-577-3994 stanb@sr.hp.com

----------------------------------------------------------------------