Hi Les and vrf,

> This is not at all unexpected.

> The real32 has 32 bits to deal with mantissa and exponent.

> You are attempting to use 32 bits of mantissa accuracy alone.

> That can't be done. The system (Windows, UNIX, whatever) -

> not VEE, rounds off the number to the closest real32 it has.

> (Which then, of course, is too big for the int32)

>

> Les Hammer

Les is right, though I misread some of the values and thought we had a bug at first.

To put a little meat on the bones of Les's explanation for the diehards, the Real32 allocates 23 (actually 24 through the magic of "normalization") bits to the "mantissa", and the other 9 bits to the sign and exponent (which don't help the accuracy). The mantissa is where the "significant digits" (the accuracy) are stored in a Real.

For the Int32, there are 31 bits allocated to the significant digits, no exponent, and 1 bit to the sign (not quite true, but close enough. The integer scheme is called 2's complement, where the topmost bit has a negative weight.)

So when you try to "squeeze" the 31 bits (binary digits) of a big Int32 like 2147483647 into the 24 mantissa bits of a Real32, you have to throw away 7 bits and approximate the result in the Real32. That means that the Real32 has become a somewhat inaccurate representation of the original Int32.

How inaccurate?

The 7 bits we threw away have a value of 2^7 = 128. That's just a bit more than 2 decimal digits of accuracy that we've sacrificed in the squeeze, and it means that a whole bunch of different large Int32 values will be mapped to the same Real32 approximation. In this case, the values mapped to a value of 2147483648.0000 per the IEEE-754 math spec, which happens to be 1 unit more than the max positive value an Int32 can hold.

If you feed in the input value 2147483600 instead of 2147483647, you'll get the same error and see that the Real32 is the same 2147483648.0000. That's what I meant when I said a bunch of different Int32s will map to the same Real32 approximation. I haven't found the crossover point (where a smaller number won't cause the error because it maps to a number representable in Int32), but it's obviously something less than 2147483600.

So why use Real32s if they're so darn inaccurate?

Because they have a larger dynamic range than integers, and because they use less RAM than Real64s.

E.g. you can represent 1.23E12 as a Real32, but you can't represent this number in an Int32. With Real formats, we've traded some accuracy for increased dynamic range, using the same number of bits. With Real64, we can keep better accuracy than Real32 (Real64 has 53 bits of mantissa when normalized, so all Int32s can be accurately represented and you don't get an error), but we use twice as much RAM. RAM usage may be significant for a big array. Real32 may also be faster in computation than Real64 as well, though this could depend heavily on hardware.

Choices, choices.

Sometimes you need high accuracy (especially when counting objects), sometimes high dynamic range (picofarads vs megavolts, anyone?) Sometimes, you need neither high accuracy nor high dynamic range (e.g. a 12-bit ADC). You may need to expend some effort to choose the most suitable and efficient data type for your task.

(R&D made me write this, humph!

Regards,

Scott Bayes

Software Technical Support

Agilent Technologies, Inc.

815 14th Street S.W.

Loveland, CO, U.S.A. 80537

970 679 3799 Tel

970 635 6867 Fax

> -----Original Message-----

> From: Les Hammer [mailto:Les.Hammer@CompleteTest.com]

> Sent: Monday, February 23, 2004 5:27 PM

> To: PDL-LISTS,VRF (A-Lists,unix1)

> Subject: [vrf] RE: Real32 Number Representation

>

>

> > Dontcha just love this?

> > I don't know if this is inherent in the floating point

> representation or

> is actually a vee bug.

>

>

> This is not at all unexpected.

> The real32 has 32 bits to deal with mantissa and exponent.

> You are attempting to use 32 bits of mantissa accuracy alone.

> That can't be done. The system (Windows, UNIX, whatever) -

> not VEE, rounds off the number to the closest real32 it has.

> (Which then, of course, is too big for the int32)

>

> Les Hammer

> Partner

> Complete Test

> 720 SW 14th St.

> Loveland, CO. 80537

> Les.Hammer@CompleteTest.com

>

>

>

> ---

> You are currently subscribed to vrf as: scott_bayes@agilent.com

> To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".

> To unsubscribe send a blank email to

> "leave-vrf@it.lists.it.agilent.com".

> To send messages to this mailing list, email "vrf@agilent.com".

> If you need help with the mailing list send a message to

> "owner-vrf@it.lists.it.agilent.com".

> Send your favorite VEE example to "VRF-EXAMPLES@agilent.com"

> for possible inclusion in VEE 7.0!

>

---

You are currently subscribed to vrf as: rsb@soco.agilent.com

To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".

To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".

To send messages to this mailing list, email "vrf@agilent.com".

If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".

Send your favorite VEE example to "VRF-EXAMPLES@agilent.com" for possible inclusion in VEE 7.0!

> This is not at all unexpected.

> The real32 has 32 bits to deal with mantissa and exponent.

> You are attempting to use 32 bits of mantissa accuracy alone.

> That can't be done. The system (Windows, UNIX, whatever) -

> not VEE, rounds off the number to the closest real32 it has.

> (Which then, of course, is too big for the int32)

>

> Les Hammer

Les is right, though I misread some of the values and thought we had a bug at first.

To put a little meat on the bones of Les's explanation for the diehards, the Real32 allocates 23 (actually 24 through the magic of "normalization") bits to the "mantissa", and the other 9 bits to the sign and exponent (which don't help the accuracy). The mantissa is where the "significant digits" (the accuracy) are stored in a Real.

For the Int32, there are 31 bits allocated to the significant digits, no exponent, and 1 bit to the sign (not quite true, but close enough. The integer scheme is called 2's complement, where the topmost bit has a negative weight.)

So when you try to "squeeze" the 31 bits (binary digits) of a big Int32 like 2147483647 into the 24 mantissa bits of a Real32, you have to throw away 7 bits and approximate the result in the Real32. That means that the Real32 has become a somewhat inaccurate representation of the original Int32.

How inaccurate?

The 7 bits we threw away have a value of 2^7 = 128. That's just a bit more than 2 decimal digits of accuracy that we've sacrificed in the squeeze, and it means that a whole bunch of different large Int32 values will be mapped to the same Real32 approximation. In this case, the values mapped to a value of 2147483648.0000 per the IEEE-754 math spec, which happens to be 1 unit more than the max positive value an Int32 can hold.

If you feed in the input value 2147483600 instead of 2147483647, you'll get the same error and see that the Real32 is the same 2147483648.0000. That's what I meant when I said a bunch of different Int32s will map to the same Real32 approximation. I haven't found the crossover point (where a smaller number won't cause the error because it maps to a number representable in Int32), but it's obviously something less than 2147483600.

So why use Real32s if they're so darn inaccurate?

Because they have a larger dynamic range than integers, and because they use less RAM than Real64s.

E.g. you can represent 1.23E12 as a Real32, but you can't represent this number in an Int32. With Real formats, we've traded some accuracy for increased dynamic range, using the same number of bits. With Real64, we can keep better accuracy than Real32 (Real64 has 53 bits of mantissa when normalized, so all Int32s can be accurately represented and you don't get an error), but we use twice as much RAM. RAM usage may be significant for a big array. Real32 may also be faster in computation than Real64 as well, though this could depend heavily on hardware.

Choices, choices.

Sometimes you need high accuracy (especially when counting objects), sometimes high dynamic range (picofarads vs megavolts, anyone?) Sometimes, you need neither high accuracy nor high dynamic range (e.g. a 12-bit ADC). You may need to expend some effort to choose the most suitable and efficient data type for your task.

(R&D made me write this, humph!

Regards,

Scott Bayes

Software Technical Support

Agilent Technologies, Inc.

815 14th Street S.W.

Loveland, CO, U.S.A. 80537

970 679 3799 Tel

970 635 6867 Fax

> -----Original Message-----

> From: Les Hammer [mailto:Les.Hammer@CompleteTest.com]

> Sent: Monday, February 23, 2004 5:27 PM

> To: PDL-LISTS,VRF (A-Lists,unix1)

> Subject: [vrf] RE: Real32 Number Representation

>

>

> > Dontcha just love this?

> > I don't know if this is inherent in the floating point

> representation or

> is actually a vee bug.

>

>

> This is not at all unexpected.

> The real32 has 32 bits to deal with mantissa and exponent.

> You are attempting to use 32 bits of mantissa accuracy alone.

> That can't be done. The system (Windows, UNIX, whatever) -

> not VEE, rounds off the number to the closest real32 it has.

> (Which then, of course, is too big for the int32)

>

> Les Hammer

> Partner

> Complete Test

> 720 SW 14th St.

> Loveland, CO. 80537

> Les.Hammer@CompleteTest.com

>

>

>

> ---

> You are currently subscribed to vrf as: scott_bayes@agilent.com

> To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".

> To unsubscribe send a blank email to

> "leave-vrf@it.lists.it.agilent.com".

> To send messages to this mailing list, email "vrf@agilent.com".

> If you need help with the mailing list send a message to

> "owner-vrf@it.lists.it.agilent.com".

> Send your favorite VEE example to "VRF-EXAMPLES@agilent.com"

> for possible inclusion in VEE 7.0!

>

---

You are currently subscribed to vrf as: rsb@soco.agilent.com

To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".

To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".

To send messages to this mailing list, email "vrf@agilent.com".

If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".

Send your favorite VEE example to "VRF-EXAMPLES@agilent.com" for possible inclusion in VEE 7.0!

> I don't know if this is inherent in the floating point representation or

is actually a vee bug.

This is not at all unexpected.

The real32 has 32 bits to deal with mantissa and exponent.

You are attempting to use 32 bits of mantissa accuracy alone.

That can't be done. The system (Windows, UNIX, whatever) -

not VEE, rounds off the number to the closest real32 it has.

(Which then, of course, is too big for the int32)

Les Hammer

Partner

Complete Test

720 SW 14th St.

Loveland, CO. 80537

Les.Hammer@CompleteTest.com

---

You are currently subscribed to vrf as: rsb@soco.agilent.com

To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".

To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".

To send messages to this mailing list, email "vrf@agilent.com".

If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".

Send your favorite VEE example to "VRF-EXAMPLES@agilent.com" for possible inclusion in VEE 7.0!