Timothy McFadden wrote:

>

> Hello All.

>

> I'm having a really hard time with the type conversions in HP VEE 5.01.

> It all started when I wrote a quick little Round function (I don't think

> VEE comes with one to round a real number). I've attached a short little

> program as well.

>

> As most of us know (I believe this bug has been reported), there is a

> slight problem with floating-point calculations in VEE.

No, such roundoff errors are inherent in floating-point math. If you print

numbers to the same level of precision in a C program you'll see *exactly*

the same results.

The problem is that you are trying to represent a decimal value in binary.

This works okay for an integer number. In floating point, numbers are

represented as fractions and will only be exactly the same as the decimal

number if that decimal number can be exactly represented as a sum of inverse

powers of 2.

There was an annoying cosmetic "feature" in earlier versions of VEE where they

displayed numbers in Alphanumerics and the like to greater than the reliable

precision of a floating-point number. I was actually dumb enough to defend

this practice because all the quarrels over it focused on the inherent

floating-point errors, which are absolutely unavoidable.

Actually, what the lab people finally did was backed off the default display

precision to one digit *less* than the reliable precision of a floating-point

number, rather than one *more* digit. Duh, fighting over the wrong question.

> If you run my

> program, you will see that most of the numbers are incorrect (they are not

> in whole increments of 50m). You can see how VEE is internally storing the

> numbers by the way they are displayed in the 'Logging AlphaNumeric' object.

>

> Here's why this is bad: In my larger program, these are voltages that I

> send to an HP4145A SPA. When the HP4145A receives a 16 digit value, the

> SRQ is set to #H44 (RQS and 'Illegal Program' bits are set).

What you're doing is still leaving the value in binary floating point format.

This means that if you can't represent the number as a sum of inverse powers

of 2, you'll still end up with that annoying tail "1" (or 99999...) no matter

how you try to massage it.

The trick is to convert it to a string. This is just ASCII characters, and

is basically fixed in format. You can swap out the formula box in your program

for a TO STRING object with the transaction:

WRITE TEXT Round(A,4) STR:16 EOL

You can specify any level of precision desired, but once you exceed about 16

digits, as above, you're exceeding the resolution of a 64-bit floating point

number, and there is no way to avoid getting "junk". It's something a little

less inviolable than, say, a law of physics.

Now VEE will tend to convert values back to numeric format if you shuffle that

string value around to other objects that expect a number, but you could put

the same transaction into a Direct I/O object and have no problems.

> I've figured out a way to get rid of the last '1' by converting the number

> to a string and then truncating the string to just a few characters.

> However, when I pass this string into my force-voltage function (with the

> input set to 'Real'), the last '1' pops up again. Therefore, I can't

> even use that method!

>

> Does anyone have any experience with this?

>

> As always, any help would be greatly appreciated.

>

> -Tim

>

> ************************************************************

> Timothy McFadden

> Device Engineer, ATMEL Corp.

> Colorado Springs, Colorado

> Email: tmcfadden@atmel.com

> Desk: (719)540-3090 / Lab: (719)540-1564 / Pager: (719)636-7889

> ************************************************************

--

Greg Goebel ftp://fcext3.external.hp.com/dist/mxd/index.html

HP MXD Marketing http://hpislsup.lvld.hp.com (HP only)

pctm@lvld.hp.com http://www.geocities.com/CapeCanaveral/Launchpad/6000

>

> Hello All.

>

> I'm having a really hard time with the type conversions in HP VEE 5.01.

> It all started when I wrote a quick little Round function (I don't think

> VEE comes with one to round a real number). I've attached a short little

> program as well.

>

> As most of us know (I believe this bug has been reported), there is a

> slight problem with floating-point calculations in VEE.

No, such roundoff errors are inherent in floating-point math. If you print

numbers to the same level of precision in a C program you'll see *exactly*

the same results.

The problem is that you are trying to represent a decimal value in binary.

This works okay for an integer number. In floating point, numbers are

represented as fractions and will only be exactly the same as the decimal

number if that decimal number can be exactly represented as a sum of inverse

powers of 2.

There was an annoying cosmetic "feature" in earlier versions of VEE where they

displayed numbers in Alphanumerics and the like to greater than the reliable

precision of a floating-point number. I was actually dumb enough to defend

this practice because all the quarrels over it focused on the inherent

floating-point errors, which are absolutely unavoidable.

Actually, what the lab people finally did was backed off the default display

precision to one digit *less* than the reliable precision of a floating-point

number, rather than one *more* digit. Duh, fighting over the wrong question.

> If you run my

> program, you will see that most of the numbers are incorrect (they are not

> in whole increments of 50m). You can see how VEE is internally storing the

> numbers by the way they are displayed in the 'Logging AlphaNumeric' object.

>

> Here's why this is bad: In my larger program, these are voltages that I

> send to an HP4145A SPA. When the HP4145A receives a 16 digit value, the

> SRQ is set to #H44 (RQS and 'Illegal Program' bits are set).

What you're doing is still leaving the value in binary floating point format.

This means that if you can't represent the number as a sum of inverse powers

of 2, you'll still end up with that annoying tail "1" (or 99999...) no matter

how you try to massage it.

The trick is to convert it to a string. This is just ASCII characters, and

is basically fixed in format. You can swap out the formula box in your program

for a TO STRING object with the transaction:

WRITE TEXT Round(A,4) STR:16 EOL

You can specify any level of precision desired, but once you exceed about 16

digits, as above, you're exceeding the resolution of a 64-bit floating point

number, and there is no way to avoid getting "junk". It's something a little

less inviolable than, say, a law of physics.

Now VEE will tend to convert values back to numeric format if you shuffle that

string value around to other objects that expect a number, but you could put

the same transaction into a Direct I/O object and have no problems.

> I've figured out a way to get rid of the last '1' by converting the number

> to a string and then truncating the string to just a few characters.

> However, when I pass this string into my force-voltage function (with the

> input set to 'Real'), the last '1' pops up again. Therefore, I can't

> even use that method!

>

> Does anyone have any experience with this?

>

> As always, any help would be greatly appreciated.

>

> -Tim

>

> ************************************************************

> Timothy McFadden

> Device Engineer, ATMEL Corp.

> Colorado Springs, Colorado

> Email: tmcfadden@atmel.com

> Desk: (719)540-3090 / Lab: (719)540-1564 / Pager: (719)636-7889

> ************************************************************

--

Greg Goebel ftp://fcext3.external.hp.com/dist/mxd/index.html

HP MXD Marketing http://hpislsup.lvld.hp.com (HP only)

pctm@lvld.hp.com http://www.geocities.com/CapeCanaveral/Launchpad/6000

>

> Hello All.

>

> I'm having a really hard time with the type conversions in HP VEE 5.01.

> It all started when I wrote a quick little Round function (I don't think

> VEE comes with one to round a real number). I've attached a short little

> program as well.

>

> As most of us know (I believe this bug has been reported), there is a

> slight problem with floating-point calculations in VEE.

No, such roundoff errors are inherent in floating-point math. If you print

numbers to the same level of precision in a C program you'll see *exactly*

the same results.

The problem is that you are trying to represent a decimal value in binary.

This works okay for an integer number. In floating point, numbers are

represented as fractions and will only be exactly the same as the decimal

number if that decimal number can be exactly represented as a sum of inverse

powers of 2.

There was an annoying cosmetic "feature" in earlier versions of VEE where they

displayed numbers in Alphanumerics and the like to greater than the reliable

precision of a floating-point number. I was actually dumb enough to defend

this practice because all the quarrels over it focused on the inherent

floating-point errors, which are absolutely unavoidable.

Actually, what the lab people finally did was backed off the default display

precision to one digit *less* than the reliable precision of a floating-point

number, rather than one *more* digit. Duh, fighting over the wrong question.

> If you run my

> program, you will see that most of the numbers are incorrect (they are not

> in whole increments of 50m). You can see how VEE is internally storing the

> numbers by the way they are displayed in the 'Logging AlphaNumeric' object.

>

> Here's why this is bad: In my larger program, these are voltages that I

> send to an HP4145A SPA. When the HP4145A receives a 16 digit value, the

> SRQ is set to #H44 (RQS and 'Illegal Program' bits are set).

What you're doing is still leaving the value in binary floating point format.

This means that if you can't represent the number as a sum of inverse powers

of 2, you'll still end up with that annoying tail "1" (or 99999...) no matter

how you try to massage it.

The trick is to convert it to a string. This is just ASCII characters, and

is basically fixed in format. You can swap out the formula box in your program

for a TO STRING object with the transaction:

WRITE TEXT Round(A,4) STR:16 EOL

You can specify any level of precision desired, but once you exceed about 16

digits, as above, you're exceeding the resolution of a 64-bit floating point

number, and there is no way to avoid getting "junk". It's something a little

less inviolable than, say, a law of physics.

Now VEE will tend to convert values back to numeric format if you shuffle that

string value around to other objects that expect a number, but you could put

the same transaction into a Direct I/O object and have no problems.

> I've figured out a way to get rid of the last '1' by converting the number

> to a string and then truncating the string to just a few characters.

> However, when I pass this string into my force-voltage function (with the

> input set to 'Real'), the last '1' pops up again. Therefore, I can't

> even use that method!

>

> Does anyone have any experience with this?

>

> As always, any help would be greatly appreciated.

>

> -Tim

>

> ************************************************************

> Timothy McFadden

> Device Engineer, ATMEL Corp.

> Colorado Springs, Colorado

> Email: tmcfadden@atmel.com

> Desk: (719)540-3090 / Lab: (719)540-1564 / Pager: (719)636-7889

> ************************************************************

--

Greg Goebel ftp://fcext3.external.hp.com/dist/mxd/index.html

HP MXD Marketing http://hpislsup.lvld.hp.com (HP only)

pctm@lvld.hp.com http://www.geocities.com/CapeCanaveral/Launchpad/6000