AnsweredAssumed Answered

VRf: Data Type conversions...

Question asked by VRFuser on Jul 19, 1999
Timothy McFadden wrote:
>
> Hello All.
>
> I'm having a really hard time with the type conversions in HP VEE 5.01.
> It all started when I wrote a quick little Round function (I don't think
> VEE comes with one to round a real number).  I've attached a short little
> program as well.
>
> As most of us know (I believe this bug has been reported), there is a
> slight problem with floating-point calculations in VEE. 

No, such roundoff errors are inherent in floating-point math.  If you print
numbers to the same level of precision in a C program you'll see *exactly*
the same results.

The problem is that you are trying to represent a decimal value in binary.
This works okay for an integer number.  In floating point, numbers are
represented as fractions and will only be exactly the same as the decimal
number if that decimal number can be exactly represented as a sum of inverse
powers of 2.

There was an annoying cosmetic "feature" in earlier versions of VEE where they
displayed numbers in Alphanumerics and the like to greater than the reliable
precision of a floating-point number.  I was actually dumb enough to defend
this practice because all the quarrels over it focused on the inherent
floating-point errors, which are absolutely unavoidable. 

Actually, what the lab people finally did was backed off the default display
precision to one digit *less* than the reliable precision of a floating-point
number, rather than one *more* digit.  Duh, fighting over the wrong question.

> If you run my
> program, you will see that most of the numbers are incorrect (they are not
> in whole increments of 50m).  You can see how VEE is internally storing the
> numbers by the way they are displayed in the 'Logging AlphaNumeric' object.
>
> Here's why this is bad:  In my larger program, these are voltages that I
> send to an HP4145A SPA.  When the HP4145A receives a 16 digit value, the
> SRQ is set to #H44 (RQS and 'Illegal Program' bits are set).

What you're doing is still leaving the value in binary floating point format.
This means that if you can't represent the number as a sum of inverse powers
of 2, you'll still end up with that annoying tail "1" (or 99999...) no matter
how you try to massage it.

The trick is to convert it to a string.  This is just ASCII characters, and
is basically fixed in format.  You can swap out the formula box in your program
for a TO STRING object with the transaction:

   WRITE TEXT Round(A,4) STR:16 EOL

You can specify any level of precision desired, but once you exceed about 16
digits, as above, you're exceeding the resolution of a 64-bit floating point
number, and there is no way to avoid getting "junk".  It's something a little
less inviolable than, say, a law of physics.

Now VEE will tend to convert values back to numeric format if you shuffle that
string value around to other objects that expect a number, but you could put
the same transaction into a Direct I/O object and have no problems.

> I've figured out a way to get rid of the last '1' by converting the number
> to a string and then truncating the string to just a few characters.
> However, when I pass this string into my force-voltage function (with the
> input set to 'Real'), the last '1' pops up again.  Therefore, I can't
> even use that method!
>
> Does anyone have any experience with this?
>
> As always, any help would be greatly appreciated.
>
> -Tim
>
> ************************************************************
> Timothy McFadden
> Device Engineer, ATMEL Corp.
> Colorado Springs, Colorado
> Email: tmcfadden@atmel.com
> Desk: (719)540-3090 / Lab: (719)540-1564 / Pager: (719)636-7889
> ************************************************************

--
    Greg Goebel          ftp://fcext3.external.hp.com/dist/mxd/index.html
    HP MXD Marketing     http://hpislsup.lvld.hp.com (HP only)
    pctm@lvld.hp.com     http://www.geocities.com/CapeCanaveral/Launchpad/6000

Outcomes