AnsweredAssumed Answered

VRf-VEE_Numeric_Precision

Question asked by VRFuser on Feb 20, 1996
from: Greg Goebel / HP-MXD
      gvg@lvld.hp.com / 970-679-3030 / FAX 970-679-5971
to:   Kevin Roke / VRf
date: Wednesday, 21 February 1996 1013 MST

> I am using HP VEE to develop calibration tests for the Brunei Min Def
> Calibration Centre (I obtained our first copy of the software 3 months ago).
> When printing out a results sheet I am having problems printing some REAL
> values of limit data.
>
> For example:
>
> Set a REAL constant to the value 8.9972
> Now print it.
>
> The result is 8.997199999999999.
>
> Has anyone had this problem?

I presume you are not referring to a PRINT SCREEN, since that simply prints
out what you see, and mean you are going through a To Printer object, since
you are getting what I would expect of its default behavior.

OK ... here's how this works.  What you are seeing is the full precision of
the number as VEE understands it.  (It isn't exactly 8.9972 because you are
approximating a decimal number with a binary number, and you don't get an
exact match between the two for fractional quantities unless they are a sum
of inverse powers of 2 -- 1/4, 1/8, 1/14+1/8, and so on.)

The reason it is being printed in this full-precision / no roundoff format is
because the default format on To Printer (and most other transaction objects)
is:  WRITE TEXT A EOL, which implies "standard format":  a format that will
accept *any* input -- text or numeric -- and pass it out as best it can.

Now if you want to print out a specific precision, set this to:  WRITE TEXT
A REAL STD:6 EOL -- or whatever precision you like. 

> I can replace the REAL with a TEXT constant and get the correct print out but
> in my actual application the number is input by an operator and is checked
> against limits and accidental text input is prevented.
>
> I have also found that by increasing the number of significant digits to the
> maximum (under the File -> Edit Default Preferences) it is impossible to
> input 8.9972.

I presume you mean that you get the extended format ... 8.9971999999.  Again,
this is extended format, the box is showing you what it thinks it has.   Or,
simply put, it's doing what it was told to do.  But you can set the precision
to what you like.

This issue is common to any software that uses binary floating-point math.
It is, however, much more visible in VEE than in C, because in VEE you get
to see it happen, while in C you have to print things out to see them!

If you have more questions, let us know.

> Is there a fix?
>
> I have written a Userfunction to manipulate the number text string to give me
> the required value but is it necessary?

No, just provide the appropriate precision in the transaction objects you are
using.

> Kevin Roke
>
> Brunei Darussalam

[<>] regards -- gvg

Outcomes