# Re2: Formatting problem or bug?

Question asked by VRFuser on Sep 3, 1997

Regards,
I don't seem to have the original of this message, but can understand
the problem from the context.
One way to add a number to a text string and also format the number is
to use a TO STRING object.  If you use one transaction to define the text
portion, with NO EOL selected, and then the next transaction to format the
number into a string, you ccan control the format of the number.  For example,
if you want to get:
RESULT = 0.6
and your input value (A) is actually 0.6000000001, you can use the following
trancactions in a TO STRING object:
WRITE TEXT "RESULT = "
WRITE TEXT A REAL STD:1 EOL
The second line will format the real with 1 significant digit (this could be
changed), or you could select FIXED to set a constant number of significant
digits.

Tom Groleau
HMSC, Tucson, Az
Subject: Re: Formatting problem or bug?
Author:  BRUCE_WENNER@HP-USA-om22.om.hp.com at CCGATE
Date:    9/4/97 9:43 AM

This problem is likely due to calculation rounding and type
conversion.  Normally, numbers such as 0.6 display correctly as both
numeric and text values.  This even appears to work when calculating
0.6 by dividing the real 6.0 by 10.0.  However, when the Alloc Real
object is used to generate a linear ramp, calculation rounding due to
the computer's inability to exactly represent all numbers using binary
floating point numbers results in a number that is slightly larger
than 0.6 (0.6000000000000001 in this case).  In fact, the following
two formulas will result in slightly different numbers:

6.0/10.0   and   2.0/20.0*6.0

Both should result in the value 0.6, but the second equation will have
the same rounding error as the Alloc Real linear ramp in your example.

The standard display objects will still display this value as 0.6
since the standard numeric representation calls for 4 significant
digits.  However, in your example you are performing a calculation
that includes a text character.  Since the output from this formula
will be a text string, the real input are converted to text before the
additions are performed.  This conversion is internal and there is no
"significant digits" property to prevent all binary digits from being
converted.

If you need to get around this problem,  you'll have to format the
strings rather than accepting the internal conversion should result in
the text string you need.

Regards,

Bruce Wenner
HP St. Paul

Subject: Formatting problem or bug?
Author:  Non-HP-I.U.T. (I.U.T.@t-online.de) at HP-USA/o2=mimegw3
Date:    9/4/97 6:22 AM

hpvxd_xc@hpislsup.lvld.hp.com

Formatting problem or bug?

Whenever I use a structure similiar to that in the included small
example
programm the output for certain real values becomes irregular,
independent from
the global number format.
I'm working with HP VEE 3.12  and WfW 3.11.

GNF eV.
Klaus-Peter Francke
Rudower Chaussee 5
D-12489 Berlin
eMail: I.U.T.@t-online.de