AnsweredAssumed Answered

Re2: Formatting problem or bug?

Question asked by VRFuser on Sep 3, 1997

Regards,
        I don't seem to have the original of this message, but can understand
the problem from the context.
        One way to add a number to a text string and also format the number is
to use a TO STRING object.  If you use one transaction to define the text
portion, with NO EOL selected, and then the next transaction to format the
number into a string, you ccan control the format of the number.  For example,
if you want to get:
        RESULT = 0.6
and your input value (A) is actually 0.6000000001, you can use the following
trancactions in a TO STRING object:
        WRITE TEXT "RESULT = "
        WRITE TEXT A REAL STD:1 EOL
The second line will format the real with 1 significant digit (this could be
changed), or you could select FIXED to set a constant number of significant
digits.

Tom Groleau
HMSC, Tucson, Az
______________________________ Reply Separator _________________________________
Subject: Re: Formatting problem or bug?
Author:  BRUCE_WENNER@HP-USA-om22.om.hp.com at CCGATE
Date:    9/4/97 9:43 AM


    
     This problem is likely due to calculation rounding and type
     conversion.  Normally, numbers such as 0.6 display correctly as both
     numeric and text values.  This even appears to work when calculating
     0.6 by dividing the real 6.0 by 10.0.  However, when the Alloc Real
     object is used to generate a linear ramp, calculation rounding due to
     the computer's inability to exactly represent all numbers using binary
     floating point numbers results in a number that is slightly larger
     than 0.6 (0.6000000000000001 in this case).  In fact, the following
     two formulas will result in slightly different numbers:
    
        6.0/10.0   and   2.0/20.0*6.0
    
     Both should result in the value 0.6, but the second equation will have
     the same rounding error as the Alloc Real linear ramp in your example.
    
     The standard display objects will still display this value as 0.6
     since the standard numeric representation calls for 4 significant
     digits.  However, in your example you are performing a calculation
     that includes a text character.  Since the output from this formula
     will be a text string, the real input are converted to text before the
     additions are performed.  This conversion is internal and there is no
     "significant digits" property to prevent all binary digits from being
     converted.
    
     If you need to get around this problem,  you'll have to format the
     numeric value prior to the addition function.  Adding three formatted
     strings rather than accepting the internal conversion should result in
     the text string you need.
    
     Regards,
    
     Bruce Wenner
     HP St. Paul


______________________________ Reply Separator _________________________________
Subject: Formatting problem or bug?
Author:  Non-HP-I.U.T. (I.U.T.@t-online.de) at HP-USA/o2=mimegw3
Date:    9/4/97 6:22 AM


hpvxd_xc@hpislsup.lvld.hp.com

Formatting problem or bug?

Whenever I use a structure similiar to that in the included small
example
programm the output for certain real values becomes irregular,
independent from
the global number format.
I'm working with HP VEE 3.12  and WfW 3.11.

GNF eV.
Klaus-Peter Francke
Rudower Chaussee 5
D-12489 Berlin
eMail: I.U.T.@t-online.de



Outcomes