AnsweredAssumed Answered

Strange results with conversion of Reals

Question asked by VRFuser on Aug 30, 2001
Hi Sander,

try a To-String Object with the transaction: WRITE TEXT a REAL64 FIX:2.
You can also set the number of fract digits with a variable.

Best regards
M. Bernhardt
-----------------------------------------
Somatec B. Aufenanger & M. Bernhardt
Jagdweg 7a
D-82041 Deisenhofen

Telefon       : (089) 613 984 94
Fax           : (089) 613 981 23
Email           : mailto:Bernhardt@somatec.de
homepage     : http://www.somatec.com
-----------------------------------------


-----Ursprngliche Nachricht-----
Von: owner-vrf@lvld.agilent.com [mailto:owner-vrf@lvld.agilent.com]Im
Auftrag von sander.wubbels@philips.com
Gesendet: Freitag, 31. August 2001 11:24
An: vrf@lvld.agilent.com
Betreff: vrf Strange results with conversion of Reals


Hello,

When I try to convert a real64 to a textstring, The result is always a
string with an awful lot of digits. I want to round the string to a certain
number of digits.

If my real is 2.5 the 'asText' formula returns: "2.500010010001" It should,
in my opinion, return"2.5" or "2.50"

Does anybody know how to solve this problem?

Regards,

Sander Wubbels

-------------------------------------------------------------------------
       S. Wubbels
       Test Engineer

                MST - Personal Communications
                Philips Semiconductors
                Gerstweg 2 Nijmegen
                Tel.: (+35) (0)24 353 29 90
                E-mail: Sander.Wubbels@philips.com
-------------------------------------------------------------------------


---------------------------------------------------------------------
This is the "vrf" maillist, managed by Majordomo.  To send messages to
this maillist, just email to "vrf@lvld.agilent.com".  Subscriptions and
unsubscriptions are done through the address "vrf-request@lvld.agilent.com".
If you need details, just send a message containing the text "help"
to "vrf-request@lvld.agilent.com".
---------------------------------------------------------------------
---------------------------------------------------------------------
This is the "vrf" maillist, managed by Majordomo.  To send messages to
this maillist, just email to "vrf@lvld.agilent.com".  Subscriptions and
unsubscriptions are done through the address "vrf-request@lvld.agilent.com".
If you need details, just send a message containing the text "help"
to "vrf-request@lvld.agilent.com".
---------------------------------------------------------------------

Outcomes