AnsweredAssumed Answered

vrf - rounding and displays

Question asked by VRFuser on Nov 9, 1997
> Greetings HPVee Users,

> Sometimes there are just too many significant digits, and I would like =
> to chop all of them off except for 2.  My Default preferences have been =
> set up for Decimal, Fixed, 2??however I continue to get about a dozen =
> significant digits as the output from some of my formulas.

With some math you can easily round your numbers to some arbitrary accuracy.
See the accompaning vee program. It works on scalar data and on arrays of
(I think) any dimenison. HOWEVER they are 'computer numbers' a are bound to
inaccuracy. So you will always have e.g. 1.2300000000001 or 1.23999999999!
When you do a comparison, all you can is use:
  - the ~= operator: almost equal too
  - something like A < B+A_small_number_relative_to_you_accuracy

Remi.

------------------------------------------------
Remi Tuijtelaars,
   PHILIPS SEMICONDUCTORS,  DEV. DSC-N, BZ-2.090
   Gerstweg 2, 6534 AE Nijmegen, The Netherlands
   tel: +31-24-353-2812     fax: +31-24-353-3160
    mailto: Remi.Tuijtelaars@nym.sc.philips.com
------------------------------------------------

Outcomes