> Greetings HPVee Users,
> Sometimes there are just too many significant digits, and I would like =
> to chop all of them off except for 2. My Default preferences have been =
> set up for Decimal, Fixed, 2??however I continue to get about a dozen =
> significant digits as the output from some of my formulas.
With some math you can easily round your numbers to some arbitrary accuracy.
See the accompaning vee program. It works on scalar data and on arrays of
(I think) any dimenison. HOWEVER they are 'computer numbers' a are bound to
inaccuracy. So you will always have e.g. 1.2300000000001 or 1.23999999999!
When you do a comparison, all you can is use:
- the ~= operator: almost equal too
- something like A < B+A_small_number_relative_to_you_accuracy
Remi.
------------------------------------------------
Remi Tuijtelaars,
PHILIPS SEMICONDUCTORS, DEV. DSC-N, BZ-2.090
Gerstweg 2, 6534 AE Nijmegen, The Netherlands
tel: +31-24-353-2812 fax: +31-24-353-3160
mailto: Remi.Tuijtelaars@nym.sc.philips.com
------------------------------------------------
> Sometimes there are just too many significant digits, and I would like =
> to chop all of them off except for 2. My Default preferences have been =
> set up for Decimal, Fixed, 2??however I continue to get about a dozen =
> significant digits as the output from some of my formulas.
With some math you can easily round your numbers to some arbitrary accuracy.
See the accompaning vee program. It works on scalar data and on arrays of
(I think) any dimenison. HOWEVER they are 'computer numbers' a are bound to
inaccuracy. So you will always have e.g. 1.2300000000001 or 1.23999999999!
When you do a comparison, all you can is use:
- the ~= operator: almost equal too
- something like A < B+A_small_number_relative_to_you_accuracy
Remi.
------------------------------------------------
Remi Tuijtelaars,
PHILIPS SEMICONDUCTORS, DEV. DSC-N, BZ-2.090
Gerstweg 2, 6534 AE Nijmegen, The Netherlands
tel: +31-24-353-2812 fax: +31-24-353-3160
mailto: Remi.Tuijtelaars@nym.sc.philips.com
------------------------------------------------