AnsweredAssumed Answered

vrf VRF precision of mathematic operation

Question asked by rsb on May 23, 2003
<tw@huber-online.com> wrote:
> VRF
>
> Hello,
> I have a mathematic operartion
> Y= 17.77-(1133/X)-(4.214*10^(-5)*X)+6.752*10^(-6)*X^2-5.228*LOG10(X)
> Z=10^Y
>
> X is running from 133 to 283 in increasing 2 steps.
> When I compare the results, than they differ to the normal pocket pc (target),
> why ??

Because your calculator doesn't have the concept of data types- everything
is a real number. Most likely you're seeing integer vs real value issues.

Try making sure that you are using reals, as in

Y= 17.77-(1133./X)-(4.214*10^(-5.)*X)+6.752*10^(-6.)*X^2-5.228*LOG10(X)

While VEE appears typeless it really does have normal datatypes underneath.

Stan


--------------------------------------------------------------------------
Stan Bischof  Agilent Technologies  707-577-3994  stan_bischof@agilent.com
--------------------------------------------------------------------------

---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list,  email "vrf@it.lists.it.agilent.com". 
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".

Outcomes