AnsweredAssumed Answered

vrf RS-232 Run Time Error

Question asked by adam_kohler on Dec 21, 1999
> Subject: Re: vrf:help with exponantiation
>
> The problem is that the output of a^b is "a container of the highest
> type" of its inputs; ie if both inputs are int32, so is the output. 0.2
> as an int32 = 0, so the result it gives you is "correct" - but the
> output you require is real. If the input B of the formula box is cast to
> real (see attached example), the output will also be real, and you will
> get the correct result.

If at least one of the operands is of type real, the expression will be
evaluated as a real and will return the desired result.  Casting one or
more inputs as mentioned above is one way to force an operand to be of
real type.  A couple of other possibilities are:

Use a real constant in the expression:

A*10.0^B instead of A*10^B

Or use the asReal function to convert the integer to a real as needed:

A*10^asReal(B)

Any of the three methods will correct your problem.

Regards,

Bruce Wenner
Agilent Technologies

---------------------------------------------------------------------
This is the "vrf" maillist, managed by Majordomo.  To send messages to
this maillist, just email to "vrf@lvld.hp.com".  Subscriptions and
unsubscriptions are done through the address "vrf-request@lvld.hp.com".
If you need details, just send a message containing the text "help"
to "vrf-request@lvld.hp.com".
---------------------------------------------------------------------

Outcomes