AnsweredAssumed Answered

Formula bug and my spectrum analyzer threshold problem

Question asked by VRFuser on Jan 28, 1998
Troy Weeks wrote:

> Earlier I sent an out a help me email on the spectrum analyzer ignoring my
> commands to set threshold level and simply setting it to the current
> reference level.  Very annoying!
> In my program I use the  formula box to do some simple arithmetic
> computation.  I subtract two values. ( Namely  -34.4-30   ).  When the
> formula box computes this the result is 64.40000000000001...obscenely
> incorrect...but not the exact problem.  This is the value I put directly
> for the threshold value of the spectrum analyzer...the spectrum analyzer on
> seeing this incredibly long number of significant digits abruptly changes
> the threshold to my reference and says nothing to me.
> What am I going to do about this?

If you're using Direct I/O, just set the appropriate number of
significant digits using the appropriate REAL format.

BTW, the result above is not incorrect.  If you have decimal numbers
in binary representation you are only assured of having an exact match
if the fraction is a sum of reciprocal powers of two.   Given that you have
a 64-bit floating-point number representation you won't get the
perfect fractional result otherwise.

This is just as true in C and BASIC as it is in VEE, but it's just more
visible in VEE.  There's a document on our website that describes
floating-point formats and restrictions in detail.

[<>] regards -- gvg