AnsweredAssumed Answered

vrf Round off accuracy problem.

Question asked by VRFuser on Oct 8, 2003
Try using a real64. If that doesn't get you close enough, you can perhaps do all your calculations using integer math (ie. scale up the values to be integers). Then divide out the scale factor before using the final result.



> -----Original Message-----
> From: Goldberg, Mark [mailto:Mark.Goldberg@ATK.COM]
> Sent: Wednesday, October 08, 2003 12:19 PM
> To: VEE vrf
> Subject: [vrf] Round off accuracy problem.
>
>
> Morning All,
>
> This question is so simple I feel silly asking it but, I have
> to use some
> constants in a formula that represent the number of steps for
> a servo motor
> and the number is not remaining constant. I put in 26009.6 in
> the real32
> constant and get a distorted value. I need the exact number
> not a rounded
> off or floating point hatchet number! Any suggestions? See
> the attached
> example.
>
>  <<constant problem.vee>>
>
> Mark Goldberg
> RF/Antenna Engineer
> Millimeter Wave & RF Design
>  <<...OLE_Obj...>>
> ATK Missile Systems Company LLC
> 21050 Califa Street
> Woodland Hills, CA 91367
> TEL:  818 887-0844 x 178
> FAX:  818 887-2366
> Mark.Goldberg@atk.com
>
>
>
>
>
> ---
> You are currently subscribed to vrf as: steve_asprelli@agilent.com
> To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
> To unsubscribe send a blank email to
> "leave-vrf@it.lists.it.agilent.com".
> To send messages to this mailing list,  email
> "vrf@it.lists.it.agilent.com". 
> If you need help with the mailing list send a message to
> "owner-vrf@it.lists.it.agilent.com".
>

---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list,  email "vrf@it.lists.it.agilent.com". 
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".

Outcomes