Try using a real64. If that doesn't get you close enough, you can perhaps do all your calculations using integer math (ie. scale up the values to be integers). Then divide out the scale factor before using the final result.
> -----Original Message-----
> From: Goldberg, Mark [mailto:Mark.Goldberg@ATK.COM]
> Sent: Wednesday, October 08, 2003 12:19 PM
> To: VEE vrf
> Subject: [vrf] Round off accuracy problem.
>
>
> Morning All,
>
> This question is so simple I feel silly asking it but, I have
> to use some
> constants in a formula that represent the number of steps for
> a servo motor
> and the number is not remaining constant. I put in 26009.6 in
> the real32
> constant and get a distorted value. I need the exact number
> not a rounded
> off or floating point hatchet number! Any suggestions? See
> the attached
> example.
>
> <<constant problem.vee>>
>
> Mark Goldberg
> RF/Antenna Engineer
> Millimeter Wave & RF Design
> <<...OLE_Obj...>>
> ATK Missile Systems Company LLC
> 21050 Califa Street
> Woodland Hills, CA 91367
> TEL: 818 887-0844 x 178
> FAX: 818 887-2366
> Mark.Goldberg@atk.com
>
>
>
>
>
> ---
> You are currently subscribed to vrf as: steve_asprelli@agilent.com
> To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
> To unsubscribe send a blank email to
> "leave-vrf@it.lists.it.agilent.com".
> To send messages to this mailing list, email
> "vrf@it.lists.it.agilent.com".
> If you need help with the mailing list send a message to
> "owner-vrf@it.lists.it.agilent.com".
>
---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list, email "vrf@it.lists.it.agilent.com".
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".
> -----Original Message-----
> From: Goldberg, Mark [mailto:Mark.Goldberg@ATK.COM]
> Sent: Wednesday, October 08, 2003 12:19 PM
> To: VEE vrf
> Subject: [vrf] Round off accuracy problem.
>
>
> Morning All,
>
> This question is so simple I feel silly asking it but, I have
> to use some
> constants in a formula that represent the number of steps for
> a servo motor
> and the number is not remaining constant. I put in 26009.6 in
> the real32
> constant and get a distorted value. I need the exact number
> not a rounded
> off or floating point hatchet number! Any suggestions? See
> the attached
> example.
>
> <<constant problem.vee>>
>
> Mark Goldberg
> RF/Antenna Engineer
> Millimeter Wave & RF Design
> <<...OLE_Obj...>>
> ATK Missile Systems Company LLC
> 21050 Califa Street
> Woodland Hills, CA 91367
> TEL: 818 887-0844 x 178
> FAX: 818 887-2366
> Mark.Goldberg@atk.com
>
>
>
>
>
> ---
> You are currently subscribed to vrf as: steve_asprelli@agilent.com
> To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
> To unsubscribe send a blank email to
> "leave-vrf@it.lists.it.agilent.com".
> To send messages to this mailing list, email
> "vrf@it.lists.it.agilent.com".
> If you need help with the mailing list send a message to
> "owner-vrf@it.lists.it.agilent.com".
>
---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list, email "vrf@it.lists.it.agilent.com".
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".
> of steps for a servo motor and the number is not remaining constant.
> I put in 26009.6 in the real32 constant and get a distorted value.
Unfortuantely that's the way real32 behaves. There's no way to represent
most numbers *exactly*. Formulas and instruments need to be tolerant of
approximations.
Can you modify your formula to have 260096 as an int32 input and divide
by 10 in the formula?
Les Hammer
Vice President
Complete Test
---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list, email "vrf@it.lists.it.agilent.com".
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".