AnsweredAssumed Answered

vrf timestamp on scales

Question asked by fukui.yutaka on Jun 15, 2004
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"><HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-2022-jp"><META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45"><TITLE>Re: [vrf] is this a bug...</TITLE></HEAD><BODY><P><FONT SIZE=2>> INT32 value is out of range</FONT></P><P><FONT SIZE=2>If you keep your result as INT32, the maximum allowed will be 2^31, leaving room for the sign bit.  If you really want to keep the result as INT32 and your formula is for powers of 2, you may consider the bitShift() formula.  I don't know if VEE 6 or VEE 7 have a type for unsigned INT32.</FONT></P><P><FONT SIZE=2>-----Message d'origine-----</FONT><BR><FONT SIZE=2>De: Mike Groves [<A HREF="mailto:mikegroves@pacbell.net">mailto:mikegroves@pacbell.net</A>]</FONT><BR><FONT SIZE=2>Date: 16 juin 2004 05:29</FONT></P><P><FONT SIZE=2>The reasoning is simple. If you are doing a math function with two</FONT><BR><FONT SIZE=2>or more integers, VEE will keep the result as an integer.  As soon</FONT><BR><FONT SIZE=2>as you make one or more numbers real, you allow VEE the full range</FONT><BR><FONT SIZE=2>of real numbers for the result.</FONT></P><P><FONT SIZE=2>Adding a decimal point is a simple way to force VEE out of integer</FONT><BR><FONT SIZE=2>mode.</FONT></P><BR><P><FONT SIZE=2>-----Original Message-----</FONT><BR><FONT SIZE=2>From: FUKUI Yutaka [<A HREF="mailto:fukui.yutaka@jp.fujitsu.com">mailto:fukui.yutaka@jp.fujitsu.com</A>]</FONT><BR><FONT SIZE=2>Sent: Wednesday, June 16, 2004 2:09 AM</FONT></P><P><FONT SIZE=2>I add another information.</FONT></P><P><FONT SIZE=2>Both fomulas 2.^32 and 2^32. run fine and give the correct </FONT></P><P><FONT SIZE=2>answer of 4.295G.</FONT></P><BR><P><FONT SIZE=2>> Nick Evans wrote:</FONT><BR><FONT SIZE=2>> </FONT><BR><FONT SIZE=2>> or am I being stupid.</FONT><BR><FONT SIZE=2>> </FONT><BR><FONT SIZE=2>> If I insert a simple formula object with the formula 2^32  and run it,</FONT><BR><FONT SIZE=2>> the formula box errors with a message saying that "INT32 value is out</FONT><BR><FONT SIZE=2>> of range error number 403"  However, if I type in the formula</FONT><BR><FONT SIZE=2>> asreal64(2)^32 the formula runs fine and gives the correct answer of</FONT><BR><FONT SIZE=2>> 4.29G.</FONT><BR><FONT SIZE=2>> </FONT><BR><FONT SIZE=2>> This is really no problem for me but I am just curious why the formula</FONT><BR><FONT SIZE=2>> box defaults to an INT32 calculation and if this is specified</FONT><BR><FONT SIZE=2>> behaviour?</FONT></P>---<BR>You are currently subscribed to vrf as: rsb@soco.agilent.com<BR>To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".<BR>To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".<BR>To send messages to this mailing list,  email "vrf@agilent.com".  <BR>If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".</BODY></HTML>  

Outcomes