AnsweredAssumed Answered

vrf Rounding Numbers

Question asked by VRFuser on Jun 17, 2003
  • Comment • 6
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"><HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"><META content="MSHTML 6.00.2713.1100" name=GENERATOR><STYLE></STYLE></HEAD><BODY style="COLOR: #0000ff; FONT-FAMILY: Tahoma" bgColor=#ffffff><DIV><SPAN class=741585512-18062003><FONT size=2>To fix the 1.2399999 problem,try this: <FONTface=Arial>(intpart(a*10000+0.5)/10000)</FONT></FONT></SPAN></DIV><BLOCKQUOTE>  <DIV><FONT face="Times New Roman" color=#000000 size=2>-----Message  d'origine-----<BR><B>De:</B> Kathy Kiessling  [mailto:kathy_kiessling@yahoo.com]<BR><B>Date:</B> 18 juin 2003  01:13<BR><BR></FONT></DIV>  <DIV><FONT face=Arial size=2>First observation is that your formula is  </FONT></DIV>  <DIV><FONT face=Arial size=2></FONT> </DIV>  <DIV><FONT face=Arial size=2>(intpart(a*10000)/10000) </FONT></DIV>  <DIV><FONT face=Arial size=2></FONT> </DIV>  <DIV><FONT face=Arial size=2>rather than</FONT></DIV>  <DIV><FONT face=Arial size=2></FONT> </DIV>  <DIV><FONT face=Arial size=2>(round(a*10000)/10000)</FONT></DIV>  <DIV><FONT face=Arial size=2></FONT> </DIV>  <DIV><FONT face=Arial size=2>The difference becomes apparent on the last  digit, since intpart will always go low.  </FONT></DIV>  <DIV><FONT face=Arial size=2></FONT> </DIV>  <DIV><FONT face=Arial size=2>For the occasional 1.2399999.  9.11 does the  same thing, and if you enter it in your Real constant box with the fixed 99  display you had set up, then the phenomenon is visible.  The result is OK  after rounding.  Why does it happen in the first place?  I think it  has to do with the representation of real numbers using floating point  representation, which is sometimes not exact.  Last time I tried to look  this up in order to understand and explain it better, I was up to 2 am so I'll  just stop here.</FONT></DIV>  <DIV><FONT face=Arial color=#000000 size=2></FONT> </DIV>  <BLOCKQUOTE  style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">    <DIV style="FONT: 10pt arial">----- Original Message ----- </DIV>    <DIV    style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>    <A title=RBLEDSOE@Coinco.com href="mailto:RBLEDSOE@Coinco.com">Ross    Bledsoe</A> </DIV>    <DIV style="FONT: 10pt arial"><B>Sent:</B> Tuesday, June 17, 2003 1:44    PM</DIV>    <DIV><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial    color=#000000 size=2></FONT><FONT face=Arial color=#000000    size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><BR></DIV>    <DIV><SPAN class=320383920-17062003><FONT face=Tahoma color=#0000ff    size=2>The formula that I use in the attached program works 99.999% of the    time. I occasionally get a number back like 1.23399999999999 instead of    1.234.  I tried numerous solutions to alleviate the problem to no    avail.  Any ideas</FONT></SPAN></DIV>    <DIV><SPAN class=320383920-17062003><FONT face=Tahoma color=#0000ff    size=2></FONT></SPAN><SPAN class=320383920-17062003><FONT face=Tahoma    color=#0000ff size=2>Ross</FONT></SPAN></DIV>    <BLOCKQUOTE>      <DIV class=OutlookMessageHeader><FONT face="Times New Roman"      size=2>-----Original Message-----<BR><B>From:</B> Kathy Kiessling      [mailto:kathy_kiessling@yahoo.com]<BR><B>Sent:</B> Monday, June 16, 2003      9:42 PM<BR></FONT></DIV><FONT face=Arial color=#000000 size=2>      <DIV><FONT face=Tahoma color=#0000ff></FONT><FONT face="Times New Roman"      color=#0000ff></FONT><BR>Here are two ways to do it.</DIV>      <DIV><FONT face=Tahoma color=#0000ff></FONT> </DIV></FONT>      <BLOCKQUOTE      style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">        <DIV style="FONT: 10pt arial">----- Original Message ----- </DIV>        <DIV        style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>        <A title=dwilson@pwscom.com href="mailto:dwilson@pwscom.com">David        Wilson</A> </DIV>        <DIV style="FONT: 10pt arial"><B>Sent:</B> Monday, June 16, 2003 5:32        PM</DIV>        <DIV><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial        color=#000000 size=2></FONT><FONT face=Arial color=#000000        size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><FONT        face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000        size=2></FONT><BR></DIV>        <DIV><SPAN class=118332700-05072003>Does anyone know how to get VEE to        eliminate unwanted numbers after a decimal point. I am capturing        data from an instrument and only want to display the        first decimal point.  The round(x) function will return        only whole numbers which are useless for me.  </SPAN></DIV>        <DIV><SPAN class=118332700-05072003><FONT face=Arial color=#000000        size=2></FONT></SPAN> </DIV>        <DIV><SPAN        class=118332700-05072003>    For example, I am        pinging a current reading from a power supply that reads 6.7863 and only        want to display 6.7 on my VEE front panel.  </SPAN></DIV>        <DIV><FONT face=Arial color=#000000    size=2></FONT> </DIV></BLOCKQUOTE></BLOCKQUOTE>    <P></P><FONT size=2></FONT>    <HR>  </BLOCKQUOTE></BLOCKQUOTE>---<BR>You are currently subscribed to vrf as: rsb@soco.agilent.com<BR>To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".<BR>To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".<BR>To send messages to this mailing list,  email "vrf@it.lists.it.agilent.com".  <BR>If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".</BODY></HTML>  
No one else has this question

Outcomes