<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"><HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"><META content="MSHTML 6.00.2713.1100" name=GENERATOR><STYLE></STYLE></HEAD><BODY style="COLOR: #0000ff; FONT-FAMILY: Tahoma" bgColor=#ffffff><DIV><SPAN class=741585512-18062003><FONT size=2>To fix the 1.2399999 problem,try this: <FONTface=Arial>(intpart(a*10000+0.5)/10000)</FONT></FONT></SPAN></DIV><BLOCKQUOTE> <DIV><FONT face="Times New Roman" color=#000000 size=2>-----Message d'origine-----<BR><B>De:</B> Kathy Kiessling [mailto:kathy_kiessling@yahoo.com]<BR><B>Date:</B> 18 juin 2003 01:13<BR><BR></FONT></DIV> <DIV><FONT face=Arial size=2>First observation is that your formula is </FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>(intpart(a*10000)/10000) </FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>rather than</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>(round(a*10000)/10000)</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>The difference becomes apparent on the last digit, since intpart will always go low. </FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>For the occasional 1.2399999. 9.11 does the same thing, and if you enter it in your Real constant box with the fixed 99 display you had set up, then the phenomenon is visible. The result is OK after rounding. Why does it happen in the first place? I think it has to do with the representation of real numbers using floating point representation, which is sometimes not exact. Last time I tried to look this up in order to understand and explain it better, I was up to 2 am so I'll just stop here.</FONT></DIV> <DIV><FONT face=Arial color=#000000 size=2></FONT> </DIV> <BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px"> <DIV style="FONT: 10pt arial">----- Original Message ----- </DIV> <DIV style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B> <A title=RBLEDSOE@Coinco.com href="mailto:RBLEDSOE@Coinco.com">Ross Bledsoe</A> </DIV> <DIV style="FONT: 10pt arial"><B>Sent:</B> Tuesday, June 17, 2003 1:44 PM</DIV> <DIV><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><BR></DIV> <DIV><SPAN class=320383920-17062003><FONT face=Tahoma color=#0000ff size=2>The formula that I use in the attached program works 99.999% of the time. I occasionally get a number back like 1.23399999999999 instead of 1.234. I tried numerous solutions to alleviate the problem to no avail. Any ideas</FONT></SPAN></DIV> <DIV><SPAN class=320383920-17062003><FONT face=Tahoma color=#0000ff size=2></FONT></SPAN><SPAN class=320383920-17062003><FONT face=Tahoma color=#0000ff size=2>Ross</FONT></SPAN></DIV> <BLOCKQUOTE> <DIV class=OutlookMessageHeader><FONT face="Times New Roman" size=2>-----Original Message-----<BR><B>From:</B> Kathy Kiessling [mailto:kathy_kiessling@yahoo.com]<BR><B>Sent:</B> Monday, June 16, 2003 9:42 PM<BR></FONT></DIV><FONT face=Arial color=#000000 size=2> <DIV><FONT face=Tahoma color=#0000ff></FONT><FONT face="Times New Roman" color=#0000ff></FONT><BR>Here are two ways to do it.</DIV> <DIV><FONT face=Tahoma color=#0000ff></FONT> </DIV></FONT> <BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px"> <DIV style="FONT: 10pt arial">----- Original Message ----- </DIV> <DIV style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B> <A title=dwilson@pwscom.com href="mailto:dwilson@pwscom.com">David Wilson</A> </DIV> <DIV style="FONT: 10pt arial"><B>Sent:</B> Monday, June 16, 2003 5:32 PM</DIV> <DIV><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><FONT face=Arial color=#000000 size=2></FONT><BR></DIV> <DIV><SPAN class=118332700-05072003>Does anyone know how to get VEE to eliminate unwanted numbers after a decimal point. I am capturing data from an instrument and only want to display the first decimal point. The round(x) function will return only whole numbers which are useless for me. </SPAN></DIV> <DIV><SPAN class=118332700-05072003><FONT face=Arial color=#000000 size=2></FONT></SPAN> </DIV> <DIV><SPAN class=118332700-05072003> For example, I am pinging a current reading from a power supply that reads 6.7863 and only want to display 6.7 on my VEE front panel. </SPAN></DIV> <DIV><FONT face=Arial color=#000000 size=2></FONT> </DIV></BLOCKQUOTE></BLOCKQUOTE> <P></P><FONT size=2></FONT> <HR> </BLOCKQUOTE></BLOCKQUOTE>---<BR>You are currently subscribed to vrf as: rsb@soco.agilent.com<BR>To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".<BR>To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".<BR>To send messages to this mailing list, email "vrf@it.lists.it.agilent.com". <BR>If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".</BODY></HTML>
> pinging the *next* object
Pierre, that is amazing! I'm happily wrong and thanks for the tip.
-SHAWN-
---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list, email "vrf@it.lists.it.agilent.com".
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".
This sounds like a good application of the extremely versatile
To String object with a single transaction set up something like:
write text a real32 fix:1 eol
Stan
---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list, email "vrf@it.lists.it.agilent.com".
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".