AnsweredAssumed Answered

accuracy of the Yincrement

Question asked by fanfan006 on May 19, 2009
Latest reply on May 20, 2009 by fanfan006

I am using MSO6012A. I know how the Yincrement is calculated in the preamble. With 8-bit values (BYTE formatted data) I have 2^8 = 256 possible voltage values.  This means the A/D measurement will be between 0 and 255.  If I set the scope to 20mV per division, and there are 8 divisions vertically, then I have +-0.08V available on the screen.  The voltage resolution or Yincrement value will then be calculated as 0.16V/256 = 0.000625V. I have no problem with this settings.

When I set the scope as 20mv/division, and I can read from the preamble that Yincrement=0.000625V. It is correct.

However if I am using 10mv/division (which is the limit of my scope), the result from the preamble still says Yincrement=0.000625V. I thought a correct value should be 0.01*8/256=0.0003125. Am I wrong here?