AnsweredAssumed Answered

Internal cal accuracy 8753ES

Question asked by b.d.hall on Mar 2, 2010
Latest reply on Mar 3, 2010 by Dr_joel
I have been comparing the 1-port error terms calculated by an 8753ES with an external calculation of the same terms, using the same raw measurement data and (hopefully) the same nominal measurement data based on calkit definitions.

Without going into all the detail (yet, I will if we need more info) I find typical agreement (in the difference between terms) of about 10^-5 (some terms are closer, but that is the ball park). I was expecting better, so I have been looking for mistakes in my calculations, but have not found anything.

Can you tell me whether the internal 8753ES calculations are done in single precision?

It seems that the calkit parameters I downloaded to the 8753 (for the user-defined kit in the calibration) are stored as single-precision floats, because when I read them back from the VNA I see different values in some cases (e.g., a 50 downloaded is not read back from the VNA as exactly 50, which seems odd. It is read back as something like 49.9999xx etc).

So, perhaps I should not be surprised that an external double precision calibration calculation does not match the internal routine better?