I have been comparing the 1-port error terms calculated by an 8753ES with an external calculation of the same terms, using the same raw measurement data and (hopefully) the same nominal measurement data based on calkit definitions.

Without going into all the detail (yet, I will if we need more info) I find typical agreement (in the difference between terms) of about 10^-5 (some terms are closer, but that is the ball park). I was expecting better, so I have been looking for mistakes in my calculations, but have not found anything.

Can you tell me whether the internal 8753ES calculations are done in single precision?

It seems that the calkit parameters I downloaded to the 8753 (for the user-defined kit in the calibration) are stored as single-precision floats, because when I read them back from the VNA I see different values in some cases (e.g., a 50 downloaded is not read back from the VNA as exactly 50, which seems odd. It is read back as something like 49.9999xx etc).

So, perhaps I should not be surprised that an external double precision calibration calculation does not match the internal routine better?

Without going into all the detail (yet, I will if we need more info) I find typical agreement (in the difference between terms) of about 10^-5 (some terms are closer, but that is the ball park). I was expecting better, so I have been looking for mistakes in my calculations, but have not found anything.

Can you tell me whether the internal 8753ES calculations are done in single precision?

It seems that the calkit parameters I downloaded to the 8753 (for the user-defined kit in the calibration) are stored as single-precision floats, because when I read them back from the VNA I see different values in some cases (e.g., a 50 downloaded is not read back from the VNA as exactly 50, which seems odd. It is read back as something like 49.9999xx etc).

So, perhaps I should not be surprised that an external double precision calibration calculation does not match the internal routine better?

Why do this crazy 6 byte representation: well back in the day, (early 8510) when we had to build our own microprosser (it was a bit-slice machine), it would take just too much time for the math libraries to convert to a common mantissa to add and subtract complex number, plus memory was very limited. Fortunately for us, before shipment of the 8510 (which was in development for between 12 and 17 years depending upon who you ask), Motorola came out with the 6400 micro-P, so we didn't have to get into the microprocessor business. I think a similar processor is used in many of the current washing machines and coffee makers.

You will see residuals of this effect as periodic bumps in the time domain response as well, about -90 dB down.