I am using HP8753ES VNA. I got the frequency response from 100k-6GHz and also the time domain response for a coaxial cable. When I used the FFT to transform the time domain signal to frequency domain, why the result is different from what I directly got from the VNA? similarily, the IFFT also didn't work well.

The transformation from frequency to time domain domain isn't as simple as an just taking an IFFT on a VNA. Take a look at this app note:

http://literature.cdn.keysight.com/litweb/pdf/5989-5723EN.pdf

This should give an idea of the complexity involved. We use proprietary algorithms to perform transformations.

Best regards,

David