Hello, This is a interesting question and could use more detailed information to properly give an answer. I first would be curious to know if the calculation would be in the Frequency or Time domain. At first glance it appears to be best suited in the Frequency domain. If this is true then you might re-post the question to the Spectrum Analyzer group instead of the scopes group.
Here is a link to view all the Forums Discussion Forums
Doing a little Googling I found this link and it comes pretty close to giving equations to calculate phase noise to jitter period. Please see if this may help too. http://www.maxim-ic.com/appnotes.cfm/an_pk/3359
Retrieving data ...