outlier rejection scheme and basic improvements for epta
TRANSCRIPT
Overview Outlier Rejection Scheme Summary & Future Work
Outlier Rejection Scheme and basic improvements for EPTAPulsar Timing
—-FAST/Future Pulsar Symposium 8
Jun Wang
Universität Bielefeldhttps://www.AstronDog.com/
June 27, 2019
Jun Wang Xi’an ORSEPT June 27, 2019 1 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Schedule
1 OverviewEuropean Pulsar Timing Array(EPTA) TelescopesBrief Description of the DatasetMotivation & Evaluation scheme
2 Outlier Rejection Scheme1. Median Absolute Deviation(MAD)2. Normalized Residual Gaussian Distribution Fitting
3 Summary & Future Work
Jun Wang Xi’an ORSEPT June 27, 2019 2 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Overview
Jun Wang Xi’an ORSEPT June 27, 2019 3 / 23
Overview Outlier Rejection Scheme Summary & Future Work
European Pulsar Timing Array(EPTA) Telescopes
European Pulsar Timing Array(EPTA) Telescopes
Figure: The telescopes of the EPTA.
Telescope List
• Effelsberg(100m)
• Westerbork(96m)
• Nançay(92m)
• Lovell(76m)
• Sardinia(64m)
Jun Wang Xi’an ORSEPT June 27, 2019 4 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Brief Description of the Dataset
Data-sets and Telescopes Parameters
PulsarsJ0218+4232 J1713+0747J1939+2134 J2145-0750
Telescopes Parameters
Telescope BW(MHz) Freq(MHz) nchan nbin
Effelsberg 200 1347.5 128 256
Westerbork 160 1380 512 256
Nançay 512 1484 128 256
Lovell 400 1532 1600 256
Jun Wang Xi’an ORSEPT June 27, 2019 5 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Brief Description of the Dataset
Epochs of all observations
Jun Wang Xi’an ORSEPT June 27, 2019 6 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Motivation & Evaluation scheme
Outliers influence the Frequency-resolved timing
Figure: A frequency series of PSR J2145-0750, Nancay.
Outliers
1. Incomplete RFI Removal.
2. Low S/N.
3. Bad Channels.
4. Incorrect Calibration.
5. Instrumental failure.
6. ...
Jun Wang Xi’an ORSEPT June 27, 2019 7 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Motivation & Evaluation scheme
How to determine if the Scheme is optimal
Comprehensively analyze the following facors:
1. Minimise RMS.
2. Chisq is slightly larger than 1.
3. Maximize the Total Information Content√
Nσ
.
4. More ToAs are preserved.
Jun Wang Xi’an ORSEPT June 27, 2019 8 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Outlier Rejection Scheme
Jun Wang Xi’an ORSEPT June 27, 2019 9 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Scheme Evolution
Previous thoughts:
1. Goodness-of-fit criterion in the template fit process (0.5-1.5).
2. Median Standard Deviation(MAD).
3. Fitting the residual with Gaussian distribution model and removenon-Gaussian ToAs.
4. S/N criterion(S/N above 8).
Conclusion:
1. Median standard deviation.(k = 3)
2. Fitting the Normalized residual with Gaussian distribution model.(k = 3)
Jun Wang Xi’an ORSEPT June 27, 2019 10 / 23
Overview Outlier Rejection Scheme Summary & Future Work
1. Median Absolute Deviation(MAD)
Subsection 1
1. Median Absolute Deviation(MAD)
Jun Wang Xi’an ORSEPT June 27, 2019 11 / 23
Overview Outlier Rejection Scheme Summary & Future Work
1. Median Absolute Deviation(MAD)
What is MAD
The Commonly used Standard Deviation(SD) is the square root of the variance.For a univariate data set X1, X2, ..., Xn, the MAD is defined as the median of theabsolute deviations from the data’s median:
MAD = median(|Xi −median(Xi )|) (1)
Figure: Weakness of SD in outlier detection.
Jun Wang Xi’an ORSEPT June 27, 2019 12 / 23
Overview Outlier Rejection Scheme Summary & Future Work
1. Median Absolute Deviation(MAD)
Why MAD
0.15 0.20 0.25 0.30 0.35 0.40Points
0
2
4
6
8
10
Coun
ts
Figure: a) Normal distribution, n = 91, mean = 0.27, me-dian = 0.27, standard deviation = 0.06.
0.5 1.0 1.5 2.0 2.5 3.0Points
0
2
4
6
8
10
Coun
ts
Figure: b) Asymmetry due to an outlier, n = 91, mean =0.39, median = 0.27, standard deviation = 0.59.
Advantage of Median Standard Deviation
1. Aimed at symmetric distribution.
2. MAD is more robust than Standard Deviation (SD).
3. MAD is totally immune to the sample size.
Jun Wang Xi’an ORSEPT June 27, 2019 13 / 23
Overview Outlier Rejection Scheme Summary & Future Work
1. Median Absolute Deviation(MAD)
Raw Residuals from J1713+0747, JBO (NTOA = 28376)
Jun Wang Xi’an ORSEPT June 27, 2019 14 / 23
Overview Outlier Rejection Scheme Summary & Future Work
1. Median Absolute Deviation(MAD)
Example on MAD, MAD
Jun Wang Xi’an ORSEPT June 27, 2019 15 / 23
Overview Outlier Rejection Scheme Summary & Future Work
1. Median Absolute Deviation(MAD)
Example on MAD,after MAD(NTOA = 22588)
Jun Wang Xi’an ORSEPT June 27, 2019 16 / 23
Overview Outlier Rejection Scheme Summary & Future Work
2. Normalized Residual Gaussian Distribution Fitting
Subsection 2
2. Normalized Residual Gaussian Distribution Fitting
Jun Wang Xi’an ORSEPT June 27, 2019 17 / 23
Overview Outlier Rejection Scheme Summary & Future Work
2. Normalized Residual Gaussian Distribution Fitting
HOW
After employing the MAD scheme, the distribution of ToAs becomes more Gaussian.We then fit the histogram of Normalized Residual (residual divided by ToA uncertainty)with Gaussian Distribution:
f (x) =1
√2πσ
e−(x−µ)2
2σ2 (2)
and remove residuals which are 3σ away.
Jun Wang Xi’an ORSEPT June 27, 2019 18 / 23
Overview Outlier Rejection Scheme Summary & Future Work
2. Normalized Residual Gaussian Distribution Fitting
Examples on Gaussian Fitting
20 15 10 5 0 5 10 15 20Timing residual / ToA uncertainty
0.0
0.1
0.2
0.3
0.4
0.5
0.6Pe
rcen
tage
Jun Wang Xi’an ORSEPT June 27, 2019 19 / 23
Overview Outlier Rejection Scheme Summary & Future Work
2. Normalized Residual Gaussian Distribution Fitting
After Gaussian Fitting(NTOA = 20978)
Jun Wang Xi’an ORSEPT June 27, 2019 20 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Summary & Future Work
Jun Wang Xi’an ORSEPT June 27, 2019 21 / 23
Overview Outlier Rejection Scheme Summary & Future Work
Summary & In Future
Summary
1. The Wrms is much lower than before, but it’s still not great.
2. The reduced chi-squared is good and very comparable to the best resultspeople typically get.
3. The present optimal outlier rejection scheme is first use MAD, thenNormalised Residuals Fitting.
In Near Future
1. Decide which CCA is best.
2. Decide which template is best.
3. Check optimal ToA bandwidth.
4. Ask for more pulsars to be included.
Jun Wang Xi’an ORSEPT June 27, 2019 22 / 23
Overview Outlier Rejection Scheme Summary & Future Work
The End
Thank you!
Jun Wang Xi’an ORSEPT June 27, 2019 23 / 23