PyRAF/Pipeline Removal of PyRAF/Pipeline Removal of SAA Persistence from SAA Persistence from

NICMOS DataNICMOS Data

PyRAF/Pipeline Removal of PyRAF/Pipeline Removal of SAA Persistence from SAA Persistence from

NICMOS DataNICMOS Data

Elizabeth A. BarkerElizabeth A. Barker

Vicki LaidlerVicki Laidler

Eddie BergeronEddie Bergeron

Anton KoekemoerAnton Koekemoer

Elizabeth A. BarkerElizabeth A. Barker

Vicki LaidlerVicki Laidler

Eddie BergeronEddie Bergeron

Anton KoekemoerAnton Koekemoer

TIPS Meeting18 October 2007

South Atlantic AnomalySouth Atlantic AnomalySouth Atlantic AnomalySouth Atlantic Anomaly

Part of inner Van Allen radiation Part of inner Van Allen radiation beltbelt

Higher density of charged particlesHigher density of charged particles 8-9 times per day8-9 times per day About half of all orbitsAbout half of all orbits

Part of inner Van Allen radiation Part of inner Van Allen radiation beltbelt

Higher density of charged particlesHigher density of charged particles 8-9 times per day8-9 times per day About half of all orbitsAbout half of all orbits

Cosmic Ray PersistenceCosmic Ray PersistenceCosmic Ray PersistenceCosmic Ray Persistence

Electrons trapped during SAA passageElectrons trapped during SAA passage Exponential decay for release of Exponential decay for release of

electronselectrons Lower SNR in Lower SNR in shortshort exposures means exposures means

persistence has persistence has smallersmaller contribution to contribution to total noisetotal noise

Persistence becomes a Persistence becomes a moremore significant significant noise source in noise source in longlong exposures exposures

Fixed noise pattern per SAA passage Fixed noise pattern per SAA passage (different from pure random noise)(different from pure random noise)

Electrons trapped during SAA passageElectrons trapped during SAA passage Exponential decay for release of Exponential decay for release of

electronselectrons Lower SNR in Lower SNR in shortshort exposures means exposures means

persistence has persistence has smallersmaller contribution to contribution to total noisetotal noise

Persistence becomes a Persistence becomes a moremore significant significant noise source in noise source in longlong exposures exposures

Fixed noise pattern per SAA passage Fixed noise pattern per SAA passage (different from pure random noise)(different from pure random noise)

Persistence RemovalPersistence RemovalPersistence RemovalPersistence Removal

““Post-SAA Darks” [Bergeron & Najita, Post-SAA Darks” [Bergeron & Najita, 1998]1998]

Different from standard calibration dark Different from standard calibration dark imagesimages

2 images taken back-to-back2 images taken back-to-back First exposure starts 174 sec after SAA First exposure starts 174 sec after SAA

exitexit Each 256 sec exposureEach 256 sec exposure BLANK filterBLANK filter Implemented as automatically Implemented as automatically

scheduled exposures since Cycle 11scheduled exposures since Cycle 11

““Post-SAA Darks” [Bergeron & Najita, Post-SAA Darks” [Bergeron & Najita, 1998]1998]

Different from standard calibration dark Different from standard calibration dark imagesimages

2 images taken back-to-back2 images taken back-to-back First exposure starts 174 sec after SAA First exposure starts 174 sec after SAA

exitexit Each 256 sec exposureEach 256 sec exposure BLANK filterBLANK filter Implemented as automatically Implemented as automatically

scheduled exposures since Cycle 11scheduled exposures since Cycle 11

Post-SAA DarksPost-SAA DarksPost-SAA DarksPost-SAA Darks

Persistence ModelPost-SAA Dark

SAA-impacted ImagesSAA-impacted ImagesSAA-impacted ImagesSAA-impacted Images

Calibrated Image SAAcleaned Image

NIC1, F090M, 448 sec, 918 sec since SAA exitNIC1, F090M, 448 sec, 918 sec since SAA exit

SAA-impacted ImagesSAA-impacted ImagesSAA-impacted ImagesSAA-impacted Images

Calibrated, Pedsub Image SAAcleaned, Pedsub Image

NIC2, F215N, 1400 sec, 917 sec since SAA exitNIC2, F215N, 1400 sec, 917 sec since SAA exit

Persistence Removal Persistence Removal AlgorithmAlgorithm

Persistence Removal Persistence Removal AlgorithmAlgorithm

Recommend pedestal correcting calibrated imageRecommend pedestal correcting calibrated image Use previously determined scale factor of decay Use previously determined scale factor of decay

between post-SAA darks (0.54-0.56)between post-SAA darks (0.54-0.56) [Bergeron & Dickinson, NICMOS ISR 2003-010]

Create Image of PersistenceCreate Image of Persistence Weighted combination of post-SAA darks CR rejection Remove pedestal

Iteratively scale and subtract persistence image from Iteratively scale and subtract persistence image from science imagescience image Fitting to the minimum total noise in image

Repeat pedestal correction on final output imageRepeat pedestal correction on final output image

Recommend pedestal correcting calibrated imageRecommend pedestal correcting calibrated image Use previously determined scale factor of decay Use previously determined scale factor of decay

between post-SAA darks (0.54-0.56)between post-SAA darks (0.54-0.56) [Bergeron & Dickinson, NICMOS ISR 2003-010]

Create Image of PersistenceCreate Image of Persistence Weighted combination of post-SAA darks CR rejection Remove pedestal

Iteratively scale and subtract persistence image from Iteratively scale and subtract persistence image from science imagescience image Fitting to the minimum total noise in image

Repeat pedestal correction on final output imageRepeat pedestal correction on final output image

High and Low Signal High and Low Signal PopulationsPopulations

High and Low Signal High and Low Signal PopulationsPopulations

Long SAA passage Long SAA passage durationduration

Widely varying CR Widely varying CR energiesenergies

Earliest CR hits Earliest CR hits have decayed have decayed mostmost

Latest CR hits Latest CR hits have higher have higher persistence signalpersistence signal

Long SAA passage Long SAA passage durationduration

Widely varying CR Widely varying CR energiesenergies

Earliest CR hits Earliest CR hits have decayed have decayed mostmost

Latest CR hits Latest CR hits have higher have higher persistence signalpersistence signal

Iteratively Fitting Persistence Iteratively Fitting Persistence ModelModel

[Bergeron & Dickenson ISR 2003-010][Bergeron & Dickenson ISR 2003-010]

Iteratively Fitting Persistence Iteratively Fitting Persistence ModelModel

[Bergeron & Dickenson ISR 2003-010][Bergeron & Dickenson ISR 2003-010]

Persistence level depends on:Persistence level depends on: Length/depth of SAA passage Time since SAA exit Science exposure time

Iteratively subtract scaled persistence modelIteratively subtract scaled persistence model Step through different multiplicative factors Measure RMS width of pixel histogram for each

factor Minimum RMS width indicates best factor Subtract persistence model from science exposure,

after scaling by best factor

Persistence level depends on:Persistence level depends on: Length/depth of SAA passage Time since SAA exit Science exposure time

Iteratively subtract scaled persistence modelIteratively subtract scaled persistence model Step through different multiplicative factors Measure RMS width of pixel histogram for each

factor Minimum RMS width indicates best factor Subtract persistence model from science exposure,

after scaling by best factor

Best Persistence FitBest Persistence FitBest Persistence FitBest Persistence Fit

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Figure 12: Bergeron & Dickinson ISR 2003-010

~ (129-46)/129

Testing PyRAF SAAcleanTesting PyRAF SAAcleanTesting PyRAF SAAcleanTesting PyRAF SAAclean

Conversion from IDL Conversion from IDL [Bergeron][Bergeron] to to Python by Vicki LaidlerPython by Vicki Laidler

Data Included:Data Included: All cameras SAA-impacted Flat-fielded Pedestal-corrected (pedsub)

Data Excluded:Data Excluded: Grism data (not flat-fielded) Polarizers

Conversion from IDL Conversion from IDL [Bergeron][Bergeron] to to Python by Vicki LaidlerPython by Vicki Laidler

Data Included:Data Included: All cameras SAA-impacted Flat-fielded Pedestal-corrected (pedsub)

Data Excluded:Data Excluded: Grism data (not flat-fielded) Polarizers

Testing SAAclean - ResultsTesting SAAclean - ResultsTesting SAAclean - ResultsTesting SAAclean - Results

MaximumMaximum

Error CorrectionError CorrectionMedianMedian

Error CorrectionError Correction

CamerCameraa

# of # of ImageImage

ss

High High Signal Signal

DomainDomain

Low Low Signal Signal

DomainDomain

High High Signal Signal

DomainDomain

Low Low Signal Signal

DomainDomain

11 415415 49.08%49.08% 29.43%29.43% 1.818%1.818% 0.132%0.132%

22 15951595 59.93%59.93% 53.94%53.94% 1.319%1.319% 0.122%0.122%

33 21332133 31.63%31.63% 27.41%27.41% 3.171%3.171% 0.174%0.174%

PyRAF SAAclean - InputPyRAF SAAclean - InputPyRAF SAAclean - InputPyRAF SAAclean - Inputcalcimage = calcimage = Input (usually ped) file to calculate correctionInput (usually ped) file to calculate correctiontargimage = targimage = Input (ped or cal) file to correctInput (ped or cal) file to correctoutput = output = Output cleaned datafile (targimage with correction applied)Output cleaned datafile (targimage with correction applied)clobber = no clobber = no Overwrite output files if they already exist?Overwrite output files if they already exist?(readsaaper = no) (readsaaper = no) Read SAA persistence image from file (If no, construct it.)Read SAA persistence image from file (If no, construct it.)saaperfile = saaper.fits saaperfile = saaper.fits Filename for SAA persistence imageFilename for SAA persistence image(writesaaper = yes) (writesaaper = yes) Write SAA persistence image?Write SAA persistence image?(flatsaaper = yes) (flatsaaper = yes) Flat-field SAA persistence image before analysisFlat-field SAA persistence image before analysis(darkpath = saaref$) (darkpath = saaref$) Path to dark reference filesPath to dark reference files

(scale = 0.54) (scale = 0.54) Scale factor for constructing persistence imageScale factor for constructing persistence image(wf1 = 0.7) (wf1 = 0.7) Weight for first SAA exposureWeight for first SAA exposure(wf2 = 0.3) (wf2 = 0.3) Weight for second SAA exposureWeight for second SAA exposure(crthresh = 0.3) (crthresh = 0.3) Threshold for CR rejectionThreshold for CR rejection

(noisethresh = 1.0) (noisethresh = 1.0) Noise reduction threshold (percent)Noise reduction threshold (percent)(binsigfrac = 0.3) (binsigfrac = 0.3) Stddev fraction for excluding narrow binsStddev fraction for excluding narrow bins(stepsize = 0.008) (stepsize = 0.008) Increment multiplier for SAA scale factor fittingIncrement multiplier for SAA scale factor fitting

(fitthresh = yes) (fitthresh = yes) Solve for threshold value? (If no, uses value of thresh)Solve for threshold value? (If no, uses value of thresh)(thresh = ) (thresh = ) Threshold to separate high/low signal domain in SAA Threshold to separate high/low signal domain in SAA

persistence imagepersistence image(histbinwidth = 0.001)(histbinwidth = 0.001) Bin width for histogram in threshold fittingBin width for histogram in threshold fitting(nclip = 3) (nclip = 3) Number of clipping iterations for threshold fittingNumber of clipping iterations for threshold fitting(hirange = 0.4) (hirange = 0.4) Maximum multiplier for high signal domainMaximum multiplier for high signal domain(lorange = 0.25) (lorange = 0.25) Maximum multiplier for low signal domainMaximum multiplier for low signal domain(fitmult = yes) (fitmult = yes) Fit to determine multiplier for minimum noise?Fit to determine multiplier for minimum noise?(applied = ) (applied = ) Cleaning applied to which domains?Cleaning applied to which domains?(hi_nr = ) (hi_nr = ) Noise reduction in high signal domainNoise reduction in high signal domain(lo_nr = ) (lo_nr = ) Noise reduction in low signal domainNoise reduction in low signal domainalldiags = no alldiags = no Write out all possible diagnostic files?Write out all possible diagnostic files?(diagroot = diag) (diagroot = diag) Root filename for diagnostic filesRoot filename for diagnostic files

PyRAF SAAclean - OutputPyRAF SAAclean - OutputPyRAF SAAclean - OutputPyRAF SAAclean - Outputsaaclean version 0.99devInput files: n8xw12n8q_subisr.fits n8xw12n8q_subisr.fitssci image : using DQ extension for badpixpostsaa dark #1 : using DQ extension for badpixfailing over to /data/cdbs5/nref//m9c1047pn_msk.fitspostsaa dark #2 : using DQ extension for badpixfailing over to /data/cdbs5/nref//m9c1047pn_msk.fitsUsing scale factor of 0.54 to construct persistence imageflatfile : using DQ extension for badpixmedian used in flatfielding: 0.0379231178101Coefficients for gauss-poly fit to persistence model histogram:Gaussian (low signal component) terms:Amplitude, Mean, Sigma: 1026.073392 17.503406 9.216971Polynomial terms:Constant, Linear, Quadratic:47.687849 0.422538 -0.003921Threshold for hi/lo: 0.0995250821636Npixels hi/lo: 9596 55940Results summary for high domain:chi2 for parabola fit = 0.000420518773132min-noise (best) scale factor is: 0.311645043762effective noise at this factor (electrons at gain 5.400000): 49.903919noise reduction (percent) : 38.5484443442Results summary for low domain:chi2 for parabola fit = 4.05079716407e-05min-noise (best) scale factor is: 0.189995821621effective noise at this factor (electrons at gain 5.400000): 36.192606noise reduction (percent) : 7.21719319318Applying noise reduction in both domains

PyRAF SAAclean - ProductsPyRAF SAAclean - ProductsPyRAF SAAclean - ProductsPyRAF SAAclean - Products

Files Output:Files Output:

•SAA cleaned SAA cleaned imageimage

•Persistence Persistence image (if image (if requested)requested)

Keyword DescriptionSAAPERS SAA persistence imageSCNPSCL Scale factor used to construct persistence imageSCNPMDN Median used in flatfielding persistence imageSCNTHRSH Threshold dividing high and low signal domainsSCNHNPIX Number of pixels in high signal domain (HSD)SCNLNPIX Number of pixels in low signal domain (LSD)SCNHCHI2 HSD chi squared for parabola fitSCNHSCL HSD scale factor for min noiseSCNHEFFN HSD effective noise at SCNGAINSCNHNRED HSD noise reduction (percent)SCNLCHI2 LSD chi squared for parabola fitSCNLSCL LSD scale factor for min noiseSCNLEFFN LSD effective noise at SCNGAINSCNLNRED LSD noise reduction (percent)SCNAPPLD To which domain(s) was SAA cleaning appliedSAACNTAB Reference table (with task parameters)SAACPDGR Pedigree of reference tableSAADFILE SAA dark reference image fileSAADPDGR Pedigree of reference imageSAACORR Correct for SAA signatureSAADONE Status of SAA signature correction

PyRAF SAAclean in PipelinePyRAF SAAclean in PipelinePyRAF SAAclean in PipelinePyRAF SAAclean in Pipeline

To be implemented in OPUS 2008.1To be implemented in OPUS 2008.1 After After calnica calnica *_cal.fits *_cal.fits PedsubPedsub *cal.fits*cal.fits file file SAAcleanSAAclean SAAclean correction applied to SAAclean correction applied to *_cal.fits*_cal.fits file (without pedestal file (without pedestal correction) correction)

Outputs modified Outputs modified *_cal.fits*_cal.fits file file

To be implemented in OPUS 2008.1To be implemented in OPUS 2008.1 After After calnica calnica *_cal.fits *_cal.fits PedsubPedsub *cal.fits*cal.fits file file SAAcleanSAAclean SAAclean correction applied to SAAclean correction applied to *_cal.fits*_cal.fits file (without pedestal file (without pedestal correction) correction)

Outputs modified Outputs modified *_cal.fits*_cal.fits file file