ORGANISATION OF THE MSDP DATA PROCESSING

Thierry ROUDIERNadège MEUNIERPierre MEIN

MSDP Workshop, Tarbes, 18-20th January 2006

PLAN

Codes : choice and availability Organisation of the directories, files : input The parameter files (short) The different steps through the processing (short) The output files : interpretation More details about the parameter files associated to each step

Code: choice and availibility

Only existing code : developped by Pierre Mein

Public : available on our web site http://bass2000.bagn.obs-mip.fr/

Acknowledgements in publications

SOFTWARE AND DOCUMENTATION

SOFTWARE: http://bass2000.bagn.obs-mip.fr/New2003/Pages/DPSM/dpsm_acceuil.html

DOCUMENTATION:

GENERAL: - readme.txt general user guide - auto.txt user guide for msdpauto - sequence.txt example of data for msdpauto - param.txt parameter list of ms.par

OTHERS : - correction.txt parameter list to modify in various cases - captions.txt plot control -filenames.txt filename description at the different processing steps - remarks.txt a few examples and difficulties - journal.txt list of successive improvements of the code - signs.txt give the sign of the result - widget.txt widgets information (not updated) - vtt.txt information for the VTT observations

Organisation of the files and directories

Parameter files Data files (scan, flat-fields, dark

current, field-stop) IDL routine msdpauto

create the directory for output files cut raw 3D files into 2D files (1 im / file) create the parameter file ms.par start the fortran code ms1

Fortran code ms1 process the data

Individual 2D files 1 b3 file (scan), with n images : 1 starting

time Creation of n 2D image file : filename

including an artificial time (increment of 1) example :

Usefull to limit the number of files to process after this step (tob1, tob2)

Actual time of each image : .log file obtained at THEMIS

c031031_13182784_00111c031031_13182785_00111c031031_13182786_00111c031031_13182787_00111c031031_13182788_00111

t*fts sequence.par

/data/ /data2/auto/

msdpauto

tyyyy.par

N, L, S

/data2/auto/dir_date_Nseq_L/key1

key.par

b*.fts

ms.par

Parameters

Conversion

ms1Computation

Option /no_fort

MSDP DATA PROCESSING

N=sb=seq.

L=cm=line

S=qv=Stokes

ms1

x*L*z*L*

y*L*Sc*L*Sd*L*Sq*L*Sr*L*Sp*L*S

geo.ps g*Lflat.ps f*L*Sgrid.pscmd*.ps quick.ps j*L*Scmr*.psprof.pssq*L*S.pssp*L*S.ps

ms.lisscan.lis

readmsdp

Quick-look

Spectroheliog

Averages

Channels

Bisect.

Profiles

Calib.

steps results .ps files Ascii files

The parameter files Telescope related file : tyyyy.par

include instrumentalk set-up informations

can change over the years Sequence related file : sequence.par

1 line per sequence liste of files to process association obs / calib info steps, polarimetry … see header keywords

Processing info : ms.par (BIG FILE !) characterizes sequence + line

Steps Corrections Files Output results

geom calib

Power fctsScattered lightNormalizationSmoothing Profile curvatureFourier filteringCospatiality

2D - correlAverage departures

geoflatbmc

cmd

b

c

d

q

r

p

Aligned and calibrated channelsPossible direct inversionavoiding interpolation corrections

Individual maps I, v, B//Possible destretching

Large maps I, v, B//

Individual maps Profiles I, Q, U, V with calibrated central wavelength

Large spectrohéliog. I,Q,U,V Inversions with constant

quick

cmr

prof

Like cmd except cospatiality

Like quick except 2D - correl

The different steps through the processing

The ouput files: interpretations

One postscript file per step Binary files with results Ascii files with messages

GEO.PS

Intensity gradients

A D

BE

The extrema define the channel edges in i

The program computes the regression line for projected vectors (AD,BE,…) on i and j

Channels location

Intensities

Flat fields

Line curvature correction Mean profile determination Elimination of the mean profile Check that the result is « flat » :

flat.ps

FLAT.PSMinimum signal (line core ) + parabolic adjustement

Shift at same between 2 successive channels (ltrj) Mean profile of successive channels

Mean profile after transmission correction for the 1st window

Control of the even and odd interlacing channels (box 16 channels)

Mean profile kept

Start of 1st channel

Idem 2nd window

Channel cut along i

Start of the last channel

Channel cut along j

Cuts mean along j for all the channels

Idem 2nd box

Results

For each step, one file containing everything (I, B// … )

Order and number of the images in the file depends on : observing condition (polarization or not) number of steps chosen for the output Stokes

profiles As many q* or p* files as the number of

Stokes parameters A file per scan To read the files : IDL routine readmsdp

Standard quicklook output filewith no polarization

images #1, 2, 3, 4 : intensities (close to line center, aver. I at ±6Δλ , diff. between I at ±6Δλ ~ V// and aver. I at ±6Δλ bissector)

image #5 : V// at ±6Δλ (bissector)

Δλ=dlambda/2 if 9 channels, dlambda/4 if 16 channels

dlambda = distance between channels

images #1, 3, 5, 7 : intensities (close to line center, aver I at ±6Δλ , diff. between I at ±6Δλ ~V// and aver. I at ±6Δλ bissector) ; repeated n times (n=number of Stokes meas.)

image #8 : co-spatiality map : diff. between I at ±6Δλ

images #2, 4, 6 : Stokes Q (or U, V) close to line center and at ±6Δλ, + difference when Stokes V (~ B//)

image #9 : V// at ±6Δλ (bissector) image #10 : B// at ±6Δλ (bissector)

Standard quicklook output filewith polarization

Final output p* filewith no polarization

images #1 to17 : Stokes I profile around line center, ±nΔλ, and n from –8 to +8

images #19, 21 : V// at ±4Δλ and ±8Δλ (bissector)

images #18, 20 : aver. I at ±4Δλ and ±8Δλ (bissector)

images #1 to 17 : Stokes I profile around line center, ±nΔλ, and n from –8 to +8

images #18 to 34 : Stokes profile around line center, ±nΔλ, and n from –8 to +8

images #37, 41 : V// at ±4Δλ and ±8Δλ (bissector)

images #38, 42 : B// at ±4Δλ and ±8Δλ (bissector) if Stokes V

images #35, 39 : aver. I at ±4Δλ and ±8Δλ (bissector)

images #36, 40 : diff. between I at ±4Δλ and ±8Δλ for cospatiality tests (bissector)

Final output p* filewith polarization

ASCII files

scan.lis : small text file ms.lis : very long file, prints and

warning for all steps of the computation

Back to the parameter files

tyyyy.par sequence.par ms.par

tyyyy.PAR

tyyyy.par (THEMIS), pyyyy.par (Pic du Midi), vyyyy.par (VTT), myyyy.par (Meudon)

yyyy : year (may be constant or change)

Contents instrumental configuration processing and output options :

WARNING ; example number of points in the

profiles lmpr1*2+1 ; Δλ = lbd1r1

(nl) lbd ncha grorder nbox jt1000 ja1000 jb1000 1 4861 9 47 1 2 4861 16 46 2 3 5173 16 44 2 4 5876 16 38 2 2903 83 5 5890 16 38 2 6 5896 16 38 2 7 6103 16 37 3 8 6563 9 34 1 9 6563 16 34 2 10 8542 16 26 2

(nbox) inveri inverj invi invj invern inverl invers nlisd nlisr 1 1 1 1 0 0 1 0 0 0 2 0 0 1 0 1 0 1 2 2 3 0 0 1 0 1 0 1 2 2 4 1 1 1 0 0 1 0 0 0

tl sb sx sy sz cm bs yy mm dd lbd go stx dt sty ny ng nq qv nb bt qp sd

1 3 3 3 3 2 16 03 10 17 0 0 0 60 0 0 4 3 0 1 0 0 2

1 5 5 5 5 0 16 00 08 24 8542 0 5000 60 8500 4 3 1 1 1 0 0 1

1 6 6 6 6 0 16 00 08 24 5890 0 5000 60 11000 3 3 3 3 1 0 0 1

1 8 8 8 8 0 16 00 08 24 4861 0 5000 60 11000 3 3 1 1 1 0 0 1

1 9 9 9 9 0 16 00 08 24 4861 0 5000 60 11000 3 3 1 1 1 0 0 1

end

Télescop

séquence number

obs.

d.c.

f.f.

f.s.

caméra

channelnumber

date

grating order

X step

t between scans in 1/10 de s.

polarisation

burst

up to the stage « q » ou « p »

0 = sun1 = dec2=linux

SEQUENCE.PAR

Manual or =0 for file header

MS.PAR

Parameters : fixed (derived from tyyyy.par,

sequence.par, headers, …) variables depending on the

options, problems

Choose the data level : ixy, igeo, iflat, ibmc, icmd, iquick, icmr, iprof, igrayq, igrayp

Modify the thresholds (geometry determination, rejection, …) : milgeo, si, sj, sgi, sgj, etc.

Remove pieces of images (borders) : nob, nob2, ix1, ix2, etc…

Choose the output spatial step : milsec Normalize intensities (in case of clouds) : norma Symetrize the image (scanning, Stokes sign,

direction) : inveri, inverj, invi, invj, invers, etc … Filter and smooth : crecd, w1d, w2d, w3d, lcrecq,

etc. Choose the chords : lmpd, lbd1d, lbpasd Choose to print the results

Main options

ANNEXE

tel dob nseq nline ncam1 ncam2 1 20031017 3 2

MSDPBMS WAVELNTH GRORDER FSLTH FSWTH STEP_X NBSTEP_X 16 5896 0 60000 300000 5000 20 STEPGRID NBSTGRID GRID_MAX GRID_PER GRID_WID SEQ_STOK BURST 8500 4 0 0 0 3 0

Télescop Date obsSequence number

Camera number

Parameters non used in ms.par

MS.PAR

FILE obs.par

nm lbda dlbd mupris mustep minpro xfirst

8 5896 80 3300 800 500

Number of channel c/ (window)

Distance between 2 channels (mAngs.).

Lambda (Angs.)

multi-slit stepbox (micron)

Translation between channels(prisms box) (micron)

Normalisation of the profile, value ajusted at the line center

nwinp mgrim nquv ipos burst select polord

2 4 3 4 1

ntmax priscan jypas interc uint

0 0 5000 15

Number of (window) / image

Nombre d’état de polarisation

Number of positions Y-scan (in polarisation)

Number of imagesby burst

Maximun number , step of the grid (in polarisation)

Number of’imagesby scan

Prisms order For the field

Step in X of the sweep (here 5’’.0) (arcsec/1000) Approximative distance

Between the end and the beginning of the channels fUnity=pixel CCD

win kdecal

2 0

1 50

nbcln nblgn li lj invern

1035 921 133000 9000 1

1035 921 133000 9000 1

Number of the window

Channels interlacing

Number of pixels in the window in i

Nombre de pixelsDe la fenêtre en j

Field size arcsec in i (*1000)

Field size in arcsec en j (*1000)

To modify the channel order

cqp

inveri inverj inverl norma scatter etal

1 0 0 0 0 0

ix1 ix2 jy1 jy2 jyq1 jyq2

0 133000 500 8500 500 8500

0 133000 500 8500 500 8500

Symmetrize the maps / i

Symmetrize maps / j

Reverse the orientation (lambda)

Normalize intensity(example: clouds)

Diffusion rate(scatter/1000)

not used

Take off the edgein x ,in arcsec

Take off the edgein y , in arcsec

Same for the out files« p » et « q » 

invi invj istep invers (istep et invers echange)

1 0 8500 0

Reverse out mapsReverse the signs of Stokes parame ters

Step in Y (STEP_Y)(arcsec/1000) en polarisation

FILE exe.par

dir

/home/lafon/dpsm/data/dir3_2/

filter b000000_000_000_000000_m0000_00000000.fts

ixy igeo iflat ibmc

1 1 1 1

icmd iquick icmr iprof igrayq igrayp

1 1 0 0 1 0

Directory of files b

Filter of files b

Différentes step : 1 for use 0 else

tob1 tob2

0000000024000000

tdc1 tdc2

0000000024000000

tfs1 tfs2

0000000024000000

tff1 tff2 nff

0000000024000000 1

24000000

24000000

Start and end of the observation to be traited

Hours min et max of dark current

Hours min et max of field stop

Hours min et max of flat field Numbre offlat fields used divided by nqff

tcl1 tcl2

0000000024000000

sundec iswap intert ipermu nqseul milsec

0 1 600 1 0 250

bmg

si sj sgi sgj milang milgeo nleft nright

0 15 15 15 0 3500 0 0

0 15 15 15 0 3500 0 0

Hours for geometric calibrations

Ordinateur type

No used

Swap or non

Minimun time-stepBetween 2 scans (1/100) seconde

Echange X et Y

Number of couples (if polarisation)

out put pixel size, here 0.25 arcsec

IntensitY threshold in i et j to detect the channels

Geometry thresholdRegression differencein 1/1000 de pixel

Channels anglegradients intensity threshold inn i et j to detect the channels

To determine the channel left edge du (right) from neighbourg channel.

cmf

inclin milrec calfs caldeb

1 500 0 1

cqp

ideb igri itgri itana jtana calana milalp milzero ijlis

0 12000 33500 16298 0 0 0 0 0

Intery

ilisdr jlisdr mincmd maxcmd ilisqp jlisqp

Type of computation for the relative Channels transmission

Threshold for alignementbetween FF and FS

Beam translations of the separator for polarisation in i and j.

Computation by the program of grid position (polarisation)

Util size in arcsec/1000 of grid plages (polarisation)

Grid period arcsec/1000

1st point of the first util plage of the grid arcsec/1000 (position)

Shift adjustement xy of of the analysor (polar. circ.)

Intensity change for the signalbefore interpol. (I **a )

Spatial smoothing ,noise

Type of the detection of the line shape

cmd

cented sumd nlisd curvd crecd w1d w2d w3d ratiod

1 0 2 0 2000 0 1 0

Intensity line core computation

Direct output from channels

Profil smoothing Curvature correction by using neighbourg points .

Fourier Filtering to correct « cannelures « 

lmpd lbd1d lbpasd

0 0 0

0 0 0

2 1500 1500

quick

crecq milsigq lcrecq

0 2000 0

cmr

center sumr nlisr curvr crecr w1r w2r w3r ratior

1 0 2 0 2000 0 1 0

lmpr lbd1r lbpasr

7 500 500

0 0 0

0 1500 1500

Réjection by computing the mean of values with gap graeter than sigma *milsigq/1000.

Mean gap correction

Smooth in y

Parameter définitions identical to those of « cmd »

Spectrohéliogrammes (no used at cmd step because car non calibratec).

bissectors

Sum and différence (blue and red wings)

1s rope : 1.5 * dlbd=1.5 * 80= 120 mA2 2nd rope : 3.0 * dlbd=3.0 * 80= 240 mA

prof

crecp milsigp lcrecp

0 2000 0

FILE fix.par

reg lin linref iplotg iplotf nqff

0 0 0 2 4 3

npol

1

bmg

(win) i1 i2m j1 j2m lip jeps intvi intvj

1 1 0 1 0 40 20 30 20

2 1 0 1 0 40 20 30 20

Plot géo.ps

Plot of flat.ps

FIX PARAMETERS

Define the Stokes parameters succession for flat field

No used

No used

Window number

1st pixel used and gap to the last pixel in i et j

Interval between in i used to measure the channels curvature, here 40%

Interval to search in j the edges i n i (gretaer length ) at + or - jeps pixels

Intervals in i et j to compute the means to detect the edges in j and i

(win) leps n1 distor normsq dlxy

1 40 1 1

2 40 1 1

bmc

idc dxr100 dyr100 dxrmms dyrmms

1 0 0

cmf

smoothi smoothj il1p il2p isym iextra iff

0 0 10 90 0

l

1st util channel Take into account the channel curvature

For the dark current

Window number

The corrections in each Falt field channel are replaced or not by means

Restric the mean profile computations of the spectral line

Symmetrized profile

Search interval of points with gradients maximun to +/- leps

Small shift between flat field and scan images

dxdust dydust x1dust x2dust y1dust y2dustWcs ncs acs1 zcs1 bcs1 acs2 zcs2 bcs2

(win) curv iliss jparli lispro deconv

1 1 10 5 10

2 1 10 5 10

(win) jt100 ja100 jb100 jz100 jtcor

1 0 0 0 0

2 0 0 0 0

cmd/cmr

longw lat absord absorr mps cstok

0 0 1 1 1

Smoothing in i before the detection of the line core

Parabolic smoothing in j before the detection of the line core

Mean profil smoothing used to compute the corrections

Window number

No used Profil in absorption or emission for files d

To take care of the curvature of the line

Window number

Translation in j, in pixel/100, corresponding to the difference in between 2 channels

Define the tilt and curvature of the line in each channel

Same for files r

Specify the velocity unit in m/s

If 0 parameters are computed by the program

quick

lcorq jlap2q icormq copasq milcoq decmq

0 0 0 0 0 0

prof

lcorp jlap2p icormp copasp milcop decmp

0 0 0 0 0 0

gray

igrq jgrq igrp jgrp imax

3 2 4 2 0

Indice of the used board for the 2D spatial correlation

½ interval of overposition between 2 frames of the scan

size for the correlation computation Step for the computation of the first derivative over x

The result is not taking into accountif the maximun pf the 2D correlation is less than milcoq /1000

No usedParameters identical to quick

Number of plots in horizontal et vertical files q

Idem files p Maximum number of pixels in y direction y for all the sweep. Allows to adjust the graphic scale p and q

graphicsd control

blackq whiteq blackp whitep dreject rreject reject 0 1 0 1 1 0 0 0 1 2 0 0 0 1 3 0 0 0 1 4 ---------------------------------------- 0 1 0 1 30

end

0 et 1 to visualisaze ( same as TVSCL d’IDL)

0 and 0 no view