hl_reduce (Hoher List reduction package for IRAF)


Version 0.9 (29/10/01), Oliver Cordes

Sternwarte der Universität Bonn
Table of contents:
  1. Data aquisition
  2. File selection
  3. Bias combination/correction
  4. Flatfield combination/correction
  5. Bad pixel correction
  6. Science frame reduction
  7. Nomenclature of files
  8. Description of all functions
  9. Download and install
Data aquisition

For a good data reduction we recommend to make as much bias and flat fields as possible. Usually the best way is to make 5 bias frames in each readout mode (full_p4s,c1408_p4s,...) in the evening before and in the morning after the observing. NOTE:Check the temperature of the HOLICAM camera in the dome., it should always be -100C. Different temperatures lead to unwished effects. :-)
File selection

The subpoint may be confusing, but it is absolutly necessary. Most of the reduction functions are working with file lists. Before you make any reduction step, sort your data. We recommend a separate directory for bias, flat fields and science frames (e.g. one directory for each object!). In all directory a so called "input list" has to be created. It conta i ns the original FITS filename in the first column and an iraf filename in the second column. The easiest way to create such an input list is the use of the unix command ls: just type ls *.fits > inlist . If you e.g. in the flatfield directory files from more than one filter you can use the task hl_selectfiles for creating the input list. Use *.fits as filemask,   inlist as database file, FILTERS as keyword and e.g. V as exact value to select all V flat fields.  Now you have to put in the iraf filenames in the input list. Ususally the best way is to use the  number of the file as the iraf name. For this work we use the emacs editor. Load the input list into the buffer and mark the first number in the first line (put the cursor befor the first number and press Ctrl-space ). Then move the cursor to the end of the file to the last number in the last line. Press Ctrl-x r g and then a and the region with all the numbers is copied into a temporary buffer. Move now the cursor back at the end of the first line, insert a space and inster the previous copied buffer  (press Ctrl-x r r and then a ).  Now you have a valid input list for the reduction functions. NOTE: The functions don't check, if the input list is valid and produce weird files!
Bias combination/correction

The bias correction is the most complex and important step in the data reduction. If you look at single science and bias frames you'll see a border in the middle of the frame which seperates the upper and lower part of the images and different levels in each part. This effect comes from the dual read out technique used with HOLICAM.  First we have to convert all fits files into the IRAF format (NOTE: set imtype to hhh with set imtype=hhh in yopur login.cl!) with hl_rfits inlist worklist ( inlist : see File selection ). worklist contains the  names of the processed files. Now we use the command hl_biascor worklist biaslist to use the overscan region for a simple bias correction. The resulting images, biaslist holds the filenames, must be combined using the IRAF command imcombine . Use the parameter @biaslist as input value, average as combine value and pclip as reject value. If you look at this combined image you'll see all the electronics effects which can't be corrected with the overscan region. For the bias correction you have to substract this image from each science frame which was processed with hl_biascor before. The problem with this method is noise which comes from the electronics. Each time you use this image (bias/flatfield correction) you add noise to the science frame. To get rid of this effect we developed a procedure to create a synthetic bias frame which contains only the electronic structures and no noise. For this procedure you'll have to use the command hl_synthbias . This command tries to fit the lower and upper part of the inout bias frame seperately with a legendre, chebyshev, spline1 or spline3 funtion with a given order. The fitting procedure is divided into a column fitting and a line fitting procedure. From experience we found that fitting of both components leeds to an acceptable result, but you can choose which fit component you want. First edit the parameter list of this command with epa hl_synthbias .  Set  imname with the input bias you want to synthesize and oname with the resulting image, e.g. if bias_256 is the input name use synth_bias_256 for the resulting name. We got the best fit result with the chebyshev funtion of the order of 21 . Then type  hl_synthbias split to split the bias frame into a lower and upper part.  Now you can fit each part with hl_synthbias synthlower and hl_synthbias synthupper . During the fit process you see in a xgterm window the original noisy funtion and the fit with a dashed line.  On top of the window also the rms of the fit is printed. To proceed with the process press q . If you are not satisfied with the fit then you can change the parameters with epa hl_synthbias and repeat the fitting commands. After the fitting you have to reconstruct the synthetic bias image with hl_synthbias combine . As the last step you can remove all temporary files with hl_synthbias clean . To check if you have had removed the noise you can type imarith bias_256 - synth_bias_256 bias_256_noise. If you check bias_256_noise you'll hopefully see only noise. Now you can use the synthetic bias image, a so called masterbias,  for the bias correction.
Flatfield combination/correction

After creating a masterbias (see Bias combination/correction ) the next step in a usual data redcution is the creation of a masterflat. For this purpose put all flat field images of one run with the same filter in one directory. Then create the inlist for all the files (see File selection ). Before you can start to reduce the flat images you should check that no image is overexposed or have less flux. Use the command imstat *.fits[subfield] to do statistics for each frame. subfield means a central part of the image below or over the middle line, e.g. if you have 256x256 pix images the subfield is something like 1:256,1:120 or 1:256,136:256 .   NOTE: If the mean value exceeds 60000 the stddev value decreases! This means that the images is overexposed. Delete now all overexposed or underexposed images from inlist . Now use the command hl_reduceflats inlist biasframe flatfieldname to create a master flat. The parameter biasframe and flatfieldname specify the masterbias and the resulting masterflat. The masterflat is already normalized to 1! Last step in this part of the reduction is to check the quality of the masterflat.  It is absolutly necessary if you've used flat fields from several nights. Usually the flat structures don't change so much, but .... you know! Take one of the processed flat fields (e.g. f_bias_12345) and divide this image with the masterflat with imarith  f_bias_12345 / masterflat flat_test . If you now check the image flat_test you shouldn't see any flat structures. If this check fails you have to devide the flat fields into groups (ususally nights) and create a masterflat for each group. Sometimes it is possible to combine one or two nights, but the combination of all nights fails.
Bad pixel correction

For the correct handling of bad pixels and bad columns in the CCD frames you can perform the IRAF bad pixel command. For each image size we have a special bad pixel file (see Download Area). To correct a list of images with the same dimensions use the following command: hl_badpix worklist badpixfile. worklist holds the filename of the images which have to be corrected and badpixfile is the bad pixel file (see the IRAF manual for further description). This command creates an output file names pixlist containing the processed images.
Science frame reduction

After the successful creation of the masterbias and the masterflats for each filter you can now perform the reduction of the science frame. First create the inlist of all science frames with the same image size (see File selection ). For the data reduction  you have to use the command hl_reduce2k (the name is somewhat historical and has nothing to do with the input image size!), e.g. hl_reduce2k inlist biasframe flatframe. The parameter inlist is the input list and biasframe and flatframe specifiy the masterbias (see Bias combination/correction ) and masterflat (see Flatfield combination/correction ) images. The resulting filename are stored in flatlist. After this basic data reduction you should perfom the bad pixel correction (see Bad pixel correction). For this work use the command hl_badpix flatlist badpixfile. The names of the now fully reduced frames are store in the file pixlist. This file can be used e.g. to convert the files back into the FITS file format with the hl_wfits command for further analysis with other software packages (e.g. IDL).
Nomenclature of files

xxxxxx = number of the image

hl_xxxxxx.fits                    = raw image
xxxxxx.hh[hd]                   = in IRAF format converted image
bias_xxxxxx.hh[hd]         = image afer overscan correction
f_bias_xxxxxx.hh[hd]     = image after complete bias and flat correction
Description of all functions:

hl_badpix worklist badpixfile


hl_biascor worklist outlist

hl_biacor performs an overscan correction on all files listed in worklist. The names of the corrected files are stored in outlist .


hl_imcopy source_image destination_image

The imcopy command in iraf has some bugs, it deletes the object name of the image. hl_imcopy dublicates an images which must be in .hhh format. Normally this function ist used inside other functions.


hl_reduce2k iolist bias_image  flat_image

hl_reduce2k performs the complete data reduction bias correction and flatfielding on a number of files which are listed in iolist. bias_image and flat_image are the  masterbias and masterflat frames used for the reduction. The command produces a output list of all reduced images named flatlist.


hl_reduceflats iolist bias_frame flat_image

The hl_reduceflats command reduces all frames listed in iolist and combines them to a masterflat flat_image. bias_frame is the masterbias used for this action.


hl_rfits iolist outlist

hl_rfits converts all files from iolist into the IRAF hhh format (see File selection for the format of the iolist). The names of the IRAF files are stored in outlist. NOTE: Don't forget to set imtype to hhh with set imtype=hhh in your login.cl!


hl_selectfiles filemask database_file keyword exact_value

hl_selectfiles select all files with the given filemask from the current directory which match the exact_value in the FITS-header entry  keyword. The filenames of the selected files are stored in the file database_file.

Example:  hl_selectfiles *.fits inlist FILTERS V

This command selects all V exposures  with the extension .fits .


hl_synthbias command

With the hl_synthbias command you can synthesize a bias frame. command indicates the subcommand you want to use:
Other parameters:
The best way to set all parameters is to use epa hl_synthbias. For one of our reduction we used an order of 21 and the chebyshev funtion. Note higher orders can result in some sinus like functions which don't fit the bias structure. Check also the fitting behavour near x=0 if you use the legendre function. For further information about the fitting process read the manual for the IRAF command linebias and colbias.


hl_wfits iolist outlist

hl_wfits convert the FILES in iolist from the IRAF format into the FITS format (see File selection for the format of the iolist). outlist holds the resulting filenames.

Back to the index

Last changed: 29/10/01, Oliver Cordes (email: ocordes@astro.uni-bonn.de)