## "@(#) $Id: README,v 1.23 2006/02/02 11:46:25 jpritcha Exp $"
README file for the ESO-MIDAS FEROS Data Reduction System Environment at the
MPG/ESO-2.20m Telescope, ESO La Silla.

This archive (FEROS-DRS.tgz) contains the FEROS Data Reduction System MIDAS
scripts as implemented at the MPG/ESO-2.20m telescope, ESO, La Silla, plus
some tools and associated files and two DRS Initial guess files which can be
used to startup the DRS.

0.0 Requirements
================
The scripts have been prepared under Linux with a view to portability.

The prepRawData script relies on "gethead" which is part of the wcs-tools
package which can be downloaded from$
      http://tdc-www.harvard.edu/software/wcstools.html

Please report any problems you experience with the MIDAS scripts or the
other files in the archive to:

     ls-hires@eso.org


1.0 Installation
================
Normally this archive should be extracted into the user's home directory,
though you may want to extract to a dedicated subdirectory to avoid the
possibility of overwriting already existing files.

In the following it will be assumed the user creates a directory FEROS-DRS in
their Home directory in which to extract the file:

   mkdir $HOME/FEROS-DRS
   cd $HOME/FEROS-DRS
   tar -zxvf <dir>/FEROS-DRS.tgz
or alternatively...
   gzip -dc <dir>/FEROS-DRS.tgz | tar -xvf -

This should produce six directories and this README file...

bin
dat
fmt
guess
midwork
README
saved_session

You also need to define an alias for inmidas.feros and add
$HOME/FEROS-DRS/bin to your PATH environment variable...

*) For tcsh, add the following lines to .cshrc (or .tcshrc):

setenv FEROS_DRS     "${HOME}/FEROS-DRS"
setenv FEROS_MIDWORK "${FEROS_DRS}/midwork"
alias inmidas.feros  "inmidas -m ${FEROS_MIDWORK}"
setenv PATH ${PATH}:${FEROS_DRS}/bin
if ( $?PERLLIB ) then
  setenv PERLLIB ${PERLLIB}:${FEROS_DRS}/lib/perl
else
  setenv PERLLIB ${FEROS_DRS}/lib/perl
endif

You *MIGHT* also/alternatively (e.g. on Mac-OSX) need:

if ( $?PERL5LIB ) then
  setenv PERL5LIB ${PERL5LIB}:${FEROS_DRS}/lib/perl
else
  setenv PERL5LIB ${FEROS_DRS}/lib/perl
endif



*) For Bash, add the following lines to .bashrc

export FEROS_DRS="${HOME}/FEROS-DRS"
export FEROS_MIDWORK="${FEROS_DRS}/midwork"
alias inmidas.feros="inmidas -m ${FEROS_MIDWORK}"
export PATH=${PATH}:${FEROS_DRS}/bin
export PERLLIB=${PERLLIB}:${FEROS_DRS}/lib/perl

You *MIGHT* also/alternatively (e.g. on Mac-OSX) need:

export PERL5LIB=${PERL5LIB}:${FEROS_DRS}/lib/perl

Under bash it doesn't matter whether PERLLIB and/or PERL5LIB pre-exist or not.


To get MIDAS running in a nice blue separate xterminal just like at
the telescope use the following for you inmidas.feros alias:

*) For tcsh, add the following lines to .cshrc (or .tcshrc):

alias inmidas.feros "xterm -ls -bg blue -fg white -geometry 80x34+0-6 -fn 8x16 -sb -e inmidas -m ${FEROS_MIDWORK}"

*) For Bash, add the following lines to .bashrc

alias inmidas.feros="xterm -ls -bg blue -fg white -geometry 80x34+0-6 -fn 8x16 -sb -e inmidas -m ${FEROS_MIDWORK:-$HOME/FEROS-DRS/midwork}"

1.1 Installation on w2p2off
---------------------------
For installation on w2p2off, i.e. at the 2.20m telescope, the following
environment variables must be set (normally in .pecs/misc-all.env) for the
account running the pipeline (normally user astro):

## For the FEROS Pipeline...
export PATH=${HOME}/bin:${HOME}/scripts:${PATH}
export PERLLIB=${PERLLIB}:/vlt/INTROOT/E2P2OPS/lib/perl
export FEROS_DRS="${HOME}/FEROS-DRS"
export FEROS_MIDWORK="${FEROS_DRS}/midwork"
export PATH=${PATH}:${FEROS_DRS}/bin
export FEROSPackRedFiles="Y"
export FEROSProcImage="Y"
export FEROSmkHTML="Y"
export FEROSDoQC="Y"
export FEROSUseAFDRSFITS="Y"
export FEROSFFHVImFitsFile="/data/E2P2OPS/FEROS-Archive/FFHV"
export FEROSDRStars2DFS="Y"
export FEROSDRSDS9mkPS="Y"
export FEROSDRSDS9useVLTFN="Y"
export FEROSAutoInit="Y"
export FEROSQCURLroot="file:///data/E2P2OPS/FEROS-QC/html/QC"
## Because it is installed in /data/E2P2OPS/FEROS-DRS...
export PATH=${PATH}:/data/E2P2OPS/FEROS-DRS/bin
## To include Log entries in AstroCD packages:
export CopyLogEntryToDataRed="Y"

Normally these are set in the ~/astro/.pecs/misc-all.env file.

1.2 Screen Colour Depth
-----------------------
In the file midwork/login.prg it is assumed the X11 Screen Colour Depth will
be 24-bit. If you do not have this you will need to comment out the line:

init/displ p5=RGBQ

1.3 Graphic and display windows
-------------------------------
Several of the integral MIDAS prg files now require FIVE MIDAS graphic
windows (with IDs 0, 1, 2, 3, 4) to exist before running. By default these
are created in a rather simple fashion by the login.prg file, however
several other more sophisticated examples are provided in the files
midwork/creaDandG.prg.alt*. To use one of these examples you can simply make
a softlink pointing at the relevant one, e.g.:

   cd $HOME/FEROS-DRS/midwork
   ln -s creaDandG.prg.alt0 creaDandG.prg

If you want to customise one of the alternatives, then (for example):

   cd $HOME/FEROS-DRS/midwork
   cp creaDandG.prg.alt0 creaDandG.prg
   $EDITOR creaDandG.prg

or if you use the same midwork directory from different computers with
different Video capabilities you may want to create a HOST specific
creaDandG.prg:

   cd $HOME/FEROS-DRS/midwork
   cp creaDandG.prg.alt0 creaDandG.prg.${HOST}
   $EDITOR creaDandG.prg.${HOST}

In either of the two above cases the files creaDandG.prg,
creaDandG.prg.${HOST} will survive an installation to a new version of the
FEROS-DRS.tgz package. If you make a link but then edit the original file,
these modifications will be overwritten when a new version of the
FEROS-DRS.tgz package is installed.


2.0 Use
=======

As an example, assume we want to reduce the data from the night of
2003-01-21...


2.1 Prepare the data directory
------------------------------

   cd ~/FEROS-DRS
   ferosPrepDateDir 2003-01-21

This will create a directory 2003-01-21 and then make softlinks pointing to
the initial guess files. With no -g <guess> option to ferosPrepDateDir, the
default initial guess (9001) will be used. If the initial guess files do not
already exist they will be created in the separate directory guess9001.


2.2 Extract the data from the Archive CDs
-----------------------------------------

   ferosPrepRawData 2003-01-21 -r <cdrom-Mount-Point>/2003-01-21

This will uncompress the files on the FEROS Archive CD and rename them with
their original name (i.e. in fero<NNNN>.mt format for BIAS images or
FEROS_ech_... for FIERA images). The script will automatically detect if the
image was created with FIERA and if so then attempt to make softlinks of the
form fero<NNNN>.mt using the ferosMkferoNNNNLink script.

You may need to do the above command several times if the data from one
night is on more than one CD.

In addition to the Image files, the LOG files will be copied to. If there
are images you are not interested in they can be easily removed using the
log file and the ferosCleanByLog script. Simple edit the log file and remove
the entries for images you do NOT want to reduce and then run the ferosCleanByLog
script:

   cd 2003-01-21
   <Your_favorite_editor> FEROS.2003-01-21.log
   ferosCleanByLog FEROS.2003-01-21.log

2.3 Run the DRS
---------------

   inmidas.feros

And then follow the instructions in ~/FEROS-DRS/midwork/README...


2.4 Pack Reduced files
----------------------

In order to package the reduced files for each reduction as well as the
Guess and Session archives, you need to set the FEROSPackRedFiles
environment variable:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
   setenv FEROSPackRedFiles "Y"

*) For Bash, add the following line to .bashrc
   export FEROSPackRedFiles="Y"


2.5 Make Quality Control Plots
------------------------------

In order to create Quality Control plots, you need to set the FEROSQCPlots
environment variable:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
   setenv FEROSQCPlots "Y"

*) For Bash, add the following line to .bashrc
   export FEROSQCPlots="Y"

This is done automatically at the telescope by detection of the hostname
being equal to w2p2off.


2.6 Make HTML summary pages
---------------------------

In order create HTML summary pages, you need to set the FEROSmkHTML
environment variable:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
   setenv FEROSmkHTML "Y"

*) For Bash, add the following line to .bashrc
   export FEROSmkHTML="Y"


2.7 Choice of filenames for FITS versions of DRS products
---------------------------------------------------------

To have VLT compliant name format for output DRS files, i.e.
r.<ARCFILE>.nnnn.fits (See Appendix 1 below for naming conventions), you
need to set the FEROSUseAFDRSFITS environment variable:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
   setenv FEROSUseAFDRSFITS "Y"

*) For Bash, add the following line to .bashrc
   export FEROSUseAFDRSFITS="Y"


2.8 Inclusion of FFHV TCCD images in DRS tar files
--------------------------------------------------

To include FFHV TCCD images in the DRS product tar files, you need to set
the FEROSFFHVImFitsFile environment variable:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
   setenv FEROSFFHVImFitsFile "<directory>"

*) For Bash, add the following line to .bashrc
   export FEROSFFHVImFitsFile="<directory>"


2.9 Concatenate tar files
-------------------------

To concatenate files to already existing DRS product tar files set the FEROSConcatTarFiles environment variable as follows:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
   setenv FEROSConcatTarFiles "Y"

*) For Bash, add the following line to .bashrc
   export FEROSConcatTarFiles="Y"

2.10 Archive DRS tar files in ESO Science Archive
------------------------------------------------

To copy the the DRS product tar files (including TCCD images) into the ESO
Science Archive via the DFS, you need to set the FEROSDRStars2DFS environment
variable:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
   setenv FEROSDRStars2DFS "Y"

*) For Bash, add the following line to .bashrc
   export FEROSDRStars2DFS="Y"

Needless to say, this option is ONLY available when the DRS is in use at La
Silla on the Offline machine (w2p2off).

2.11 Creation of RAW image PS/PNG files
---------------------------------------

To create PS and/or PNG files (PNGs will then be included in the html pages set
the following environment variables:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
  setenv FEROSDRSDS9mkPS "Y"
  setenv FEROSDRSDS9mkPNG "Y"

*) For Bash, add the following line to .bashrc
  export FEROSDRSDS9mkPS="Y"
  export FEROSDRSDS9mkPNG="Y"

You must have the ds9 software installed (preferably the latest version, 4.0b7
at this writing) and there must exist a softlink ferosDRSds9 which points to
the ds9 binary (though the software will try to create this if it does not
already exist if the ds9 binary can be found using the which command, see
ferosDRSds9cmd)

PNG files are created from the PS files so PNG creation implies PS creation
also. If these variables do not exist nothing happens.

Furthermore the name of the output PS/PNG files can be controlled using the
FEROSDRSDS9useVLTFN environment variable: If set as follows:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
  setenv FEROSDRSDS9useVLTFN "Y"

*) For Bash, add the following line to .bashrc
  export FEROSDRSDS9useVLTFN="Y"

output files will have the filesname FEROS.yyyy-mm-ddThh:mm:ss.sss.ps/png,
otherwise they will be fero<NNNN>.ps/png.

The resulting PNG files will be included in the html files whichever filename
format is used.


2.12 AutoInit
-------------

At the telescope the the daily health check allows initialisation of both
1x1 and 2x2 binning modes. In order to use the correct initialisation during
the night it is necessary to enable AutoInit detection. To do so set the
following environment variable:

*) For tcsh, add the following line to .cshrc (or .tcshrc):
  setenv FEROSAutoInit "Y"

*) For Bash, add the following line to .bashrc
  export FEROSAutoInit="Y"

This will override any manual setting of the FEROS configuration, e.g. to
make a Optimum extraction. Therefore to be able to control the reduction in
Visitor mode, in the MIDAS DRS session set the following MIDAS variable as
follows:

write/key symAutoInit/c/1/1 "N"

The the extraction mode (for example) can be set to OPTIMUM as follows:

set/feros EXT_MODE=O

This functionality is implemented in autoreduce.prg.


2.13 Display QC web pages after init...
---------------------------------------

*) For tcsh, add the following line to .cshrc (or .tcshrc):
  setenv FEROSQCURLroot "<URL>"

*) For Bash, add the following line to .bashrc
  export FEROSQCURLroot="<URL>"


John Pritchard
2006-19-19

============================================================================

Appendix 1
==========
If FEROSUseAFDRSFITS="Y" then DRS products converted to FITS files according
to the following convention:

mDPID = modified DPID = FEROS.YYYY-MM-DDTHH:MM:ss.sss
i.e. without the .fits extension

The one dimensional extracted frames are:

r.<DPID>.1101.fits  <==>  f<NNNN>1.bdf
r.<DPID>.1102.fits  <==>  f<NNNN>2.bdf

The complete list:

r.<DPID>.0011.fits  <==>  back_str_C1.bdf
r.<DPID>.0021.fits  <==>  back_str_D1.bdf
r.<DPID>.0031.fits  <==>  back_str_S1.bdf
r.<DPID>.0011.fits  <==>  back_str_C2.bdf
r.<DPID>.0022.fits  <==>  back_str_D2.bdf
r.<DPID>.0032.fits  <==>  back_str_X2.bdf

Standard extraction:
r.<DPID>.1041.fits  <==>  bf<NNNN>ext1.bdf
r.<DPID>.1042.fits  <==>  bf<NNNN>ext2.bdf
r.<DPID>.1051.fits  <==>  f<NNNN>ext1.bdf
r.<DPID>.1052.fits  <==>  f<NNNN>ext2.bdf
r.<DPID>.1061.fits  <==>  rebinned1.bdf
r.<DPID>.1062.fits  <==>  rebinned2.bdf
r.<DPID>.1063.fits or r.<DPID>.1071.fits  <==>  rebinnedNoFF1.bdf
r.<DPID>.1064.fits or r.<DPID>.1072.fits  <==>  rebinnedNoFF2.bdf
r.<DPID>.1101.fits  <==>  f<NNNN>1.bdf
r.<DPID>.1102.fits  <==>  f<NNNN>2.bdf
r.<DPID>.1103.fits  <==>  noFFf<NNNN>1.bdf
r.<DPID>.1104.fits  <==>  noFFf<NNNN>2.bdf
r.<DPID>.1251.fits  <==>  f<NNNN>10039.bdf
r.<DPID>.1252.fits  <==>  f<NNNN>20039.bdf
r.<DPID>.1253.fits  <==>  noFFf<NNNN>10039.bdf
r.<DPID>.1254.fits  <==>  noFFf<NNNN>20039.bdf
..
r.<DPID>.1631.fits  <==>  f<NNNN>10001.bdf
r.<DPID>.1632.fits  <==>  f<NNNN>20001.bdf
r.<DPID>.1633.fits  <==>  noFFf<NNNN>10001.bdf
r.<DPID>.1634.fits  <==>  noFFf<NNNN>20001.bdf

Optimum extraction:
r.<DPID>.2041.fits  <==>  bo<NNNN>ext1.bdf
r.<DPID>.2042.fits  <==>  bo<NNNN>ext2.bdf
r.<DPID>.2051.fits  <==>  o<NNNN>ext1.bdf
r.<DPID>.2052.fits  <==>  o<NNNN>ext2.bdf
r.<DPID>.2061.fits  <==>  rebinned1.bdf
r.<DPID>.2062.fits  <==>  rebinned2.bdf
r.<DPID>.2063.fits or r.<DPID>.2071.fits  <==>  rebinnedNoFF1.bdf
r.<DPID>.2064.fits or r.<DPID>.2072.fits  <==>  rebinnedNoFF2.bdf
r.<DPID>.2101.fits  <==>  o<NNNN>1.bdf
r.<DPID>.2102.fits  <==>  o<NNNN>2.bdf
r.<DPID>.2103.fits  <==>  noFFo<NNNN>1.bdf
r.<DPID>.2104.fits  <==>  noFFo<NNNN>2.bdf
r.<DPID>.2251.fits  <==>  o<NNNN>10039.bdf
r.<DPID>.2252.fits  <==>  o<NNNN>20039.bdf
r.<DPID>.2253.fits  <==>  noFFo<NNNN>10039.bdf
r.<DPID>.2254.fits  <==>  noFFo<NNNN>20039.bdf
..
r.<DPID>.2631.fits  <==>  o<NNNN>10001.bdf
r.<DPID>.2632.fits  <==>  o<NNNN>20001.bdf
r.<DPID>.2633.fits  <==>  noFFo<NNNN>10001.bdf
r.<DPID>.2634.fits  <==>  noFFo<NNNN>20001.bdf

