LSST Applications  21.0.0-172-gfb10e10a+18fedfabac,22.0.0+297cba6710,22.0.0+80564b0ff1,22.0.0+8d77f4f51a,22.0.0+a28f4c53b1,22.0.0+dcf3732eb2,22.0.1-1-g7d6de66+2a20fdde0d,22.0.1-1-g8e32f31+297cba6710,22.0.1-1-geca5380+7fa3b7d9b6,22.0.1-12-g44dc1dc+2a20fdde0d,22.0.1-15-g6a90155+515f58c32b,22.0.1-16-g9282f48+790f5f2caa,22.0.1-2-g92698f7+dcf3732eb2,22.0.1-2-ga9b0f51+7fa3b7d9b6,22.0.1-2-gd1925c9+bf4f0e694f,22.0.1-24-g1ad7a390+a9625a72a8,22.0.1-25-g5bf6245+3ad8ecd50b,22.0.1-25-gb120d7b+8b5510f75f,22.0.1-27-g97737f7+2a20fdde0d,22.0.1-32-gf62ce7b1+aa4237961e,22.0.1-4-g0b3f228+2a20fdde0d,22.0.1-4-g243d05b+871c1b8305,22.0.1-4-g3a563be+32dcf1063f,22.0.1-4-g44f2e3d+9e4ab0f4fa,22.0.1-42-gca6935d93+ba5e5ca3eb,22.0.1-5-g15c806e+85460ae5f3,22.0.1-5-g58711c4+611d128589,22.0.1-5-g75bb458+99c117b92f,22.0.1-6-g1c63a23+7fa3b7d9b6,22.0.1-6-g50866e6+84ff5a128b,22.0.1-6-g8d3140d+720564cf76,22.0.1-6-gd805d02+cc5644f571,22.0.1-8-ge5750ce+85460ae5f3,master-g6e05de7fdc+babf819c66,master-g99da0e417a+8d77f4f51a,w.2021.48
LSST Data Management Base Package
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.obs.decam.ingest.DecamParseTask Class Reference
Inheritance diagram for lsst.obs.decam.ingest.DecamParseTask:
lsst.pipe.tasks.ingest.ParseTask

Public Member Functions

def __init__ (self, *args, **kwargs)
 
def buildExpnumMapper (self, basepath)
 
def getInfo (self, filename, filetype="raw")
 
def getDestination (self, butler, info, filename, filetype="raw")
 
def getInfo (self, filename)
 
def getInfoFromMetadata (self, md, info=None)
 
def translate_date (self, md)
 
def translate_filter (self, md)
 
def getDestination (self, butler, info, filename)
 

Static Public Member Functions

def getExtensionName (md)
 

Public Attributes

 expnumMapper
 
 instcalPrefix
 
 dqmaskPrefix
 
 wtmapPrefix
 

Static Public Attributes

 ConfigClass = ParseConfig
 
 translator_class = None
 

Detailed Description

Parse an image filename to get the required information to
put the file in the correct location and populate the registry.

Definition at line 95 of file ingest.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.obs.decam.ingest.DecamParseTask.__init__ (   self,
*  args,
**  kwargs 
)

Definition at line 102 of file ingest.py.

102 
103  _translatorClass = DecamTranslator
104 
105  def __init__(self, *args, **kwargs):
106  super(ParseTask, self).__init__(*args, **kwargs)
107 
108  self.expnumMapper = None
109 
110  # Note that these should be syncronized with the fields in
111  # root.register.columns defined in config/ingest.py
112  self.instcalPrefix = "instcal"

Member Function Documentation

◆ buildExpnumMapper()

def lsst.obs.decam.ingest.DecamParseTask.buildExpnumMapper (   self,
  basepath 
) 
Extract exposure numbers from filenames to set self.expnumMapper

Parameters
----------
basepath : `str`
    Location on disk of instcal, dqmask, and wtmap subdirectories.

Definition at line 127 of file ingest.py.

127  self.dqmaskPrefix: None}
128  self.expnumMapper[expnum][prefix] = fileName
129 
130  def buildExpnumMapper(self, basepath):
131  """Extract exposure numbers from filenames to set self.expnumMapper
132 
133  Parameters
134  ----------
135  basepath : `str`
136  Location on disk of instcal, dqmask, and wtmap subdirectories.
137  """
138  self.expnumMapper = {}
139 
140  instcalPath = basepath
141  dqmaskPath = re.sub(self.instcalPrefix, self.dqmaskPrefix, instcalPath)
142  wtmapPath = re.sub(self.instcalPrefix, self.wtmapPrefix, instcalPath)
143  if instcalPath == dqmaskPath:
144  raise RuntimeError("instcal and mask directories are the same")
145  if instcalPath == wtmapPath:
146  raise RuntimeError("instcal and weight map directories are the same")
147 
148  if not os.path.isdir(dqmaskPath):
149  raise OSError("Directory %s does not exist" % (dqmaskPath))
150  if not os.path.isdir(wtmapPath):
151  raise OSError("Directory %s does not exist" % (wtmapPath))
152 
153  # Traverse each directory and extract the expnums
154  for path, prefix in zip((instcalPath, dqmaskPath, wtmapPath),

◆ getDestination() [1/2]

def lsst.pipe.tasks.ingest.ParseTask.getDestination (   self,
  butler,
  info,
  filename 
)
inherited 
Get destination for the file

@param butler      Data butler
@param info        File properties, used as dataId for the butler
@param filename    Input filename
@return Destination filename

Reimplemented in lsst.pipe.tasks.ingestCalibs.CalibsParseTask, and lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.

Definition at line 207 of file ingest.py.

207  def getDestination(self, butler, info, filename):
208  """Get destination for the file
209 
210  @param butler Data butler
211  @param info File properties, used as dataId for the butler
212  @param filename Input filename
213  @return Destination filename
214  """
215  raw = butler.get("raw_filename", info)[0]
216  # Ensure filename is devoid of cfitsio directions about HDUs
217  c = raw.find("[")
218  if c > 0:
219  raw = raw[:c]
220  return raw
221 
222 

◆ getDestination() [2/2]

def lsst.obs.decam.ingest.DecamParseTask.getDestination (   self,
  butler,
  info,
  filename,
  filetype = "raw" 
) 
Get destination for the file

Parameters
----------
butler : `lsst.daf.persistence.Butler`
    Data butler.
info : data ID
    File properties, used as dataId for the butler.
filename : `str`
    Input filename.

Returns
-------
raw : `str`
    Destination filename.

Definition at line 228 of file ingest.py.

228 
229  return phuInfo, infoList
230 
231  def getDestination(self, butler, info, filename, filetype="raw"):
232  """Get destination for the file
233 
234  Parameters
235  ----------
236  butler : `lsst.daf.persistence.Butler`
237  Data butler.
238  info : data ID
239  File properties, used as dataId for the butler.
240  filename : `str`
241  Input filename.
242 
243  Returns
244  -------
245  raw : `str`
246  Destination filename.
247  """
248  raw = butler.get("%s_filename"%(filetype), info)[0]
249  # Ensure filename is devoid of cfitsio directions about HDUs
250  c = raw.find("[")
251  if c > 0:

◆ getExtensionName()

def lsst.pipe.tasks.ingest.ParseTask.getExtensionName (   md)
staticinherited 
 Get the name of a FITS extension.

Parameters
----------
md : `lsst.daf.base.PropertySet`
    FITS header metadata.

Returns
-------
result : `str` or `None`
    The string from the EXTNAME header card if it exists. None otherwise.

Definition at line 126 of file ingest.py.

126  def getExtensionName(md):
127  """ Get the name of a FITS extension.
128 
129  Parameters
130  ----------
131  md : `lsst.daf.base.PropertySet`
132  FITS header metadata.
133 
134  Returns
135  -------
136  result : `str` or `None`
137  The string from the EXTNAME header card if it exists. None otherwise.
138  """
139  try:
140  ext = md["EXTNAME"]
141  except KeyError:
142  ext = None
143  return ext
144 

◆ getInfo() [1/2]

def lsst.pipe.tasks.ingest.ParseTask.getInfo (   self,
  filename 
)
inherited 
Get information about the image from the filename and its contents

Here, we open the image and parse the header, but one could also look at the filename itself
and derive information from that, or set values from the configuration.

Parameters
----------
filename : `str`
    Name of file to inspect

Returns
-------
phuInfo : `dict`
    File properties
infoList : `list`
    List of file properties for each extension

Reimplemented in lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.

Definition at line 80 of file ingest.py.

80  def getInfo(self, filename):
81  """Get information about the image from the filename and its contents
82 
83  Here, we open the image and parse the header, but one could also look at the filename itself
84  and derive information from that, or set values from the configuration.
85 
86  Parameters
87  ----------
88  filename : `str`
89  Name of file to inspect
90 
91  Returns
92  -------
93  phuInfo : `dict`
94  File properties
95  infoList : `list`
96  List of file properties for each extension
97  """
98  md = readMetadata(filename, self.config.hdu)
99  fix_header(md, translator_class=self.translator_class)
100  phuInfo = self.getInfoFromMetadata(md)
101  if len(self.config.extnames) == 0:
102  # No extensions to worry about
103  return phuInfo, [phuInfo]
104  # Look in the provided extensions
105  extnames = set(self.config.extnames)
106  extnum = 0
107  infoList = []
108  while len(extnames) > 0:
109  extnum += 1
110  try:
111  md = readMetadata(filename, extnum)
112  fix_header(md, translator_class=self.translator_class)
113  except Exception as e:
114  self.log.warning("Error reading %s extensions %s: %s", filename, extnames, e)
115  break
116  ext = self.getExtensionName(md)
117  if ext in extnames:
118  hduInfo = self.getInfoFromMetadata(md, info=phuInfo.copy())
119  # We need the HDU number when registering MEF files.
120  hduInfo["hdu"] = extnum
121  infoList.append(hduInfo)
122  extnames.discard(ext)
123  return phuInfo, infoList
124 
set
daf::base::PropertySet * set
Definition: fits.cc:912
lsst::afw::fits::readMetadata
std::shared_ptr< daf::base::PropertyList > readMetadata(std::string const &fileName, int hdu=DEFAULT_HDU, bool strip=false)
Read FITS header.
Definition: fits.cc:1657

◆ getInfo() [2/2]

def lsst.obs.decam.ingest.DecamParseTask.getInfo (   self,
  filename,
  filetype = "raw" 
) 
Get metadata header info from multi-extension FITS decam image file.

The science pixels, mask, and weight (inverse variance) are
stored in separate files each with a unique name but with a
common unique identifier EXPNUM in the FITS header.  We have
to aggregate the 3 filenames for a given EXPNUM and return
this information along with that returned by the base class.

Parameters
----------
filename : `str`
    Image file to retrieve info from.
filetype : `str`
    One of "raw" or "instcal".

Returns
-------
phuInfo : `dict`
    Primary header unit info.
infoList : `list` of `dict`
    Info for the other HDUs.

Notes
-----
For filetype="instcal", we expect a directory structure that looks
like the following:

.. code-block:: none

    dqmask/
    instcal/
    wtmap/

The user creates the registry by running:

.. code-block:: none

    ingestImagesDecam.py outputRepository --filetype=instcal --mode=link instcal/*fits

Definition at line 155 of file ingest.py.

155  (self.instcalPrefix, self.dqmaskPrefix, self.wtmapPrefix)):
156  self._listdir(path, prefix)
157 
158  def getInfo(self, filename, filetype="raw"):
159  """Get metadata header info from multi-extension FITS decam image file.
160 
161  The science pixels, mask, and weight (inverse variance) are
162  stored in separate files each with a unique name but with a
163  common unique identifier EXPNUM in the FITS header. We have
164  to aggregate the 3 filenames for a given EXPNUM and return
165  this information along with that returned by the base class.
166 
167  Parameters
168  ----------
169  filename : `str`
170  Image file to retrieve info from.
171  filetype : `str`
172  One of "raw" or "instcal".
173 
174  Returns
175  -------
176  phuInfo : `dict`
177  Primary header unit info.
178  infoList : `list` of `dict`
179  Info for the other HDUs.
180 
181  Notes
182  -----
183  For filetype="instcal", we expect a directory structure that looks
184  like the following:
185 
186  .. code-block:: none
187 
188  dqmask/
189  instcal/
190  wtmap/
191 
192  The user creates the registry by running:
193 
194  .. code-block:: none
195 
196  ingestImagesDecam.py outputRepository --filetype=instcal --mode=link instcal/*fits
197  """
198  if filetype == "instcal":
199  if self.expnumMapper is None:
200  self.buildExpnumMapper(os.path.dirname(os.path.abspath(filename)))
201 
202  # Note that phuInfo will have
203  # 'side': 'X', 'ccd': 0
204  phuInfo, infoList = super(DecamParseTask, self).getInfo(filename)
205  expnum = phuInfo["visit"]
206  phuInfo[self.instcalPrefix] = self.expnumMapper[expnum][self.instcalPrefix]
207  phuInfo[self.dqmaskPrefix] = self.expnumMapper[expnum][self.dqmaskPrefix]
208  phuInfo[self.wtmapPrefix] = self.expnumMapper[expnum][self.wtmapPrefix]
209  for info in infoList:
210  expnum = info["visit"]
211  info[self.instcalPrefix] = self.expnumMapper[expnum][self.instcalPrefix]
212  info[self.dqmaskPrefix] = self.expnumMapper[expnum][self.dqmaskPrefix]
213  info[self.wtmapPrefix] = self.expnumMapper[expnum][self.wtmapPrefix]
214 
215  elif filetype == "raw":
216  phuInfo, infoList = super(DecamParseTask, self).getInfo(filename)
217  for info in infoList:
218  info[self.instcalPrefix] = ""
219  info[self.dqmaskPrefix] = ""
220  info[self.wtmapPrefix] = ""
221 
222  # Some data IDs can not be extracted from the zeroth extension
223  # of the MEF. Add them so Butler does not try to find them
224  # in the registry which may still yet to be created.
225  for key in ("ccdnum", "hdu", "ccd", "calib_hdu"):
226  if key not in phuInfo:
227  phuInfo[key] = 0

◆ getInfoFromMetadata()

def lsst.pipe.tasks.ingest.ParseTask.getInfoFromMetadata (   self,
  md,
  info = None 
)
inherited 
Attempt to pull the desired information out of the header

This is done through two mechanisms:
* translation: a property is set directly from the relevant header keyword
* translator: a property is set with the result of calling a method

The translator methods receive the header metadata and should return the
appropriate value, or None if the value cannot be determined.

@param md      FITS header
@param info    File properties, to be supplemented
@return info

Definition at line 145 of file ingest.py.

145  def getInfoFromMetadata(self, md, info=None):
146  """Attempt to pull the desired information out of the header
147 
148  This is done through two mechanisms:
149  * translation: a property is set directly from the relevant header keyword
150  * translator: a property is set with the result of calling a method
151 
152  The translator methods receive the header metadata and should return the
153  appropriate value, or None if the value cannot be determined.
154 
155  @param md FITS header
156  @param info File properties, to be supplemented
157  @return info
158  """
159  if info is None:
160  info = {}
161  for p, h in self.config.translation.items():
162  value = md.get(h, None)
163  if value is not None:
164  if isinstance(value, str):
165  value = value.strip()
166  info[p] = value
167  elif p in self.config.defaults:
168  info[p] = self.config.defaults[p]
169  else:
170  self.log.warning("Unable to find value for %s (derived from %s)", p, h)
171  for p, t in self.config.translators.items():
172  func = getattr(self, t)
173  try:
174  value = func(md)
175  except Exception as e:
176  self.log.warning("%s failed to translate %s: %s", t, p, e)
177  value = None
178  if value is not None:
179  info[p] = value
180  return info
181 

◆ translate_date()

def lsst.pipe.tasks.ingest.ParseTask.translate_date (   self,
  md 
)
inherited 
Convert a full DATE-OBS to a mere date

Besides being an example of a translator, this is also generally useful.
It will only be used if listed as a translator in the configuration.

Reimplemented in lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.

Definition at line 182 of file ingest.py.

182  def translate_date(self, md):
183  """Convert a full DATE-OBS to a mere date
184 
185  Besides being an example of a translator, this is also generally useful.
186  It will only be used if listed as a translator in the configuration.
187  """
188  date = md.getScalar("DATE-OBS").strip()
189  c = date.find("T")
190  if c > 0:
191  date = date[:c]
192  return date
193 
strip
bool strip
Definition: fits.cc:911

◆ translate_filter()

def lsst.pipe.tasks.ingest.ParseTask.translate_filter (   self,
  md 
)
inherited 
Translate a full filter description into a mere filter name

Besides being an example of a translator, this is also generally useful.
It will only be used if listed as a translator in the configuration.

Reimplemented in lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.

Definition at line 194 of file ingest.py.

194  def translate_filter(self, md):
195  """Translate a full filter description into a mere filter name
196 
197  Besides being an example of a translator, this is also generally useful.
198  It will only be used if listed as a translator in the configuration.
199  """
200  filterName = md.getScalar("FILTER").strip()
201  filterName = filterName.strip()
202  c = filterName.find(" ")
203  if c > 0:
204  filterName = filterName[:c]
205  return filterName
206 

Member Data Documentation

◆ ConfigClass

lsst.pipe.tasks.ingest.ParseTask.ConfigClass = ParseConfig
staticinherited

Definition at line 70 of file ingest.py.

◆ dqmaskPrefix

lsst.obs.decam.ingest.DecamParseTask.dqmaskPrefix

Definition at line 110 of file ingest.py.

◆ expnumMapper

lsst.obs.decam.ingest.DecamParseTask.expnumMapper

Definition at line 105 of file ingest.py.

◆ instcalPrefix

lsst.obs.decam.ingest.DecamParseTask.instcalPrefix

Definition at line 109 of file ingest.py.

◆ translator_class

lsst.pipe.tasks.ingest.ParseTask.translator_class = None
staticinherited

Definition at line 71 of file ingest.py.

◆ wtmapPrefix

lsst.obs.decam.ingest.DecamParseTask.wtmapPrefix

Definition at line 111 of file ingest.py.

The documentation for this class was generated from the following file:
Generated on Fri Nov 26 2021 10:16:26 for LSST Applications by doxygen 1.9.1